Sep 12 00:27:28.669093 ip-10-0-130-132 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 12 00:27:28.669106 ip-10-0-130-132 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 12 00:27:28.669116 ip-10-0-130-132 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 12 00:27:28.669457 ip-10-0-130-132 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 12 00:27:38.138375 ip-10-0-130-132 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 12 00:27:38.138389 ip-10-0-130-132 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 92271039919b4913b01ddc1e9fe19310 --
Sep 12 00:29:27.672881 ip-10-0-130-132 systemd[1]: Starting Kubernetes Kubelet...
Sep 12 00:29:28.074014 ip-10-0-130-132 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 00:29:28.074014 ip-10-0-130-132 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 12 00:29:28.074014 ip-10-0-130-132 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 00:29:28.074014 ip-10-0-130-132 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 12 00:29:28.074014 ip-10-0-130-132 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 00:29:28.074847 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.074104    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 12 00:29:28.078201 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078176    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:29:28.078201 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078197    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:29:28.078201 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078204    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:29:28.078201 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078209    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078213    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078219    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078223    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078227    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078231    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078235    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078239    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078243    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078247    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078251    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078255    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078259    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078262    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078266    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078270    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078273    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078277    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078290    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078294    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:29:28.078463 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078298    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078302    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078306    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078310    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078314    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078318    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078323    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078326    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078330    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078334    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078339    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078345    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078353    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078362    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078367    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078373    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078378    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078382    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078390    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:29:28.079322 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078395    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078399    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078403    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078408    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078412    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078417    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078423    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078429    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078434    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078439    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078445    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078450    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078455    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078460    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078464    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078468    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078472    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078476    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078480    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:29:28.080144 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078484    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078488    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078495    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078499    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078503    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078507    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078512    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078517    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078522    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078526    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078531    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078535    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078539    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078544    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078548    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078553    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078557    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078562    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078584    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:29:28.080937 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078589    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078594    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078599    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078604    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078608    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078612    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078617    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078622    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078627    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078631    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078635    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078639    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078643    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078647    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078652    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078656    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078660    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078665    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.078669    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079867    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:29:28.081767 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079878    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079883    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079888    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079892    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079900    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079907    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079912    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079917    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079922    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079927    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079931    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079936    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079940    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079945    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079950    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079955    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079959    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079963    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079968    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079972    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:29:28.082711 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079977    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079982    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079986    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079990    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079995    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.079999    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080003    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080007    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080039    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080046    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080050    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080056    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080060    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080064    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080068    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080073    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080077    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080082    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080087    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:29:28.083527 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080091    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080095    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080099    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080103    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080108    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080112    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080116    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080121    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080127    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080131    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080136    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080140    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080144    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080148    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080153    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080157    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080161    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080166    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080170    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080174    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:29:28.084130 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080179    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080183    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080187    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080192    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080197    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080205    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080211    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080215    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080219    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080223    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080231    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080237    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080242    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080246    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080250    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080254    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080259    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080264    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080268    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:29:28.084780 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080272    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080277    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080282    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080286    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080290    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080294    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080298    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080302    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080307    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080311    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080315    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080319    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080323    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080328    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080332    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080335    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080340    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080345    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080349    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.080353    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:29:28.085262 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081005    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081014    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081020    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081024    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081029    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081033    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081037    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081041    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081045    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081050    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081054    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081058    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081062    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081066    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081071    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081075    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081079    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081084    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081087    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081092    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:29:28.085806 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081096    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081101    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081105    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081109    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081113    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081117    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081121    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081125    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081130    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081134    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081138    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081142    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081146    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081151    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081155    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081159    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081163    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081168    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081172    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081176    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:29:28.086295 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081180    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081184    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081188    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081193    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081197    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081201    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081206    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081209    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081213    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081218    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081222    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081226    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081230    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081235    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081239    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081244    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081249    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081254    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081258    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081262    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:29:28.086814 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081267    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081271    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081276    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081284    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081289    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081295    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081300    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081304    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081309    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081314    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081319    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081324    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081328    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081333    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081337    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081342    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081347    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081355    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081361    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:29:28.087318 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081366    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081370    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081375    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081380    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081384    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081389    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081394    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081399    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081404    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081408    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081413    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081417    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081422    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081431    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081436    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081440    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081444    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081448    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081453    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.081457    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:29:28.087802 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081591    2567 flags.go:64] FLAG: --address="0.0.0.0"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081645    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081658    2567 flags.go:64] FLAG: --anonymous-auth="true"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081666    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081672    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081678    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081684    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081692    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081697    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081702    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081711    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081716    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081721    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081726    2567 flags.go:64] FLAG: --cgroup-root=""
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081731    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081737    2567 flags.go:64] FLAG: --client-ca-file=""
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081742    2567 flags.go:64] FLAG: --cloud-provider="external"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081747    2567 flags.go:64] FLAG: --cluster-dns="[]"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081753    2567 flags.go:64] FLAG: --cluster-domain=""
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081758    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081764    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081770    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081775    2567 flags.go:64] FLAG: --container-log-max-files="5"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081781    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 12 00:29:28.088323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081786    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081802    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081808    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081813    2567 flags.go:64] FLAG: --contention-profiling="false"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081818    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081823    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081829    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081834    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081840    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081845    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081849    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081855    2567 flags.go:64] FLAG: --enable-load-reader="false"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081860    2567 flags.go:64] FLAG: --enable-server="true"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081865    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081873    2567 flags.go:64] FLAG: --event-burst="100"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081878    2567 flags.go:64] FLAG: --event-qps="50"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081883    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081888    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081895    2567 flags.go:64] FLAG: --eviction-hard=""
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081902    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081907    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081912    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081918    2567 flags.go:64] FLAG: --eviction-soft=""
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081923    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081928    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 12 00:29:28.088945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081933    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081938    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081943    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081947    2567 flags.go:64] FLAG: --fail-swap-on="true"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081953    2567 flags.go:64] FLAG: --feature-gates=""
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081960    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081965    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081969    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081977    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081982    2567 flags.go:64] FLAG: --healthz-port="10248"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081987    2567 flags.go:64] FLAG: --help="false"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081992    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.081997    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082002    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082007    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082013    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082019    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082024    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082028    2567 flags.go:64] FLAG: --image-service-endpoint=""
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082033    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082037    2567 flags.go:64] FLAG: --kube-api-burst="100"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082042    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082047    2567 flags.go:64] FLAG: --kube-api-qps="50"
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082052    2567 flags.go:64] FLAG: --kube-reserved=""
Sep 12 00:29:28.089563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082057    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082063    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082069    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082074    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082078    2567 flags.go:64] FLAG: --lock-file=""
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082083    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082088    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082092    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082102    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082107    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082111    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082116    2567 flags.go:64] FLAG: --logging-format="text"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082121    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082127    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082132    2567 flags.go:64] FLAG: --manifest-url=""
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082137    2567 flags.go:64] FLAG: --manifest-url-header=""
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082146    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082151    2567 flags.go:64] FLAG: --max-open-files="1000000"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082158    2567 flags.go:64] FLAG: --max-pods="110"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082163    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082168    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082172    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082177    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082182    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082187    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 12 00:29:28.090212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082192    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082204    2567 flags.go:64] FLAG: --node-status-max-images="50"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082208    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082214    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082219    2567 flags.go:64] FLAG: --pod-cidr=""
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082224    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082233    2567 flags.go:64] FLAG: --pod-manifest-path=""
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082237    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082244    2567 flags.go:64] FLAG: --pods-per-core="0"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082248    2567 flags.go:64] FLAG: --port="10250"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082253    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082258    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bbf13da4267ae155"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082264    2567 flags.go:64] FLAG: --qos-reserved=""
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082270    2567 flags.go:64] FLAG: --read-only-port="10255"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082276    2567 flags.go:64] FLAG: --register-node="true"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082281    2567 flags.go:64] FLAG: --register-with-taints=""
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082288    2567 flags.go:64] FLAG: --registry-burst="10"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082293    2567 flags.go:64] FLAG: --registry-qps="5"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082297    2567 flags.go:64] FLAG: --reserved-cpus=""
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082302    2567 flags.go:64] FLAG: --reserved-memory=""
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082308    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082314    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082320    2567 flags.go:64] FLAG: --rotate-certificates="false"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082324    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082331    2567 flags.go:64] FLAG: --runonce="false"
Sep 12 00:29:28.090857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082336    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082341    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082345    2567 flags.go:64] FLAG: --seccomp-default="false"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082350    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082355    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082360    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082365    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082370    2567 flags.go:64] FLAG: --storage-driver-password="root"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082375    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082379    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082384    2567 flags.go:64] FLAG: --storage-driver-user="root"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082389    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082394    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082400    2567 flags.go:64] FLAG: --system-cgroups=""
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082404    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082414    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082419    2567 flags.go:64] FLAG: --tls-cert-file=""
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082424    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082430    2567 flags.go:64] FLAG: --tls-min-version=""
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082435    2567 flags.go:64] FLAG: --tls-private-key-file=""
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082439    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082444    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082448    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082453    2567 flags.go:64] FLAG: --v="2"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082460    2567 flags.go:64] FLAG: --version="false"
Sep 12 00:29:28.091491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082466    2567 flags.go:64] FLAG: --vmodule=""
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082479    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.082485    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082689    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082698    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082703    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082709    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082714    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082719    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082723    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082730    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082736    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082740    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082745    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082749    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082753    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082758    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082762    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082766    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082771    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:29:28.092159 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082775    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082779    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082785    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082790    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082794    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082798    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082802    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082807    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082811    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082815    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082819    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082823    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082827    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082832    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082836    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082840    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082844    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082853    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082859    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082864    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:29:28.092854 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082868    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082872    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082876    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082880    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082884    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082888    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082893    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082897    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082901    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082905    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082910    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082914    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082918    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082923    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082929    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082933    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082937    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082944    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082950    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082955    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:29:28.093351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082959    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082964    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082968    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082972    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082979    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082984    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082989    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082994    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.082999    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083004    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083073    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083391    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083398    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083403    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083409    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083414    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083419    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083424    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083429    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:29:28.093868 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083434    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083440    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083445    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083450    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083461    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083466    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083470    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083477    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083482    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083486    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083491    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083496    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083501    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083506    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083511    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083515    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083520    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083528    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083533    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083537    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:29:28.094340 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083541    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083545    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.083549    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.084232    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.091671    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.091693    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.092644    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.092653    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092714    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092721    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092725    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092728    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092731    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092733    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092736    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:29:28.094846 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092739    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092744    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092748    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092751    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092754    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092757    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092760    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092763    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092767    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092771    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092775    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092777    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092780    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092783    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092786    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092789    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092791    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092794    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092797    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:29:28.095266 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092799    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092802    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092805    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092809    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092812    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092815    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092818    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092821    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092823    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092826    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092828    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092831    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092834    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092837    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092840    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092842    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092845    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092847    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092850    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092853    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:29:28.095743 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092856    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092859    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092862    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092864    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092867    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092870    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092872    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092875    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092878    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092881    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092883    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092886    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092889    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092891    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092894    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092898    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092900    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092904    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092908    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:29:28.096233 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092911    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092915    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092918    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092920    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092923    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092926    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092928    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092931    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092933    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092936    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092939    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092941    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092944    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092947    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092950    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092952    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092955    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092957    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092960    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092962    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:29:28.096728 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092965    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092967    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092970    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092972    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092975    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092977    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092980    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092982    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092986    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092989    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092992    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092995    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.092998    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093000    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:29:28.097219 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.093006    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093121    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093125    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093128    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093131    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093134    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093137    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093140    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093142    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093147    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093150    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093153    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093158    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093161    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093163    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093166    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093169    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093171    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093174    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:29:28.097582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093176    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093179    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093182    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093185    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093188    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093190    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093193    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093196    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093198    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093201    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093203    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093206    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093209    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093211    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093214    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093217    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093219    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093222    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093225    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093227    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:29:28.098099 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093230    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093233    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093235    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093238    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093241    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093243    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093246    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093248    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093251    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093254    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093256    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093258    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093261    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093264    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093266    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093269    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093272    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093274    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093277    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093280    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:29:28.098595 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093282    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093285    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093287    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093290    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093293    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093297    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093300    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093303    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093306    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093308    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093311    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093313    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093316    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093319    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093322    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093324    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093326    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093329    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093331    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:29:28.099090 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093335    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093337    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093341    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093344    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093347    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093350    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093352    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093355    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093358    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093361    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093363    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093366    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093369    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093372    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093375    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093377    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093380    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093383    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093385    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093388    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:29:28.099580 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093390    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:29:28.100075 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:28.093392    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:29:28.100075 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.093398    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 00:29:28.100075 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.093535    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 12 00:29:28.100075 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.097426    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 12 00:29:28.100075 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.098487    2567 server.go:1019] "Starting client certificate rotation"
Sep 12 00:29:28.100075 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.098600    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 00:29:28.100075 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.098635    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 00:29:28.101013 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.100997    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 00:29:28.102272 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.102251    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 00:29:28.115459 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.115429    2567 log.go:25] "Validated CRI v1 runtime API"
Sep 12 00:29:28.118684 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.118662    2567 log.go:25] "Validated CRI v1 image API"
Sep 12 00:29:28.119705 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.119680    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 12 00:29:28.124522 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.124496    2567 fs.go:135] Filesystem UUIDs: map[21971e32-79b1-4013-8b9c-d235039fef1e:/dev/nvme0n1p3 33930af1-9fe6-4764-b6eb-5cea708dcd7a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 12 00:29:28.124619 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.124520    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 12 00:29:28.128877 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.128847    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 00:29:28.131825 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.131702    2567 manager.go:216] Machine: {Timestamp:2026-09-12 00:29:28.130382732 +0000 UTC m=+0.334383545 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096706 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28bbef1021b2cc56385dbe63d6715c SystemUUID:ec28bbef-1021-b2cc-5638-5dbe63d6715c BootID:92271039-919b-4913-b01d-dc1e9fe19310 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d1:78:59:9a:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d1:78:59:9a:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:d3:58:d7:dc:51 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 12 00:29:28.131825 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.131821    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 12 00:29:28.131938 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.131922    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 12 00:29:28.134398 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.134361    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 12 00:29:28.134544 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.134401    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-132.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 12 00:29:28.135217 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.135206    2567 topology_manager.go:138] "Creating topology manager with none policy"
Sep 12 00:29:28.135257 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.135218    2567 container_manager_linux.go:309] "Creating device plugin manager"
Sep 12 00:29:28.135257 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.135227    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 00:29:28.135794 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.135783    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 00:29:28.136602 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.136591    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 12 00:29:28.140204 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.140188    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 00:29:28.140335 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.140325    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 12 00:29:28.140367 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.140358    2567 kubelet.go:486] "Attempting to sync node with API server"
Sep 12 00:29:28.140395 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.140368    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 12 00:29:28.140395 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.140388    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 12 00:29:28.140446 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.140398    2567 kubelet.go:398] "Adding apiserver pod source"
Sep 12 00:29:28.140446 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.140407    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 12 00:29:28.140820 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.140808    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 00:29:28.140886 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.140827    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 00:29:28.143468 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.143451    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 12 00:29:28.145924 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.145905    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 12 00:29:28.146024 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.145945    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 12 00:29:28.147668 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.147649    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 12 00:29:28.147668 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.147670    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 12 00:29:28.147812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.147680    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 12 00:29:28.147812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.147688    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 12 00:29:28.147812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.147702    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 12 00:29:28.147812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.147711    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 12 00:29:28.147812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.147731    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 12 00:29:28.147812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.147741    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 12 00:29:28.147812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.147752    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 12 00:29:28.147812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.147761    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 12 00:29:28.147812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.147774    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 12 00:29:28.147812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.147783    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 12 00:29:28.148077 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.147964    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zw8vm"
Sep 12 00:29:28.148470 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.148459    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 12 00:29:28.148517 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.148472    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 12 00:29:28.148969 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.148950    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 12 00:29:28.149048 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.149027    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 12 00:29:28.152325 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.152305    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-132.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 12 00:29:28.152809 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.152793    2567 server.go:1268] "Started kubelet"
Sep 12 00:29:28.152920 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.152873    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 12 00:29:28.152968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.152897    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 12 00:29:28.153020 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.153005    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 12 00:29:28.153902 ip-10-0-130-132 systemd[1]: Started Kubernetes Kubelet.
Sep 12 00:29:28.155926 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.155902    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 12 00:29:28.156517 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.155912    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zw8vm"
Sep 12 00:29:28.158935 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.158917    2567 server.go:310] "Adding debug handlers to kubelet server"
Sep 12 00:29:28.160430 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.160409    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 12 00:29:28.161380 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.160481    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-132.ec2.internal.18d46b55c87a01c0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-132.ec2.internal,UID:ip-10-0-130-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-132.ec2.internal,},FirstTimestamp:2026-09-12 00:29:28.152760768 +0000 UTC m=+0.356761580,LastTimestamp:2026-09-12 00:29:28.152760768 +0000 UTC m=+0.356761580,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-132.ec2.internal,}"
Sep 12 00:29:28.163150 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.163132    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 12 00:29:28.163668 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.163637    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 12 00:29:28.163978 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.163955    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found"
Sep 12 00:29:28.164242 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.164226    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 12 00:29:28.164300 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.164248    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 12 00:29:28.164300 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.164229    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 12 00:29:28.164387 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.164334    2567 reconstruct.go:97] "Volume reconstruction finished"
Sep 12 00:29:28.164387 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.164344    2567 reconciler.go:29] "Reconciler: start to sync state"
Sep 12 00:29:28.164720 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.164695    2567 factory.go:55] Registering systemd factory
Sep 12 00:29:28.164720 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.164714    2567 factory.go:223] Registration of the systemd container factory successfully
Sep 12 00:29:28.164956 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.164941    2567 factory.go:153] Registering CRI-O factory
Sep 12 00:29:28.164998 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.164962    2567 factory.go:223] Registration of the crio container factory successfully
Sep 12 00:29:28.165031 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.165019    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 12 00:29:28.165070 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.165061    2567 factory.go:103] Registering Raw factory
Sep 12 00:29:28.165104 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.165076    2567 manager.go:1196] Started watching for new ooms in manager
Sep 12 00:29:28.165506 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.165492    2567 manager.go:349] Starting recovery of all containers
Sep 12 00:29:28.165606 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.165566    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 12 00:29:28.165718 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.165690    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 12 00:29:28.179678 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.179654    2567 manager.go:354] Recovery completed
Sep 12 00:29:28.184378 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.184362    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:29:28.186632 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.186611    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:29:28.186705 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.186650    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:29:28.186705 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.186661    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:29:28.187492 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.187477    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 12 00:29:28.187492 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.187489    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 12 00:29:28.187625 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.187506    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 00:29:28.187819 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.187802    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:29:28.188657 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.188642    2567 policy_none.go:49] "None policy: Start"
Sep 12 00:29:28.188736 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.188663    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 12 00:29:28.188736 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.188678    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 12 00:29:28.188736 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.188709    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:29:28.188870 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.188743    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:29:28.188870 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.188757    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:29:28.189730 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.189717    2567 policy_none.go:47] "Start"
Sep 12 00:29:28.236585 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.225745    2567 manager.go:337] "Starting Device Plugin manager"
Sep 12 00:29:28.236585 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.225786    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 12 00:29:28.236585 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.225809    2567 server.go:87] "Starting device plugin registration server"
Sep 12 00:29:28.236585 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.226640    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 12 00:29:28.236585 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.226665    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 12 00:29:28.236585 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.226809    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 12 00:29:28.236585 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.226924    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 12 00:29:28.236585 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.226932    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 12 00:29:28.236585 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.227485    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 12 00:29:28.236585 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.227538    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-132.ec2.internal\" not found"
Sep 12 00:29:28.265584 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.265539    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 12 00:29:28.266781 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.266766    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 12 00:29:28.266857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.266826    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 12 00:29:28.266893 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.266859    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 12 00:29:28.266997 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.266967    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 12 00:29:28.268930 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.268906    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:29:28.327834 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.327745    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:29:28.328908 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.328890    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:29:28.329004 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.328923    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:29:28.329004 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.328935    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:29:28.329004 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.328960    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.336021 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.336003    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.336103 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.336026    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-132.ec2.internal\": node \"ip-10-0-130-132.ec2.internal\" not found"
Sep 12 00:29:28.349246 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.349218    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found"
Sep 12 00:29:28.367790 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.367548    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal"]
Sep 12 00:29:28.367928 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.367857    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:29:28.369342 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.369327    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:29:28.369443 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.369360    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:29:28.369443 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.369378    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:29:28.370830 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.370815    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:29:28.370956 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.370935    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.370998 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.370983    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:29:28.371606 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.371589    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:29:28.371718 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.371619    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:29:28.371718 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.371639    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:29:28.371718 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.371596    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:29:28.371718 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.371692    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:29:28.371718 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.371706    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:29:28.372893 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.372872    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.372893 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.372896    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:29:28.373529 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.373515    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:29:28.373621 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.373541    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:29:28.373621 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.373553    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:29:28.408482 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.408456    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-132.ec2.internal\" not found" node="ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.413366 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.413350    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-132.ec2.internal\" not found" node="ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.449368 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.449340    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found"
Sep 12 00:29:28.465880 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.465855    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27e2d2995fe2c1dca3b74b7652c936c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal\" (UID: \"27e2d2995fe2c1dca3b74b7652c936c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.465966 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.465885    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e2d2995fe2c1dca3b74b7652c936c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal\" (UID: \"27e2d2995fe2c1dca3b74b7652c936c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.465966 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.465902    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eddb3576bde467541695f2fe0d5bc466-config\") pod \"kube-apiserver-proxy-ip-10-0-130-132.ec2.internal\" (UID: \"eddb3576bde467541695f2fe0d5bc466\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.550019 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.549955    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found"
Sep 12 00:29:28.568005 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.566687    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27e2d2995fe2c1dca3b74b7652c936c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal\" (UID: \"27e2d2995fe2c1dca3b74b7652c936c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.568005 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.566756    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e2d2995fe2c1dca3b74b7652c936c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal\" (UID: \"27e2d2995fe2c1dca3b74b7652c936c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.568005 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.566787    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eddb3576bde467541695f2fe0d5bc466-config\") pod \"kube-apiserver-proxy-ip-10-0-130-132.ec2.internal\" (UID: \"eddb3576bde467541695f2fe0d5bc466\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.568005 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.566834    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eddb3576bde467541695f2fe0d5bc466-config\") pod \"kube-apiserver-proxy-ip-10-0-130-132.ec2.internal\" (UID: \"eddb3576bde467541695f2fe0d5bc466\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.568005 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.566910    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27e2d2995fe2c1dca3b74b7652c936c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal\" (UID: \"27e2d2995fe2c1dca3b74b7652c936c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.568005 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.566950    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e2d2995fe2c1dca3b74b7652c936c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal\" (UID: \"27e2d2995fe2c1dca3b74b7652c936c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.650769 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.650735    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found"
Sep 12 00:29:28.711186 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.711145    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.715908 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.715891    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:28.751067 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.751030    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found"
Sep 12 00:29:28.851539 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.851505    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found"
Sep 12 00:29:28.951956 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:28.951867    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-132.ec2.internal\" not found"
Sep 12 00:29:28.976776 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:28.976749    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:29:29.064552 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.064520    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:29.072836 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.072815    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 00:29:29.074932 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.074916    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal"
Sep 12 00:29:29.082483 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.082464    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 00:29:29.098651 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.098620    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 12 00:29:29.098797 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.098777    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 12 00:29:29.098844 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.098814    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 12 00:29:29.140912 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.140880    2567 apiserver.go:52] "Watching apiserver"
Sep 12 00:29:29.156909 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.156803    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 12 00:29:29.158539 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.158510    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v","openshift-image-registry/node-ca-7pvhz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal","openshift-multus/multus-additional-cni-plugins-v88kn","openshift-multus/multus-rkhbc","openshift-multus/network-metrics-daemon-qvqwv","openshift-network-diagnostics/network-check-target-jlqxr","kube-system/global-pull-secret-syncer-q9l42","kube-system/konnectivity-agent-zrcf9","openshift-cluster-node-tuning-operator/tuned-tp9d2","openshift-dns/node-resolver-h688x","openshift-network-operator/iptables-alerter-2zgfm","openshift-ovn-kubernetes/ovnkube-node-fh7jf"]
Sep 12 00:29:29.159954 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.159940    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zrcf9"
Sep 12 00:29:29.161373 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.161353    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2zgfm"
Sep 12 00:29:29.161946 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.161928    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-65822\""
Sep 12 00:29:29.162197 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.162182    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 12 00:29:29.162283 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.162221    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 12 00:29:29.162456 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.162437    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7pvhz"
Sep 12 00:29:29.163403 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.163388    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 12 00:29:29.163491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.163441    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.163966 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.163951    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:29:29.164071 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.164018    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-11 00:24:28 +0000 UTC" deadline="2028-02-19 05:26:09.448268855 +0000 UTC"
Sep 12 00:29:29.164124 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.164074    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12604h56m40.28419768s"
Sep 12 00:29:29.164170 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.164125    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 12 00:29:29.164260 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.164242    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 12 00:29:29.164521 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.164505    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.164996 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.164980    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 12 00:29:29.165093 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.164996    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 12 00:29:29.165093 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.165077    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t7zbh\""
Sep 12 00:29:29.165204 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.165191    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 12 00:29:29.165238 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.165210    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6mc6h\""
Sep 12 00:29:29.165632 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.165618    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 12 00:29:29.165764 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.165636    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 12 00:29:29.165764 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.165696    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.166052 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.166036    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rkjms\""
Sep 12 00:29:29.166116 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.166068    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 12 00:29:29.166169 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.166130    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 12 00:29:29.166216 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.166177    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 12 00:29:29.166216 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.166199    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 12 00:29:29.166731 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.166715    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 12 00:29:29.166787 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.166743    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 12 00:29:29.166824 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.166744    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 12 00:29:29.167175 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.167142    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 12 00:29:29.167175 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.167160    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 12 00:29:29.167306 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.167183    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mf2s8\""
Sep 12 00:29:29.167306 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.167219    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:29.167306 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.167292    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:29.167623 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.167609    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 12 00:29:29.167681 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.167611    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xgzzl\""
Sep 12 00:29:29.168895 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.168876    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-slash\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.168968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.168908    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.168968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.168931    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e-konnectivity-ca\") pod \"konnectivity-agent-zrcf9\" (UID: \"34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e\") " pod="kube-system/konnectivity-agent-zrcf9"
Sep 12 00:29:29.168968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.168951    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:29.169071 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.169001    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:29.169071 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.168951    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-var-lib-openvswitch\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.169071 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169044    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-run-ovn\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.169071 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169060    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-node-log\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.169201 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169074    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c994x\" (UniqueName: \"kubernetes.io/projected/99f21261-c1a7-40c2-880a-dd622834a861-kube-api-access-c994x\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.169201 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169089    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e01af40-0cf2-43d6-83c5-12215a4c0a4f-serviceca\") pod \"node-ca-7pvhz\" (UID: \"0e01af40-0cf2-43d6-83c5-12215a4c0a4f\") " pod="openshift-image-registry/node-ca-7pvhz"
Sep 12 00:29:29.169201 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169102    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-kubelet\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.169545 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169516    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-run-netns\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.169669 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169650    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99f21261-c1a7-40c2-880a-dd622834a861-env-overrides\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.169739 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169685    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-cnibin\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.169739 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169720    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-cni-binary-copy\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.169829 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169754    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.169829 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169797    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e-agent-certs\") pod \"konnectivity-agent-zrcf9\" (UID: \"34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e\") " pod="kube-system/konnectivity-agent-zrcf9"
Sep 12 00:29:29.169915 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169840    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e01af40-0cf2-43d6-83c5-12215a4c0a4f-host\") pod \"node-ca-7pvhz\" (UID: \"0e01af40-0cf2-43d6-83c5-12215a4c0a4f\") " pod="openshift-image-registry/node-ca-7pvhz"
Sep 12 00:29:29.169915 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169887    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tb2s\" (UniqueName: \"kubernetes.io/projected/0e01af40-0cf2-43d6-83c5-12215a4c0a4f-kube-api-access-7tb2s\") pod \"node-ca-7pvhz\" (UID: \"0e01af40-0cf2-43d6-83c5-12215a4c0a4f\") " pod="openshift-image-registry/node-ca-7pvhz"
Sep 12 00:29:29.169995 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169926    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-run-systemd\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.169995 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169970    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99f21261-c1a7-40c2-880a-dd622834a861-ovnkube-config\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.170073 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.169999    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99f21261-c1a7-40c2-880a-dd622834a861-ovn-node-metrics-cert\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.170073 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170034    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.170073 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170065    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/620cca97-55a4-47db-9f3c-ca8912a77cba-iptables-alerter-script\") pod \"iptables-alerter-2zgfm\" (UID: \"620cca97-55a4-47db-9f3c-ca8912a77cba\") " pod="openshift-network-operator/iptables-alerter-2zgfm"
Sep 12 00:29:29.170197 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170093    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-run-ovn-kubernetes\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.170197 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170117    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.170197 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170147    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-log-socket\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.170197 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170174    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-cni-bin\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.170359 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170201    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-systemd-units\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.170359 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170253    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-etc-openvswitch\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.170359 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170306    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-cni-netd\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.170359 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170341    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99f21261-c1a7-40c2-880a-dd622834a861-ovnkube-script-lib\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.170516 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170363    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5br6f\" (UniqueName: \"kubernetes.io/projected/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-kube-api-access-5br6f\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.170516 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170405    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/620cca97-55a4-47db-9f3c-ca8912a77cba-host-slash\") pod \"iptables-alerter-2zgfm\" (UID: \"620cca97-55a4-47db-9f3c-ca8912a77cba\") " pod="openshift-network-operator/iptables-alerter-2zgfm"
Sep 12 00:29:29.170516 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170426    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjz64\" (UniqueName: \"kubernetes.io/projected/620cca97-55a4-47db-9f3c-ca8912a77cba-kube-api-access-tjz64\") pod \"iptables-alerter-2zgfm\" (UID: \"620cca97-55a4-47db-9f3c-ca8912a77cba\") " pod="openshift-network-operator/iptables-alerter-2zgfm"
Sep 12 00:29:29.170516 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170452    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-run-openvswitch\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.170516 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.170478    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-system-cni-dir\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.171650 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.171632    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.172502 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.172484    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.173815 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.173801    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h688x"
Sep 12 00:29:29.174162 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.174142    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 12 00:29:29.174228 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.174160    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 12 00:29:29.174228 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.174217    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 12 00:29:29.174295 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.174276    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m9rdj\""
Sep 12 00:29:29.174476 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.174454    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 00:29:29.174554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.174540    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:29:29.174640 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.174626    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 12 00:29:29.174801 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.174790    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6bw57\""
Sep 12 00:29:29.175083 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.175070    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:29.175138 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.175124    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:29.175773 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.175757    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 12 00:29:29.175832 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.175818    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hpjr4\""
Sep 12 00:29:29.176008 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.175996    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 12 00:29:29.194422 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.194258    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cx8sd"
Sep 12 00:29:29.202325 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.202271    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cx8sd"
Sep 12 00:29:29.244212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.244187    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:29:29.261424 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:29.261398    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddb3576bde467541695f2fe0d5bc466.slice/crio-026856680a615916aa347c272dcdedd18372c8661d0dfb565a614b42996c5d7d WatchSource:0}: Error finding container 026856680a615916aa347c272dcdedd18372c8661d0dfb565a614b42996c5d7d: Status 404 returned error can't find the container with id 026856680a615916aa347c272dcdedd18372c8661d0dfb565a614b42996c5d7d
Sep 12 00:29:29.261689 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:29.261663    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e2d2995fe2c1dca3b74b7652c936c9.slice/crio-0a5396b09288a7cf15e7f54f9308361f5d52179e19a62e7797caa7b95a6911b9 WatchSource:0}: Error finding container 0a5396b09288a7cf15e7f54f9308361f5d52179e19a62e7797caa7b95a6911b9: Status 404 returned error can't find the container with id 0a5396b09288a7cf15e7f54f9308361f5d52179e19a62e7797caa7b95a6911b9
Sep 12 00:29:29.265311 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.265298    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 12 00:29:29.265503 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.265486    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:29:29.270611 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.270548    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" event={"ID":"27e2d2995fe2c1dca3b74b7652c936c9","Type":"ContainerStarted","Data":"0a5396b09288a7cf15e7f54f9308361f5d52179e19a62e7797caa7b95a6911b9"}
Sep 12 00:29:29.270724 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.270702    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e-konnectivity-ca\") pod \"konnectivity-agent-zrcf9\" (UID: \"34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e\") " pod="kube-system/konnectivity-agent-zrcf9"
Sep 12 00:29:29.270779 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.270740    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4967c7dd-797f-4225-a54d-6b83c6827183-kubelet-config\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:29.270832 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.270798    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-socket-dir\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.270927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.270845    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjjl2\" (UniqueName: \"kubernetes.io/projected/661a55c6-221b-4cb8-b50b-ff96524e9ac8-kube-api-access-rjjl2\") pod \"node-resolver-h688x\" (UID: \"661a55c6-221b-4cb8-b50b-ff96524e9ac8\") " pod="openshift-dns/node-resolver-h688x"
Sep 12 00:29:29.270927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.270871    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-system-cni-dir\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.270927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.270916    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-multus-cni-dir\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.271047 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.270972    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-var-lib-openvswitch\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.271047 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271000    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-node-log\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.271047 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271026    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c994x\" (UniqueName: \"kubernetes.io/projected/99f21261-c1a7-40c2-880a-dd622834a861-kube-api-access-c994x\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.271176 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271053    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/661a55c6-221b-4cb8-b50b-ff96524e9ac8-hosts-file\") pod \"node-resolver-h688x\" (UID: \"661a55c6-221b-4cb8-b50b-ff96524e9ac8\") " pod="openshift-dns/node-resolver-h688x"
Sep 12 00:29:29.271176 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271082    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-node-log\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.271176 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271102    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-var-lib-openvswitch\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.271176 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271087    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-run-multus-certs\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.271349 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271181    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-sysconfig\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.271349 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271207    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/831bac43-1246-48fe-84d6-ac43968b5852-tmp\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.271349 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271232    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-kubelet\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.271349 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271270    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99f21261-c1a7-40c2-880a-dd622834a861-env-overrides\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.271349 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271294    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-cnibin\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.271349 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271318    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.271349 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271343    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-run-netns\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.271680 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271361    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e-konnectivity-ca\") pod \"konnectivity-agent-zrcf9\" (UID: \"34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e\") " pod="kube-system/konnectivity-agent-zrcf9"
Sep 12 00:29:29.271680 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271380    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-kubelet\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.271680 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271494    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn7hp\" (UniqueName: \"kubernetes.io/projected/831bac43-1246-48fe-84d6-ac43968b5852-kube-api-access-gn7hp\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.271680 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271595    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e01af40-0cf2-43d6-83c5-12215a4c0a4f-host\") pod \"node-ca-7pvhz\" (UID: \"0e01af40-0cf2-43d6-83c5-12215a4c0a4f\") " pod="openshift-image-registry/node-ca-7pvhz"
Sep 12 00:29:29.271680 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271627    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tb2s\" (UniqueName: \"kubernetes.io/projected/0e01af40-0cf2-43d6-83c5-12215a4c0a4f-kube-api-access-7tb2s\") pod \"node-ca-7pvhz\" (UID: \"0e01af40-0cf2-43d6-83c5-12215a4c0a4f\") " pod="openshift-image-registry/node-ca-7pvhz"
Sep 12 00:29:29.271882 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271714    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/620cca97-55a4-47db-9f3c-ca8912a77cba-iptables-alerter-script\") pod \"iptables-alerter-2zgfm\" (UID: \"620cca97-55a4-47db-9f3c-ca8912a77cba\") " pod="openshift-network-operator/iptables-alerter-2zgfm"
Sep 12 00:29:29.271882 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271743    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4967c7dd-797f-4225-a54d-6b83c6827183-dbus\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:29.271882 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271782    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-run-k8s-cni-cncf-io\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.271882 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.271804    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-etc-kubernetes\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.272434 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272174    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:29.272434 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272254    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.272434 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272310    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-run-ovn-kubernetes\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.272434 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272325    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-cnibin\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.272434 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272329    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e01af40-0cf2-43d6-83c5-12215a4c0a4f-host\") pod \"node-ca-7pvhz\" (UID: \"0e01af40-0cf2-43d6-83c5-12215a4c0a4f\") " pod="openshift-image-registry/node-ca-7pvhz"
Sep 12 00:29:29.272434 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272350    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.272434 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272385    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-run-ovn-kubernetes\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.272795 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272606    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.272795 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272650    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-multus-socket-dir-parent\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.272795 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272689    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-multus-conf-dir\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.272933 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272732    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-log-socket\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.272933 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272855    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-log-socket\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.272933 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272900    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-registration-dir\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.272933 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272905    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/620cca97-55a4-47db-9f3c-ca8912a77cba-iptables-alerter-script\") pod \"iptables-alerter-2zgfm\" (UID: \"620cca97-55a4-47db-9f3c-ca8912a77cba\") " pod="openshift-network-operator/iptables-alerter-2zgfm"
Sep 12 00:29:29.273102 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272951    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-var-lib-kubelet\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.273102 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.272996    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-cni-netd\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.273102 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273050    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-cni-netd\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.273102 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273056    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5br6f\" (UniqueName: \"kubernetes.io/projected/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-kube-api-access-5br6f\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.273262 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273244    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/620cca97-55a4-47db-9f3c-ca8912a77cba-host-slash\") pod \"iptables-alerter-2zgfm\" (UID: \"620cca97-55a4-47db-9f3c-ca8912a77cba\") " pod="openshift-network-operator/iptables-alerter-2zgfm"
Sep 12 00:29:29.273308 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273291    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjz64\" (UniqueName: \"kubernetes.io/projected/620cca97-55a4-47db-9f3c-ca8912a77cba-kube-api-access-tjz64\") pod \"iptables-alerter-2zgfm\" (UID: \"620cca97-55a4-47db-9f3c-ca8912a77cba\") " pod="openshift-network-operator/iptables-alerter-2zgfm"
Sep 12 00:29:29.273355 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273323    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-run\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.273355 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273349    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-host\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.273443 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273356    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/620cca97-55a4-47db-9f3c-ca8912a77cba-host-slash\") pod \"iptables-alerter-2zgfm\" (UID: \"620cca97-55a4-47db-9f3c-ca8912a77cba\") " pod="openshift-network-operator/iptables-alerter-2zgfm"
Sep 12 00:29:29.273443 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273375    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-system-cni-dir\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.273443 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273420    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-system-cni-dir\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.273443 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273426    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-hostroot\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.273644 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273461    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-sysctl-conf\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.273644 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273489    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-var-lib-kubelet\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.273644 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273521    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.273644 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273562    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.273644 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273621    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92e821e2-a347-4e2b-9b5e-9a90a815e210-multus-daemon-config\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.273822 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273794    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-sys\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.273822 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273792    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal" event={"ID":"eddb3576bde467541695f2fe0d5bc466","Type":"ContainerStarted","Data":"026856680a615916aa347c272dcdedd18372c8661d0dfb565a614b42996c5d7d"}
Sep 12 00:29:29.273951 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273921    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99f21261-c1a7-40c2-880a-dd622834a861-env-overrides\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.274018 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/831bac43-1246-48fe-84d6-ac43968b5852-etc-tuned\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.274018 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.273988    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-run-ovn\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.274212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274179    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:29.274254 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274228    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xcp5\" (UniqueName: \"kubernetes.io/projected/0a3b6564-fae6-476b-8e7d-003d4beaf870-kube-api-access-9xcp5\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:29.274299 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274262    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e01af40-0cf2-43d6-83c5-12215a4c0a4f-serviceca\") pod \"node-ca-7pvhz\" (UID: \"0e01af40-0cf2-43d6-83c5-12215a4c0a4f\") " pod="openshift-image-registry/node-ca-7pvhz"
Sep 12 00:29:29.274347 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274320    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-run-netns\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.274390 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274347    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-cni-binary-copy\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.274390 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274374    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e-agent-certs\") pod \"konnectivity-agent-zrcf9\" (UID: \"34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e\") " pod="kube-system/konnectivity-agent-zrcf9"
Sep 12 00:29:29.274474 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7plg\" (UniqueName: \"kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg\") pod \"network-check-target-jlqxr\" (UID: \"e236a93e-7f67-4f7b-bc99-96072320715a\") " pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:29.274474 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274423    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-run-systemd\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.274474 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274450    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99f21261-c1a7-40c2-880a-dd622834a861-ovnkube-config\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.274669 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274604    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99f21261-c1a7-40c2-880a-dd622834a861-ovn-node-metrics-cert\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.274669 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274640    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.274754 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274668    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-device-dir\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.274754 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274688    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-kubernetes\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.274754 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274718    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-sys-fs\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.274754 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274743    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-cnibin\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.274913 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274769    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92e821e2-a347-4e2b-9b5e-9a90a815e210-cni-binary-copy\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.274913 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274795    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvhr\" (UniqueName: \"kubernetes.io/projected/92e821e2-a347-4e2b-9b5e-9a90a815e210-kube-api-access-jdvhr\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.274913 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274819    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-cni-bin\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.274913 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274847    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/661a55c6-221b-4cb8-b50b-ff96524e9ac8-tmp-dir\") pod \"node-resolver-h688x\" (UID: \"661a55c6-221b-4cb8-b50b-ff96524e9ac8\") " pod="openshift-dns/node-resolver-h688x"
Sep 12 00:29:29.274913 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274874    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-var-lib-cni-bin\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.274913 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.274900    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-var-lib-cni-multus\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.275153 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275009    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-sysctl-d\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.275153 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275034    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-systemd\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.275153 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275063    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-lib-modules\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.275153 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275100    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-systemd-units\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.275153 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275135    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-etc-openvswitch\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.275359 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275183    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-etc-openvswitch\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.275359 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275266    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-run-ovn\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.275359 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275277    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.275359 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275312    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99f21261-c1a7-40c2-880a-dd622834a861-ovnkube-script-lib\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.275540 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275377    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.275540 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275427    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkp2w\" (UniqueName: \"kubernetes.io/projected/80f6ccea-903a-4044-b22c-25ddc2cf19d9-kube-api-access-pkp2w\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.275540 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275469    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-modprobe-d\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.275540 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275524    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-run-openvswitch\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.275743 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275579    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.275743 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275685    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-run-openvswitch\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.275743 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275709    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e01af40-0cf2-43d6-83c5-12215a4c0a4f-serviceca\") pod \"node-ca-7pvhz\" (UID: \"0e01af40-0cf2-43d6-83c5-12215a4c0a4f\") " pod="openshift-image-registry/node-ca-7pvhz"
Sep 12 00:29:29.275743 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275729    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-run-netns\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.275920 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275780    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-systemd-units\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.275920 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275807    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99f21261-c1a7-40c2-880a-dd622834a861-ovnkube-config\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.275920 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275868    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-cni-bin\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.275920 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.275910    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-run-systemd\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.276291 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.276270    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99f21261-c1a7-40c2-880a-dd622834a861-ovnkube-script-lib\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.276357 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.276305    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-cni-binary-copy\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.276357 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.276329    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 12 00:29:29.276812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.276791    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-slash\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.276888 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.276837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99f21261-c1a7-40c2-880a-dd622834a861-host-slash\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.279604 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.279580    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e-agent-certs\") pod \"konnectivity-agent-zrcf9\" (UID: \"34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e\") " pod="kube-system/konnectivity-agent-zrcf9"
Sep 12 00:29:29.279714 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.279604    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99f21261-c1a7-40c2-880a-dd622834a861-ovn-node-metrics-cert\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.279781 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.279736    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c994x\" (UniqueName: \"kubernetes.io/projected/99f21261-c1a7-40c2-880a-dd622834a861-kube-api-access-c994x\") pod \"ovnkube-node-fh7jf\" (UID: \"99f21261-c1a7-40c2-880a-dd622834a861\") " pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.280505 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.280486    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5br6f\" (UniqueName: \"kubernetes.io/projected/ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7-kube-api-access-5br6f\") pod \"multus-additional-cni-plugins-v88kn\" (UID: \"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7\") " pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.280684 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.280667    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tb2s\" (UniqueName: \"kubernetes.io/projected/0e01af40-0cf2-43d6-83c5-12215a4c0a4f-kube-api-access-7tb2s\") pod \"node-ca-7pvhz\" (UID: \"0e01af40-0cf2-43d6-83c5-12215a4c0a4f\") " pod="openshift-image-registry/node-ca-7pvhz"
Sep 12 00:29:29.285991 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.285968    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjz64\" (UniqueName: \"kubernetes.io/projected/620cca97-55a4-47db-9f3c-ca8912a77cba-kube-api-access-tjz64\") pod \"iptables-alerter-2zgfm\" (UID: \"620cca97-55a4-47db-9f3c-ca8912a77cba\") " pod="openshift-network-operator/iptables-alerter-2zgfm"
Sep 12 00:29:29.377259 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377220    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-registration-dir\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.377259 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377253    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-var-lib-kubelet\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377274    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-run\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377290    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-host\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-hostroot\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377332    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-sysctl-conf\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377349    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-var-lib-kubelet\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377360    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-run\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377360    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-registration-dir\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377367    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-var-lib-kubelet\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377401    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-hostroot\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377407    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-var-lib-kubelet\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377406    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-host\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377462    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92e821e2-a347-4e2b-9b5e-9a90a815e210-multus-daemon-config\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377467    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377481    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-sysctl-conf\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.377486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377484    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-sys\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377505    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/831bac43-1246-48fe-84d6-ac43968b5852-etc-tuned\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377522    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377532    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-sys\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377562    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xcp5\" (UniqueName: \"kubernetes.io/projected/0a3b6564-fae6-476b-8e7d-003d4beaf870-kube-api-access-9xcp5\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377606    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7plg\" (UniqueName: \"kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg\") pod \"network-check-target-jlqxr\" (UID: \"e236a93e-7f67-4f7b-bc99-96072320715a\") " pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.377627    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377672    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-device-dir\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377630    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-device-dir\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.377714    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret podName:4967c7dd-797f-4225-a54d-6b83c6827183 nodeName:}" failed. No retries permitted until 2026-09-12 00:29:29.877685276 +0000 UTC m=+2.081686077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret") pod "global-pull-secret-syncer-q9l42" (UID: "4967c7dd-797f-4225-a54d-6b83c6827183") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377761    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-kubernetes\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377786    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-sys-fs\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377808    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-cnibin\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377844    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92e821e2-a347-4e2b-9b5e-9a90a815e210-cni-binary-copy\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377858    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-sys-fs\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377867    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvhr\" (UniqueName: \"kubernetes.io/projected/92e821e2-a347-4e2b-9b5e-9a90a815e210-kube-api-access-jdvhr\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377894    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-kubernetes\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.378243 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377902    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-cnibin\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377923    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/661a55c6-221b-4cb8-b50b-ff96524e9ac8-tmp-dir\") pod \"node-resolver-h688x\" (UID: \"661a55c6-221b-4cb8-b50b-ff96524e9ac8\") " pod="openshift-dns/node-resolver-h688x"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.377965    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-var-lib-cni-bin\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378001    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-var-lib-cni-multus\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378022    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-sysctl-d\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378044    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-systemd\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378049    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92e821e2-a347-4e2b-9b5e-9a90a815e210-multus-daemon-config\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378065    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-lib-modules\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378070    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-var-lib-cni-multus\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378064    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-var-lib-cni-bin\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378092    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378128    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-systemd\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378130    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkp2w\" (UniqueName: \"kubernetes.io/projected/80f6ccea-903a-4044-b22c-25ddc2cf19d9-kube-api-access-pkp2w\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378167    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-modprobe-d\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378190    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-lib-modules\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378197    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4967c7dd-797f-4225-a54d-6b83c6827183-kubelet-config\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378235    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-sysctl-d\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378238    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/661a55c6-221b-4cb8-b50b-ff96524e9ac8-tmp-dir\") pod \"node-resolver-h688x\" (UID: \"661a55c6-221b-4cb8-b50b-ff96524e9ac8\") " pod="openshift-dns/node-resolver-h688x"
Sep 12 00:29:29.378979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378239    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-socket-dir\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378244    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378279    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4967c7dd-797f-4225-a54d-6b83c6827183-kubelet-config\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378309    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-modprobe-d\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378330    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjjl2\" (UniqueName: \"kubernetes.io/projected/661a55c6-221b-4cb8-b50b-ff96524e9ac8-kube-api-access-rjjl2\") pod \"node-resolver-h688x\" (UID: \"661a55c6-221b-4cb8-b50b-ff96524e9ac8\") " pod="openshift-dns/node-resolver-h688x"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378346    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92e821e2-a347-4e2b-9b5e-9a90a815e210-cni-binary-copy\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378362    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-system-cni-dir\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378377    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-multus-cni-dir\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378382    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80f6ccea-903a-4044-b22c-25ddc2cf19d9-socket-dir\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378398    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/661a55c6-221b-4cb8-b50b-ff96524e9ac8-hosts-file\") pod \"node-resolver-h688x\" (UID: \"661a55c6-221b-4cb8-b50b-ff96524e9ac8\") " pod="openshift-dns/node-resolver-h688x"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378409    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-system-cni-dir\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378416    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-run-multus-certs\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378435    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-multus-cni-dir\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378441    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-sysconfig\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378439    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-run-multus-certs\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378463    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/831bac43-1246-48fe-84d6-ac43968b5852-tmp\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378469    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/661a55c6-221b-4cb8-b50b-ff96524e9ac8-hosts-file\") pod \"node-resolver-h688x\" (UID: \"661a55c6-221b-4cb8-b50b-ff96524e9ac8\") " pod="openshift-dns/node-resolver-h688x"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378494    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-run-netns\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379433 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378518    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn7hp\" (UniqueName: \"kubernetes.io/projected/831bac43-1246-48fe-84d6-ac43968b5852-kube-api-access-gn7hp\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378525    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-run-netns\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378496    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/831bac43-1246-48fe-84d6-ac43968b5852-etc-sysconfig\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378542    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4967c7dd-797f-4225-a54d-6b83c6827183-dbus\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378698    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4967c7dd-797f-4225-a54d-6b83c6827183-dbus\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378726    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-run-k8s-cni-cncf-io\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378748    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-etc-kubernetes\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378762    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378772    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-host-run-k8s-cni-cncf-io\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378777    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-multus-socket-dir-parent\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378816    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-multus-conf-dir\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378822    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-multus-socket-dir-parent\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378829    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-etc-kubernetes\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.378842    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.378885    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs podName:0a3b6564-fae6-476b-8e7d-003d4beaf870 nodeName:}" failed. No retries permitted until 2026-09-12 00:29:29.87887263 +0000 UTC m=+2.082873432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs") pod "network-metrics-daemon-qvqwv" (UID: "0a3b6564-fae6-476b-8e7d-003d4beaf870") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.378877    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92e821e2-a347-4e2b-9b5e-9a90a815e210-multus-conf-dir\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.379921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.379767    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/831bac43-1246-48fe-84d6-ac43968b5852-etc-tuned\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.380323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.380102    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/831bac43-1246-48fe-84d6-ac43968b5852-tmp\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.384019 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.383998    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:29:29.384019 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.384019    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:29:29.384183 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.384031    2567 projected.go:196] Error preparing data for projected volume kube-api-access-w7plg for pod openshift-network-diagnostics/network-check-target-jlqxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:29:29.384183 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.384090    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg podName:e236a93e-7f67-4f7b-bc99-96072320715a nodeName:}" failed. No retries permitted until 2026-09-12 00:29:29.884077041 +0000 UTC m=+2.088077843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w7plg" (UniqueName: "kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg") pod "network-check-target-jlqxr" (UID: "e236a93e-7f67-4f7b-bc99-96072320715a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:29:29.385622 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.385598    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvhr\" (UniqueName: \"kubernetes.io/projected/92e821e2-a347-4e2b-9b5e-9a90a815e210-kube-api-access-jdvhr\") pod \"multus-rkhbc\" (UID: \"92e821e2-a347-4e2b-9b5e-9a90a815e210\") " pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.385728 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.385598    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjjl2\" (UniqueName: \"kubernetes.io/projected/661a55c6-221b-4cb8-b50b-ff96524e9ac8-kube-api-access-rjjl2\") pod \"node-resolver-h688x\" (UID: \"661a55c6-221b-4cb8-b50b-ff96524e9ac8\") " pod="openshift-dns/node-resolver-h688x"
Sep 12 00:29:29.385728 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.385667    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn7hp\" (UniqueName: \"kubernetes.io/projected/831bac43-1246-48fe-84d6-ac43968b5852-kube-api-access-gn7hp\") pod \"tuned-tp9d2\" (UID: \"831bac43-1246-48fe-84d6-ac43968b5852\") " pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.386168 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.386150    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xcp5\" (UniqueName: \"kubernetes.io/projected/0a3b6564-fae6-476b-8e7d-003d4beaf870-kube-api-access-9xcp5\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:29.386265 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.386250    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkp2w\" (UniqueName: \"kubernetes.io/projected/80f6ccea-903a-4044-b22c-25ddc2cf19d9-kube-api-access-pkp2w\") pod \"aws-ebs-csi-driver-node-lb96v\" (UID: \"80f6ccea-903a-4044-b22c-25ddc2cf19d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.492160 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.492123    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zrcf9"
Sep 12 00:29:29.499841 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:29.499810    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34228a0d_1c7e_49d0_8af7_a9ed2f5ab54e.slice/crio-e7ab8aadc46cbc7ad525d7292fff32784809b89804644cdd19bdcab5cebc6485 WatchSource:0}: Error finding container e7ab8aadc46cbc7ad525d7292fff32784809b89804644cdd19bdcab5cebc6485: Status 404 returned error can't find the container with id e7ab8aadc46cbc7ad525d7292fff32784809b89804644cdd19bdcab5cebc6485
Sep 12 00:29:29.513040 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.513017    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2zgfm"
Sep 12 00:29:29.519900 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:29.519876    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620cca97_55a4_47db_9f3c_ca8912a77cba.slice/crio-56fbd68135891e598c1c35b907d00ef8a1a6908b56984ef6e707bf748b2b7b30 WatchSource:0}: Error finding container 56fbd68135891e598c1c35b907d00ef8a1a6908b56984ef6e707bf748b2b7b30: Status 404 returned error can't find the container with id 56fbd68135891e598c1c35b907d00ef8a1a6908b56984ef6e707bf748b2b7b30
Sep 12 00:29:29.524414 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.524384    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7pvhz"
Sep 12 00:29:29.530837 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:29.530813    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e01af40_0cf2_43d6_83c5_12215a4c0a4f.slice/crio-344b2b481d5724636f9638176ccb70622efe006dc9d2e1b89d33ba5a025a071f WatchSource:0}: Error finding container 344b2b481d5724636f9638176ccb70622efe006dc9d2e1b89d33ba5a025a071f: Status 404 returned error can't find the container with id 344b2b481d5724636f9638176ccb70622efe006dc9d2e1b89d33ba5a025a071f
Sep 12 00:29:29.536292 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.536270    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:29.542559 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:29.542537    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f21261_c1a7_40c2_880a_dd622834a861.slice/crio-f713996479f140f5d0fea4ed4aadb4d244ee33852367ad7e52a157a089f108ac WatchSource:0}: Error finding container f713996479f140f5d0fea4ed4aadb4d244ee33852367ad7e52a157a089f108ac: Status 404 returned error can't find the container with id f713996479f140f5d0fea4ed4aadb4d244ee33852367ad7e52a157a089f108ac
Sep 12 00:29:29.556160 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.556135    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v88kn"
Sep 12 00:29:29.564064 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:29.564039    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4faa49_242a_4201_bd5a_e9e1bcfc1ea7.slice/crio-9ff369c8300d8635ac05e14aa23c4bd5c4ac80e2f5ae9d94d9c346dd0ea9eb2e WatchSource:0}: Error finding container 9ff369c8300d8635ac05e14aa23c4bd5c4ac80e2f5ae9d94d9c346dd0ea9eb2e: Status 404 returned error can't find the container with id 9ff369c8300d8635ac05e14aa23c4bd5c4ac80e2f5ae9d94d9c346dd0ea9eb2e
Sep 12 00:29:29.571442 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.571415    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rkhbc"
Sep 12 00:29:29.579351 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:29.579311    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e821e2_a347_4e2b_9b5e_9a90a815e210.slice/crio-b05f750fb95e8eae0cbad2f7e44d9b840c917cae3deb463c564d1af2cfd8fa74 WatchSource:0}: Error finding container b05f750fb95e8eae0cbad2f7e44d9b840c917cae3deb463c564d1af2cfd8fa74: Status 404 returned error can't find the container with id b05f750fb95e8eae0cbad2f7e44d9b840c917cae3deb463c564d1af2cfd8fa74
Sep 12 00:29:29.580135 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.580116    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v"
Sep 12 00:29:29.588821 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:29.588795    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f6ccea_903a_4044_b22c_25ddc2cf19d9.slice/crio-ad2448456323abd496ea176f5e3559db820e3f582fec3e9422d886bebc262888 WatchSource:0}: Error finding container ad2448456323abd496ea176f5e3559db820e3f582fec3e9422d886bebc262888: Status 404 returned error can't find the container with id ad2448456323abd496ea176f5e3559db820e3f582fec3e9422d886bebc262888
Sep 12 00:29:29.589713 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.589690    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tp9d2"
Sep 12 00:29:29.594422 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.594403    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h688x"
Sep 12 00:29:29.597648 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:29.597615    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831bac43_1246_48fe_84d6_ac43968b5852.slice/crio-33b46584a3b621464d9e6f5e858876d32a7e04876f085be328f24ca13b310d1d WatchSource:0}: Error finding container 33b46584a3b621464d9e6f5e858876d32a7e04876f085be328f24ca13b310d1d: Status 404 returned error can't find the container with id 33b46584a3b621464d9e6f5e858876d32a7e04876f085be328f24ca13b310d1d
Sep 12 00:29:29.603870 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:29:29.603838    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661a55c6_221b_4cb8_b50b_ff96524e9ac8.slice/crio-5d20994af332e698f48ec98213e29886a62292d5f4aebbc86bc6cdefb9e52a89 WatchSource:0}: Error finding container 5d20994af332e698f48ec98213e29886a62292d5f4aebbc86bc6cdefb9e52a89: Status 404 returned error can't find the container with id 5d20994af332e698f48ec98213e29886a62292d5f4aebbc86bc6cdefb9e52a89
Sep 12 00:29:29.607146 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.607123    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:29:29.881804 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.881712    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:29.881804 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.881786    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:29.882029 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.881989    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:29:29.882090 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.882052    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret podName:4967c7dd-797f-4225-a54d-6b83c6827183 nodeName:}" failed. No retries permitted until 2026-09-12 00:29:30.882034122 +0000 UTC m=+3.086034934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret") pod "global-pull-secret-syncer-q9l42" (UID: "4967c7dd-797f-4225-a54d-6b83c6827183") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:29:29.882667 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.882505    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:29:29.882667 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.882642    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs podName:0a3b6564-fae6-476b-8e7d-003d4beaf870 nodeName:}" failed. No retries permitted until 2026-09-12 00:29:30.882625226 +0000 UTC m=+3.086626028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs") pod "network-metrics-daemon-qvqwv" (UID: "0a3b6564-fae6-476b-8e7d-003d4beaf870") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:29:29.982553 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:29.982514    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7plg\" (UniqueName: \"kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg\") pod \"network-check-target-jlqxr\" (UID: \"e236a93e-7f67-4f7b-bc99-96072320715a\") " pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:29.982784 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.982766    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:29:29.982784 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.982795    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:29:29.982784 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.982808    2567 projected.go:196] Error preparing data for projected volume kube-api-access-w7plg for pod openshift-network-diagnostics/network-check-target-jlqxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:29:29.983034 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:29.982867    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg podName:e236a93e-7f67-4f7b-bc99-96072320715a nodeName:}" failed. No retries permitted until 2026-09-12 00:29:30.98284753 +0000 UTC m=+3.186848344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w7plg" (UniqueName: "kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg") pod "network-check-target-jlqxr" (UID: "e236a93e-7f67-4f7b-bc99-96072320715a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:29:30.204348 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.204255    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 00:24:29 +0000 UTC" deadline="2028-02-26 05:49:14.802458627 +0000 UTC"
Sep 12 00:29:30.204348 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.204298    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12773h19m44.598164853s"
Sep 12 00:29:30.311474 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.311331    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h688x" event={"ID":"661a55c6-221b-4cb8-b50b-ff96524e9ac8","Type":"ContainerStarted","Data":"5d20994af332e698f48ec98213e29886a62292d5f4aebbc86bc6cdefb9e52a89"}
Sep 12 00:29:30.318886 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.318204    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tp9d2" event={"ID":"831bac43-1246-48fe-84d6-ac43968b5852","Type":"ContainerStarted","Data":"33b46584a3b621464d9e6f5e858876d32a7e04876f085be328f24ca13b310d1d"}
Sep 12 00:29:30.337980 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.337851    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v" event={"ID":"80f6ccea-903a-4044-b22c-25ddc2cf19d9","Type":"ContainerStarted","Data":"ad2448456323abd496ea176f5e3559db820e3f582fec3e9422d886bebc262888"}
Sep 12 00:29:30.348020 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.347894    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v88kn" event={"ID":"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7","Type":"ContainerStarted","Data":"9ff369c8300d8635ac05e14aa23c4bd5c4ac80e2f5ae9d94d9c346dd0ea9eb2e"}
Sep 12 00:29:30.364473 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.364432    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf" event={"ID":"99f21261-c1a7-40c2-880a-dd622834a861","Type":"ContainerStarted","Data":"f713996479f140f5d0fea4ed4aadb4d244ee33852367ad7e52a157a089f108ac"}
Sep 12 00:29:30.372713 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.372670    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2zgfm" event={"ID":"620cca97-55a4-47db-9f3c-ca8912a77cba","Type":"ContainerStarted","Data":"56fbd68135891e598c1c35b907d00ef8a1a6908b56984ef6e707bf748b2b7b30"}
Sep 12 00:29:30.386588 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.386538    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rkhbc" event={"ID":"92e821e2-a347-4e2b-9b5e-9a90a815e210","Type":"ContainerStarted","Data":"b05f750fb95e8eae0cbad2f7e44d9b840c917cae3deb463c564d1af2cfd8fa74"}
Sep 12 00:29:30.390286 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.390247    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7pvhz" event={"ID":"0e01af40-0cf2-43d6-83c5-12215a4c0a4f","Type":"ContainerStarted","Data":"344b2b481d5724636f9638176ccb70622efe006dc9d2e1b89d33ba5a025a071f"}
Sep 12 00:29:30.400090 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.400051    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zrcf9" event={"ID":"34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e","Type":"ContainerStarted","Data":"e7ab8aadc46cbc7ad525d7292fff32784809b89804644cdd19bdcab5cebc6485"}
Sep 12 00:29:30.602583 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.602537    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:29:30.895004 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.892733    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:30.895004 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.892833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:30.895004 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:30.892978    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:29:30.895004 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:30.893042    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs podName:0a3b6564-fae6-476b-8e7d-003d4beaf870 nodeName:}" failed. No retries permitted until 2026-09-12 00:29:32.893024084 +0000 UTC m=+5.097024906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs") pod "network-metrics-daemon-qvqwv" (UID: "0a3b6564-fae6-476b-8e7d-003d4beaf870") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:29:30.895004 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:30.893485    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:29:30.895004 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:30.893538    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret podName:4967c7dd-797f-4225-a54d-6b83c6827183 nodeName:}" failed. No retries permitted until 2026-09-12 00:29:32.893523622 +0000 UTC m=+5.097524438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret") pod "global-pull-secret-syncer-q9l42" (UID: "4967c7dd-797f-4225-a54d-6b83c6827183") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:29:30.993855 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:30.993814    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7plg\" (UniqueName: \"kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg\") pod \"network-check-target-jlqxr\" (UID: \"e236a93e-7f67-4f7b-bc99-96072320715a\") " pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:30.994064 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:30.994046    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:29:30.994130 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:30.994076    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:29:30.994130 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:30.994089    2567 projected.go:196] Error preparing data for projected volume kube-api-access-w7plg for pod openshift-network-diagnostics/network-check-target-jlqxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:29:30.994229 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:30.994154    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg podName:e236a93e-7f67-4f7b-bc99-96072320715a nodeName:}" failed. No retries permitted until 2026-09-12 00:29:32.994134072 +0000 UTC m=+5.198134873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w7plg" (UniqueName: "kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg") pod "network-check-target-jlqxr" (UID: "e236a93e-7f67-4f7b-bc99-96072320715a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:29:31.205426 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:31.205306    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 00:24:29 +0000 UTC" deadline="2028-06-26 11:38:20.272460134 +0000 UTC"
Sep 12 00:29:31.205426 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:31.205369    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15683h8m49.067094884s"
Sep 12 00:29:31.267958 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:31.267924    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:31.268144 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:31.268050    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:31.268507 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:31.268484    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:31.268673 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:31.268608    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:31.268739 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:31.268678    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:31.268795 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:31.268751    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:31.603590 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:31.603373    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:29:32.912887 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:32.912126    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:32.912887 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:32.912205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:32.912887 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:32.912309    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:29:32.912887 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:32.912372    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs podName:0a3b6564-fae6-476b-8e7d-003d4beaf870 nodeName:}" failed. No retries permitted until 2026-09-12 00:29:36.912351314 +0000 UTC m=+9.116352127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs") pod "network-metrics-daemon-qvqwv" (UID: "0a3b6564-fae6-476b-8e7d-003d4beaf870") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:29:32.912887 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:32.912795    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:29:32.912887 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:32.912850    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret podName:4967c7dd-797f-4225-a54d-6b83c6827183 nodeName:}" failed. No retries permitted until 2026-09-12 00:29:36.912834194 +0000 UTC m=+9.116835016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret") pod "global-pull-secret-syncer-q9l42" (UID: "4967c7dd-797f-4225-a54d-6b83c6827183") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:29:33.013533 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:33.013390    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7plg\" (UniqueName: \"kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg\") pod \"network-check-target-jlqxr\" (UID: \"e236a93e-7f67-4f7b-bc99-96072320715a\") " pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:33.013771 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:33.013583    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:29:33.013771 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:33.013609    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:29:33.013771 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:33.013621    2567 projected.go:196] Error preparing data for projected volume kube-api-access-w7plg for pod openshift-network-diagnostics/network-check-target-jlqxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:29:33.013771 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:33.013692    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg podName:e236a93e-7f67-4f7b-bc99-96072320715a nodeName:}" failed. No retries permitted until 2026-09-12 00:29:37.013673084 +0000 UTC m=+9.217673888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w7plg" (UniqueName: "kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg") pod "network-check-target-jlqxr" (UID: "e236a93e-7f67-4f7b-bc99-96072320715a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:29:33.267788 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:33.267745    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:33.267978 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:33.267898    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:33.268063 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:33.268013    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:33.268063 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:33.267897    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:33.268172 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:33.268075    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:33.268172 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:33.268159    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:35.267612 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:35.267300    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:35.267612 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:35.267431    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:35.267612 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:35.267452    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:35.267612 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:35.267563    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:35.268215 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:35.267633    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:35.268215 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:35.267710    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:36.948007 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:36.947792    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:36.948007 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:36.947865    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:36.948007 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:36.947984    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:29:36.948662 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:36.948052    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs podName:0a3b6564-fae6-476b-8e7d-003d4beaf870 nodeName:}" failed. No retries permitted until 2026-09-12 00:29:44.948034556 +0000 UTC m=+17.152035369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs") pod "network-metrics-daemon-qvqwv" (UID: "0a3b6564-fae6-476b-8e7d-003d4beaf870") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:29:36.948662 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:36.947984    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:29:36.948662 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:36.948475    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret podName:4967c7dd-797f-4225-a54d-6b83c6827183 nodeName:}" failed. No retries permitted until 2026-09-12 00:29:44.948457443 +0000 UTC m=+17.152458260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret") pod "global-pull-secret-syncer-q9l42" (UID: "4967c7dd-797f-4225-a54d-6b83c6827183") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:29:37.049252 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:37.049212    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7plg\" (UniqueName: \"kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg\") pod \"network-check-target-jlqxr\" (UID: \"e236a93e-7f67-4f7b-bc99-96072320715a\") " pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:37.049447 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:37.049411    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:29:37.049447 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:37.049446    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:29:37.049557 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:37.049461    2567 projected.go:196] Error preparing data for projected volume kube-api-access-w7plg for pod openshift-network-diagnostics/network-check-target-jlqxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:29:37.049557 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:37.049525    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg podName:e236a93e-7f67-4f7b-bc99-96072320715a nodeName:}" failed. No retries permitted until 2026-09-12 00:29:45.049505923 +0000 UTC m=+17.253506742 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w7plg" (UniqueName: "kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg") pod "network-check-target-jlqxr" (UID: "e236a93e-7f67-4f7b-bc99-96072320715a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:29:37.267625 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:37.267587    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:37.267625 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:37.267622    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:37.267885 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:37.267622    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:37.267885 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:37.267730    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:37.267885 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:37.267831    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:37.268048 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:37.267891    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:39.267138 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:39.267097    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:39.267138 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:39.267146    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:39.267664 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:39.267157    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:39.267664 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:39.267232    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:39.267664 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:39.267400    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:39.267664 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:39.267552    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:41.267232 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:41.267196    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:41.267708 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:41.267196    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:41.267708 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:41.267326    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:41.267708 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:41.267434    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:41.267708 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:41.267196    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:41.267708 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:41.267531    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:43.268071 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:43.267844    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:43.268507 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:43.267854    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:43.268507 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:43.268157    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:43.268507 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:43.267883    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:43.268507 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:43.268279    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:43.268507 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:43.268382    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:45.008067 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:45.008025    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:45.008549 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:45.008111    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:45.008549 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:45.008197    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:29:45.008549 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:45.008217    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:29:45.008549 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:45.008282    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret podName:4967c7dd-797f-4225-a54d-6b83c6827183 nodeName:}" failed. No retries permitted until 2026-09-12 00:30:01.008260712 +0000 UTC m=+33.212261515 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret") pod "global-pull-secret-syncer-q9l42" (UID: "4967c7dd-797f-4225-a54d-6b83c6827183") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:29:45.008549 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:45.008304    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs podName:0a3b6564-fae6-476b-8e7d-003d4beaf870 nodeName:}" failed. No retries permitted until 2026-09-12 00:30:01.008293781 +0000 UTC m=+33.212294581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs") pod "network-metrics-daemon-qvqwv" (UID: "0a3b6564-fae6-476b-8e7d-003d4beaf870") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:29:45.109173 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:45.109116    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7plg\" (UniqueName: \"kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg\") pod \"network-check-target-jlqxr\" (UID: \"e236a93e-7f67-4f7b-bc99-96072320715a\") " pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:45.109350 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:45.109305    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:29:45.109350 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:45.109331    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:29:45.109350 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:45.109346    2567 projected.go:196] Error preparing data for projected volume kube-api-access-w7plg for pod openshift-network-diagnostics/network-check-target-jlqxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:29:45.109491 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:45.109432    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg podName:e236a93e-7f67-4f7b-bc99-96072320715a nodeName:}" failed. No retries permitted until 2026-09-12 00:30:01.109413297 +0000 UTC m=+33.313414101 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w7plg" (UniqueName: "kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg") pod "network-check-target-jlqxr" (UID: "e236a93e-7f67-4f7b-bc99-96072320715a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:29:45.268168 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:45.268071    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:45.268168 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:45.268106    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:45.268378 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:45.268086    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:45.268378 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:45.268204    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:45.268378 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:45.268275    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:45.268487 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:45.268376    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:47.267625 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:47.267592    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:47.267625 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:47.267624    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:47.268144 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:47.267642    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:47.268144 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:47.267750    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:47.268144 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:47.267887    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:47.268144 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:47.267969    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:48.453196 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:48.453158    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal" event={"ID":"eddb3576bde467541695f2fe0d5bc466","Type":"ContainerStarted","Data":"95fd959919aafcb09828d518b574530413f7eb86d923d076d06a59aa32ab046b"}
Sep 12 00:29:48.458219 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:48.457309    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tp9d2" event={"ID":"831bac43-1246-48fe-84d6-ac43968b5852","Type":"ContainerStarted","Data":"1961ceb55c9c7ed59fcfdb3b24408493434f9635b5ae8469995f689958e028c9"}
Sep 12 00:29:48.467078 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:48.467017    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-132.ec2.internal" podStartSLOduration=20.466996319 podStartE2EDuration="20.466996319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:29:48.466093697 +0000 UTC m=+20.670094519" watchObservedRunningTime="2026-09-12 00:29:48.466996319 +0000 UTC m=+20.670997144"
Sep 12 00:29:48.468349 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:48.468309    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf" event={"ID":"99f21261-c1a7-40c2-880a-dd622834a861","Type":"ContainerStarted","Data":"5e2720c694e89d1efca6ad866ea6d4262aff4b63442824f7f38f7d7222d68de2"}
Sep 12 00:29:48.468446 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:48.468431    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf" event={"ID":"99f21261-c1a7-40c2-880a-dd622834a861","Type":"ContainerStarted","Data":"b16dbc0a961777a83427ea64db03721056dcd7bef5b51a0c4ba06d94dcfeda85"}
Sep 12 00:29:48.468506 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:48.468486    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf" event={"ID":"99f21261-c1a7-40c2-880a-dd622834a861","Type":"ContainerStarted","Data":"f8ff8069d3c130a4a3346fbdb50db1a180a9181c9849e75d1a49581fd4609d78"}
Sep 12 00:29:48.468675 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:48.468535    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf" event={"ID":"99f21261-c1a7-40c2-880a-dd622834a861","Type":"ContainerStarted","Data":"0c3f27c9c687a9a20ae6b8700408fa52e0eaee7da94aebb1ca2bb90595c0b1e2"}
Sep 12 00:29:48.471180 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:48.471148    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rkhbc" event={"ID":"92e821e2-a347-4e2b-9b5e-9a90a815e210","Type":"ContainerStarted","Data":"1912c4fb43291026244e856499c01f0454f42f93a735f7ec370e88baa4aa270d"}
Sep 12 00:29:48.494928 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:48.494853    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tp9d2" podStartSLOduration=2.186827513 podStartE2EDuration="20.494830524s" totalImagesPullingTime="18.308003011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:29:48.480080338 +0000 UTC m=+20.684081160" watchObservedRunningTime="2026-09-12 00:29:48.494830524 +0000 UTC m=+20.698831348"
Sep 12 00:29:48.495963 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:48.495594    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rkhbc" podStartSLOduration=1.951313377 podStartE2EDuration="20.495553878s" totalImagesPullingTime="18.544240501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:29:48.49520585 +0000 UTC m=+20.699206673" watchObservedRunningTime="2026-09-12 00:29:48.495553878 +0000 UTC m=+20.699554704"
Sep 12 00:29:49.267900 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.267714    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:49.268069 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.267728    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:49.268121 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:49.268088    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:49.268121 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.267761    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:49.268197 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:49.267972    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:49.268245 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:49.268196    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:49.474673 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.474634    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v" event={"ID":"80f6ccea-903a-4044-b22c-25ddc2cf19d9","Type":"ContainerStarted","Data":"199c3bea147d2176cf305c615cedb252dc1226417c44e67e5aee5ba13629ae02"}
Sep 12 00:29:49.476300 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.476265    2567 generic.go:356] "Generic (PLEG): container finished" podID="ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7" containerID="b87d28a8973e7507eb1120d44ca8b366d946446097a6a3fa8f2c86be69d9d3fc" exitCode=0
Sep 12 00:29:49.476439 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.476315    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v88kn" event={"ID":"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7","Type":"ContainerDied","Data":"b87d28a8973e7507eb1120d44ca8b366d946446097a6a3fa8f2c86be69d9d3fc"}
Sep 12 00:29:49.479754 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.479722    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf" event={"ID":"99f21261-c1a7-40c2-880a-dd622834a861","Type":"ContainerStarted","Data":"704657e11bb2a6a356663bf0e1016fc9a8ca76ec34977c06e42e315e74871971"}
Sep 12 00:29:49.479852 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.479794    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf" event={"ID":"99f21261-c1a7-40c2-880a-dd622834a861","Type":"ContainerStarted","Data":"e5a9dc23c6375952459025e6488cd9daed4d7d1e8a0b07ae5f1b7c376c2527be"}
Sep 12 00:29:49.481257 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.481209    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2zgfm" event={"ID":"620cca97-55a4-47db-9f3c-ca8912a77cba","Type":"ContainerStarted","Data":"4d634bd83a58f1f3df26468a6e078993286ca3653ba04f0b64d808e6dd92233e"}
Sep 12 00:29:49.482879 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.482853    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7pvhz" event={"ID":"0e01af40-0cf2-43d6-83c5-12215a4c0a4f","Type":"ContainerStarted","Data":"938da966510050dbbcea8b625b050caeb928771a6c3a55369ba10aec28741faf"}
Sep 12 00:29:49.484640 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.484620    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zrcf9" event={"ID":"34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e","Type":"ContainerStarted","Data":"d48aacb4e0f2f6e08368c48d451a9c6b1ac40a8de4913c87b3c5ff13c7cc9f05"}
Sep 12 00:29:49.486204 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.486183    2567 generic.go:356] "Generic (PLEG): container finished" podID="27e2d2995fe2c1dca3b74b7652c936c9" containerID="3aebf540a2c5caf5552fa552e54ee11ca4f113c1911a03108858eaf5692f6cde" exitCode=0
Sep 12 00:29:49.486298 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.486254    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" event={"ID":"27e2d2995fe2c1dca3b74b7652c936c9","Type":"ContainerDied","Data":"3aebf540a2c5caf5552fa552e54ee11ca4f113c1911a03108858eaf5692f6cde"}
Sep 12 00:29:49.487746 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.487724    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h688x" event={"ID":"661a55c6-221b-4cb8-b50b-ff96524e9ac8","Type":"ContainerStarted","Data":"7235beea75781ea3c1c18f9cab715960067f251a7bef788d9f8a0160f76c02b9"}
Sep 12 00:29:49.510303 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.510249    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h688x" podStartSLOduration=3.210209226 podStartE2EDuration="21.510230364s" totalImagesPullingTime="18.300021138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:29:49.509486795 +0000 UTC m=+21.713487621" watchObservedRunningTime="2026-09-12 00:29:49.510230364 +0000 UTC m=+21.714231185"
Sep 12 00:29:49.538479 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.538418    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zrcf9" podStartSLOduration=3.185311218 podStartE2EDuration="21.538396902s" totalImagesPullingTime="18.353085684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:29:49.538303191 +0000 UTC m=+21.742304013" watchObservedRunningTime="2026-09-12 00:29:49.538396902 +0000 UTC m=+21.742397726"
Sep 12 00:29:49.551499 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.551405    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2zgfm" podStartSLOduration=3.217968931 podStartE2EDuration="21.551383843s" totalImagesPullingTime="18.333414912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:29:49.551135892 +0000 UTC m=+21.755136715" watchObservedRunningTime="2026-09-12 00:29:49.551383843 +0000 UTC m=+21.755384667"
Sep 12 00:29:49.759421 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.759393    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zrcf9"
Sep 12 00:29:49.760031 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.760004    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zrcf9"
Sep 12 00:29:49.774380 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.774285    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7pvhz" podStartSLOduration=3.451753426 podStartE2EDuration="21.774269614s" totalImagesPullingTime="18.322516188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:29:49.564321325 +0000 UTC m=+21.768322149" watchObservedRunningTime="2026-09-12 00:29:49.774269614 +0000 UTC m=+21.978270435"
Sep 12 00:29:49.815169 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:49.815137    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 12 00:29:50.238223 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:50.238087    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-12T00:29:49.815162362Z","UUID":"a7ea531b-f2fc-4583-ab01-46a77f4add41","Handler":null,"Name":"","Endpoint":""}
Sep 12 00:29:50.239853 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:50.239826    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 12 00:29:50.239994 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:50.239873    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 12 00:29:50.493217 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:50.493092    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v" event={"ID":"80f6ccea-903a-4044-b22c-25ddc2cf19d9","Type":"ContainerStarted","Data":"c314b838311de4f9957faf46fd8a68e02be157232aada86cd82ed5d1d114633c"}
Sep 12 00:29:50.495678 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:50.495626    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" event={"ID":"27e2d2995fe2c1dca3b74b7652c936c9","Type":"ContainerStarted","Data":"1ef4c7c9a42b175da63243a607687302220aeae8f3da34d490d2baca9bed12f9"}
Sep 12 00:29:50.495830 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:50.495805    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zrcf9"
Sep 12 00:29:50.496725 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:50.496705    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zrcf9"
Sep 12 00:29:50.510061 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:50.509998    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-132.ec2.internal" podStartSLOduration=21.50997432 podStartE2EDuration="21.50997432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:29:50.50922691 +0000 UTC m=+22.713227732" watchObservedRunningTime="2026-09-12 00:29:50.50997432 +0000 UTC m=+22.713975144"
Sep 12 00:29:51.268088 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:51.268051    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:51.268280 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:51.268051    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:51.268280 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:51.268204    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:51.268280 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:51.268056    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:51.268280 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:51.268259    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:51.268470 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:51.268333    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:51.500852 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:51.500813    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v" event={"ID":"80f6ccea-903a-4044-b22c-25ddc2cf19d9","Type":"ContainerStarted","Data":"632585ffefdd433017f8a6c9c7967f90564c34aed57690aebed29eb72f1b0021"}
Sep 12 00:29:51.505053 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:51.505015    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf" event={"ID":"99f21261-c1a7-40c2-880a-dd622834a861","Type":"ContainerStarted","Data":"a91bde1179435c8253c3ad9527b9d7bef902a8b4c033ef1faa05b600ba9a6aff"}
Sep 12 00:29:51.521059 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:51.520996    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb96v" podStartSLOduration=2.8717499269999998 podStartE2EDuration="23.520971766s" totalImagesPullingTime="20.649221839s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:29:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:29:51.520588357 +0000 UTC m=+23.724589171" watchObservedRunningTime="2026-09-12 00:29:51.520971766 +0000 UTC m=+23.724972589"
Sep 12 00:29:53.267746 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:53.267711    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:53.268259 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:53.267711    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:53.268259 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:53.267820    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:53.268259 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:53.267909    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:53.268259 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:53.267721    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:53.268259 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:53.267986    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:54.514260 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:54.514077    2567 generic.go:356] "Generic (PLEG): container finished" podID="ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7" containerID="afd555cc7d2094c6ada4ae2a969c90f0838b1d79d968836e776cd022b3c564f4" exitCode=0
Sep 12 00:29:54.514705 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:54.514170    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v88kn" event={"ID":"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7","Type":"ContainerDied","Data":"afd555cc7d2094c6ada4ae2a969c90f0838b1d79d968836e776cd022b3c564f4"}
Sep 12 00:29:54.518343 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:54.518314    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf" event={"ID":"99f21261-c1a7-40c2-880a-dd622834a861","Type":"ContainerStarted","Data":"4ac92c32950bc09ac0dcc597ec1c370065e75516692ec87597ad8a30eeaca8cf"}
Sep 12 00:29:54.518625 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:54.518605    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:54.536334 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:54.534317    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:54.560446 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:54.560393    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf" podStartSLOduration=8.194391031 podStartE2EDuration="26.560375728s" totalImagesPullingTime="18.365984697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:29:54.558980886 +0000 UTC m=+26.762981709" watchObservedRunningTime="2026-09-12 00:29:54.560375728 +0000 UTC m=+26.764376550"
Sep 12 00:29:55.267995 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:55.267934    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:55.267995 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:55.267987    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:55.268267 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:55.268046    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:55.268267 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:55.268065    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:55.268267 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:55.268156    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:55.268267 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:55.268233    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:55.515304 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:55.513937    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvqwv"]
Sep 12 00:29:55.515304 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:55.514232    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jlqxr"]
Sep 12 00:29:55.521899 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:55.521820    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:55.522038 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:55.521953    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:55.522038 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:55.521978    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:55.522139 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:55.522035    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:55.522613 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:55.522587    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:55.522720 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:55.522673    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:55.525636 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:55.525608    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q9l42"]
Sep 12 00:29:55.525770 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:55.525709    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:55.525838 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:55.525815    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:55.542913 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:55.542887    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:29:56.525512 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:56.525475    2567 generic.go:356] "Generic (PLEG): container finished" podID="ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7" containerID="73c610057ba15dc54edf3aa5f7b78717a1e0f38bdaf412d67abf51dde33f17fe" exitCode=0
Sep 12 00:29:56.525905 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:56.525560    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v88kn" event={"ID":"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7","Type":"ContainerDied","Data":"73c610057ba15dc54edf3aa5f7b78717a1e0f38bdaf412d67abf51dde33f17fe"}
Sep 12 00:29:57.267342 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:57.267096    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:57.267437 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:57.267244    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:57.267482 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:57.267455    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:57.267546 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:57.267244    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:57.267621 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:57.267524    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:29:57.267694 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:57.267673    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:57.530921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:57.530889    2567 generic.go:356] "Generic (PLEG): container finished" podID="ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7" containerID="dc01f9bcbd875868f1303c4c3edd1aeb9b6d39172fbff420d5c9797fd93442b6" exitCode=0
Sep 12 00:29:57.531308 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:57.530977    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v88kn" event={"ID":"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7","Type":"ContainerDied","Data":"dc01f9bcbd875868f1303c4c3edd1aeb9b6d39172fbff420d5c9797fd93442b6"}
Sep 12 00:29:59.267139 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:59.267106    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:29:59.267621 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:59.267112    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:29:59.267621 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:59.267217    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:29:59.267621 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:29:59.267232    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:29:59.267621 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:59.267319    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:29:59.267621 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:29:59.267346    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:30:01.023100 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:01.023060    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:30:01.023605 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:01.023120    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:30:01.023605 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:01.023223    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:30:01.023605 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:01.023284    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret podName:4967c7dd-797f-4225-a54d-6b83c6827183 nodeName:}" failed. No retries permitted until 2026-09-12 00:30:33.023270237 +0000 UTC m=+65.227271037 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret") pod "global-pull-secret-syncer-q9l42" (UID: "4967c7dd-797f-4225-a54d-6b83c6827183") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:30:01.023605 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:01.023221    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:30:01.023605 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:01.023386    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs podName:0a3b6564-fae6-476b-8e7d-003d4beaf870 nodeName:}" failed. No retries permitted until 2026-09-12 00:30:33.023366698 +0000 UTC m=+65.227367502 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs") pod "network-metrics-daemon-qvqwv" (UID: "0a3b6564-fae6-476b-8e7d-003d4beaf870") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:30:01.124354 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:01.124313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7plg\" (UniqueName: \"kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg\") pod \"network-check-target-jlqxr\" (UID: \"e236a93e-7f67-4f7b-bc99-96072320715a\") " pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:30:01.124543 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:01.124478    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:30:01.124543 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:01.124496    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:30:01.124543 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:01.124506    2567 projected.go:196] Error preparing data for projected volume kube-api-access-w7plg for pod openshift-network-diagnostics/network-check-target-jlqxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:30:01.124709 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:01.124557    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg podName:e236a93e-7f67-4f7b-bc99-96072320715a nodeName:}" failed. No retries permitted until 2026-09-12 00:30:33.124541319 +0000 UTC m=+65.328542124 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w7plg" (UniqueName: "kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg") pod "network-check-target-jlqxr" (UID: "e236a93e-7f67-4f7b-bc99-96072320715a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:30:01.268017 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:01.267925    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:30:01.268181 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:01.268054    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:30:01.268181 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:01.268059    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q9l42" podUID="4967c7dd-797f-4225-a54d-6b83c6827183"
Sep 12 00:30:01.268181 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:01.268109    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:30:01.268303 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:01.268209    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jlqxr" podUID="e236a93e-7f67-4f7b-bc99-96072320715a"
Sep 12 00:30:01.268345 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:01.268307    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:30:02.118752 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.118712    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-132.ec2.internal" event="NodeReady"
Sep 12 00:30:02.119189 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.118889    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 12 00:30:02.159877 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.159840    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p94bw"]
Sep 12 00:30:02.186436 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.186401    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f5zs5"]
Sep 12 00:30:02.186643 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.186544    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:02.189219 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.189176    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h7jvg\""
Sep 12 00:30:02.189379 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.189332    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 12 00:30:02.189493 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.189404    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 12 00:30:02.204579 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.204534    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p94bw"]
Sep 12 00:30:02.204722 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.204604    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f5zs5"]
Sep 12 00:30:02.204722 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.204709    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f5zs5"
Sep 12 00:30:02.207329 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.207304    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 12 00:30:02.207520 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.207502    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 12 00:30:02.207606 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.207590    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zbz7v\""
Sep 12 00:30:02.207817 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.207801    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 12 00:30:02.335513 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.335473    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ssb7\" (UniqueName: \"kubernetes.io/projected/1a25679b-9c33-46d9-a0c5-77b6922c31f6-kube-api-access-5ssb7\") pod \"ingress-canary-f5zs5\" (UID: \"1a25679b-9c33-46d9-a0c5-77b6922c31f6\") " pod="openshift-ingress-canary/ingress-canary-f5zs5"
Sep 12 00:30:02.335692 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.335526    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:02.335692 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.335548    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c884033-b8a0-4f39-a5b7-d5006599e229-tmp-dir\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:02.335692 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.335598    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a25679b-9c33-46d9-a0c5-77b6922c31f6-cert\") pod \"ingress-canary-f5zs5\" (UID: \"1a25679b-9c33-46d9-a0c5-77b6922c31f6\") " pod="openshift-ingress-canary/ingress-canary-f5zs5"
Sep 12 00:30:02.335692 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.335618    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c884033-b8a0-4f39-a5b7-d5006599e229-config-volume\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:02.335692 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.335640    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hhgf\" (UniqueName: \"kubernetes.io/projected/5c884033-b8a0-4f39-a5b7-d5006599e229-kube-api-access-9hhgf\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:02.436683 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.436651    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:02.436683 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.436686    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c884033-b8a0-4f39-a5b7-d5006599e229-tmp-dir\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:02.436838 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.436710    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a25679b-9c33-46d9-a0c5-77b6922c31f6-cert\") pod \"ingress-canary-f5zs5\" (UID: \"1a25679b-9c33-46d9-a0c5-77b6922c31f6\") " pod="openshift-ingress-canary/ingress-canary-f5zs5"
Sep 12 00:30:02.436838 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.436740    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c884033-b8a0-4f39-a5b7-d5006599e229-config-volume\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:02.436838 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.436776    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hhgf\" (UniqueName: \"kubernetes.io/projected/5c884033-b8a0-4f39-a5b7-d5006599e229-kube-api-access-9hhgf\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:02.436838 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.436833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ssb7\" (UniqueName: \"kubernetes.io/projected/1a25679b-9c33-46d9-a0c5-77b6922c31f6-kube-api-access-5ssb7\") pod \"ingress-canary-f5zs5\" (UID: \"1a25679b-9c33-46d9-a0c5-77b6922c31f6\") " pod="openshift-ingress-canary/ingress-canary-f5zs5"
Sep 12 00:30:02.437001 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:02.436978    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:30:02.437076 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:02.437064    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls podName:5c884033-b8a0-4f39-a5b7-d5006599e229 nodeName:}" failed. No retries permitted until 2026-09-12 00:30:02.937041248 +0000 UTC m=+35.141042050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls") pod "dns-default-p94bw" (UID: "5c884033-b8a0-4f39-a5b7-d5006599e229") : secret "dns-default-metrics-tls" not found
Sep 12 00:30:02.437145 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.437094    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c884033-b8a0-4f39-a5b7-d5006599e229-tmp-dir\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:02.437315 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.437296    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c884033-b8a0-4f39-a5b7-d5006599e229-config-volume\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:02.441478 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.441347    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a25679b-9c33-46d9-a0c5-77b6922c31f6-cert\") pod \"ingress-canary-f5zs5\" (UID: \"1a25679b-9c33-46d9-a0c5-77b6922c31f6\") " pod="openshift-ingress-canary/ingress-canary-f5zs5"
Sep 12 00:30:02.445646 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.445617    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ssb7\" (UniqueName: \"kubernetes.io/projected/1a25679b-9c33-46d9-a0c5-77b6922c31f6-kube-api-access-5ssb7\") pod \"ingress-canary-f5zs5\" (UID: \"1a25679b-9c33-46d9-a0c5-77b6922c31f6\") " pod="openshift-ingress-canary/ingress-canary-f5zs5"
Sep 12 00:30:02.445952 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.445933    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hhgf\" (UniqueName: \"kubernetes.io/projected/5c884033-b8a0-4f39-a5b7-d5006599e229-kube-api-access-9hhgf\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:02.549966 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.549931    2567 generic.go:356] "Generic (PLEG): container finished" podID="ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7" containerID="71f21981bc11c13d81ebdd9f9afba7754ebb02516be4c23a31f4d5e21b38e13d" exitCode=0
Sep 12 00:30:02.550128 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.549993    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v88kn" event={"ID":"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7","Type":"ContainerDied","Data":"71f21981bc11c13d81ebdd9f9afba7754ebb02516be4c23a31f4d5e21b38e13d"}
Sep 12 00:30:02.570154 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.570125    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f5zs5"
Sep 12 00:30:02.895160 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.895073    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f5zs5"]
Sep 12 00:30:02.899194 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:30:02.899160    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a25679b_9c33_46d9_a0c5_77b6922c31f6.slice/crio-843eac6994f1f9fabce87346c466273eed099cc73835da83c09ca9ad6c2c57dc WatchSource:0}: Error finding container 843eac6994f1f9fabce87346c466273eed099cc73835da83c09ca9ad6c2c57dc: Status 404 returned error can't find the container with id 843eac6994f1f9fabce87346c466273eed099cc73835da83c09ca9ad6c2c57dc
Sep 12 00:30:02.940295 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:02.940259    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:02.940468 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:02.940430    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:30:02.940505 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:02.940498    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls podName:5c884033-b8a0-4f39-a5b7-d5006599e229 nodeName:}" failed. No retries permitted until 2026-09-12 00:30:03.940480772 +0000 UTC m=+36.144481577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls") pod "dns-default-p94bw" (UID: "5c884033-b8a0-4f39-a5b7-d5006599e229") : secret "dns-default-metrics-tls" not found
Sep 12 00:30:03.267962 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:03.267926    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:30:03.268761 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:03.267926    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:30:03.268761 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:03.267926    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:30:03.270597 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:03.270541    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9g5b8\""
Sep 12 00:30:03.270597 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:03.270541    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 00:30:03.270774 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:03.270759    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 00:30:03.271523 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:03.271507    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8bjlg\""
Sep 12 00:30:03.271523 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:03.271519    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 00:30:03.271717 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:03.271534    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 00:30:03.555211 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:03.555117    2567 generic.go:356] "Generic (PLEG): container finished" podID="ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7" containerID="bd21bcde96dcf03cf07d6bbcc05497afb237a59c8d66ed277915c00d3c4c9556" exitCode=0
Sep 12 00:30:03.555380 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:03.555206    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v88kn" event={"ID":"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7","Type":"ContainerDied","Data":"bd21bcde96dcf03cf07d6bbcc05497afb237a59c8d66ed277915c00d3c4c9556"}
Sep 12 00:30:03.556440 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:03.556410    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f5zs5" event={"ID":"1a25679b-9c33-46d9-a0c5-77b6922c31f6","Type":"ContainerStarted","Data":"843eac6994f1f9fabce87346c466273eed099cc73835da83c09ca9ad6c2c57dc"}
Sep 12 00:30:03.948603 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:03.948296    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:03.948603 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:03.948437    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:30:03.948830 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:03.948663    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls podName:5c884033-b8a0-4f39-a5b7-d5006599e229 nodeName:}" failed. No retries permitted until 2026-09-12 00:30:05.948644844 +0000 UTC m=+38.152645658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls") pod "dns-default-p94bw" (UID: "5c884033-b8a0-4f39-a5b7-d5006599e229") : secret "dns-default-metrics-tls" not found
Sep 12 00:30:04.563503 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:04.563463    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v88kn" event={"ID":"ac4faa49-242a-4201-bd5a-e9e1bcfc1ea7","Type":"ContainerStarted","Data":"5ce53da2cc11839d04f3f00a9ea437e93a7d999c7a5c486794a8c75c99a943da"}
Sep 12 00:30:04.584240 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:04.584172    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v88kn" podStartSLOduration=8.015115555 podStartE2EDuration="36.584148323s" totalImagesPullingTime="28.569032768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:29:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:30:04.583826473 +0000 UTC m=+36.787827297" watchObservedRunningTime="2026-09-12 00:30:04.584148323 +0000 UTC m=+36.788149146"
Sep 12 00:30:05.965770 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:05.965728    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:05.966185 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:05.965905    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:30:05.966185 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:05.965974    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls podName:5c884033-b8a0-4f39-a5b7-d5006599e229 nodeName:}" failed. No retries permitted until 2026-09-12 00:30:09.965957767 +0000 UTC m=+42.169958572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls") pod "dns-default-p94bw" (UID: "5c884033-b8a0-4f39-a5b7-d5006599e229") : secret "dns-default-metrics-tls" not found
Sep 12 00:30:06.569853 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:06.569817    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f5zs5" event={"ID":"1a25679b-9c33-46d9-a0c5-77b6922c31f6","Type":"ContainerStarted","Data":"114a188f07b718f54869d5f765f6f2a9410a9f6fd46209924c7cebae2299be6a"}
Sep 12 00:30:06.587293 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:06.587237    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f5zs5" podStartSLOduration=1.377753314 podStartE2EDuration="4.587219831s" totalImagesPullingTime="3.209466517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:30:06.586770488 +0000 UTC m=+38.790771322" watchObservedRunningTime="2026-09-12 00:30:06.587219831 +0000 UTC m=+38.791220652"
Sep 12 00:30:09.999019 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:09.998972    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:09.999427 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:09.999135    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:30:09.999427 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:09.999204    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls podName:5c884033-b8a0-4f39-a5b7-d5006599e229 nodeName:}" failed. No retries permitted until 2026-09-12 00:30:17.99918936 +0000 UTC m=+50.203190177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls") pod "dns-default-p94bw" (UID: "5c884033-b8a0-4f39-a5b7-d5006599e229") : secret "dns-default-metrics-tls" not found
Sep 12 00:30:18.054529 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:18.054480    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:18.055152 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:18.054683    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:30:18.055152 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:18.054771    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls podName:5c884033-b8a0-4f39-a5b7-d5006599e229 nodeName:}" failed. No retries permitted until 2026-09-12 00:30:34.054749972 +0000 UTC m=+66.258750777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls") pod "dns-default-p94bw" (UID: "5c884033-b8a0-4f39-a5b7-d5006599e229") : secret "dns-default-metrics-tls" not found
Sep 12 00:30:27.542056 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:27.542025    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fh7jf"
Sep 12 00:30:33.063827 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.063767    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:30:33.063827 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.063837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:30:33.066583 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.066550    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 00:30:33.066673 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.066655    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 00:30:33.074562 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:33.074523    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 12 00:30:33.074643 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:33.074633    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs podName:0a3b6564-fae6-476b-8e7d-003d4beaf870 nodeName:}" failed. No retries permitted until 2026-09-12 00:31:37.074612456 +0000 UTC m=+129.278613260 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs") pod "network-metrics-daemon-qvqwv" (UID: "0a3b6564-fae6-476b-8e7d-003d4beaf870") : secret "metrics-daemon-secret" not found
Sep 12 00:30:33.077700 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.077681    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4967c7dd-797f-4225-a54d-6b83c6827183-original-pull-secret\") pod \"global-pull-secret-syncer-q9l42\" (UID: \"4967c7dd-797f-4225-a54d-6b83c6827183\") " pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:30:33.164447 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.164408    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7plg\" (UniqueName: \"kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg\") pod \"network-check-target-jlqxr\" (UID: \"e236a93e-7f67-4f7b-bc99-96072320715a\") " pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:30:33.166827 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.166806    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 00:30:33.176852 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.176827    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 00:30:33.188009 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.187975    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7plg\" (UniqueName: \"kubernetes.io/projected/e236a93e-7f67-4f7b-bc99-96072320715a-kube-api-access-w7plg\") pod \"network-check-target-jlqxr\" (UID: \"e236a93e-7f67-4f7b-bc99-96072320715a\") " pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:30:33.280257 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.280229    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9g5b8\""
Sep 12 00:30:33.288425 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.288391    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:30:33.291990 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.291973    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q9l42"
Sep 12 00:30:33.560460 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.560426    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q9l42"]
Sep 12 00:30:33.561043 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.561022    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jlqxr"]
Sep 12 00:30:33.564036 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:30:33.564009    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4967c7dd_797f_4225_a54d_6b83c6827183.slice/crio-a15fe4b3f94a67cc7f9a7fb15d0683cfe99b7685e0077ffe834c927b6b2846ae WatchSource:0}: Error finding container a15fe4b3f94a67cc7f9a7fb15d0683cfe99b7685e0077ffe834c927b6b2846ae: Status 404 returned error can't find the container with id a15fe4b3f94a67cc7f9a7fb15d0683cfe99b7685e0077ffe834c927b6b2846ae
Sep 12 00:30:33.564436 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:30:33.564408    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode236a93e_7f67_4f7b_bc99_96072320715a.slice/crio-a176a62414bacbb573a422a310f2cd8fb61288cbb1786bbe765712647c1b4f55 WatchSource:0}: Error finding container a176a62414bacbb573a422a310f2cd8fb61288cbb1786bbe765712647c1b4f55: Status 404 returned error can't find the container with id a176a62414bacbb573a422a310f2cd8fb61288cbb1786bbe765712647c1b4f55
Sep 12 00:30:33.630428 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.630336    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q9l42" event={"ID":"4967c7dd-797f-4225-a54d-6b83c6827183","Type":"ContainerStarted","Data":"a15fe4b3f94a67cc7f9a7fb15d0683cfe99b7685e0077ffe834c927b6b2846ae"}
Sep 12 00:30:33.631493 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:33.631472    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jlqxr" event={"ID":"e236a93e-7f67-4f7b-bc99-96072320715a","Type":"ContainerStarted","Data":"a176a62414bacbb573a422a310f2cd8fb61288cbb1786bbe765712647c1b4f55"}
Sep 12 00:30:34.069311 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:34.069276    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:30:34.069761 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:34.069399    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:30:34.069761 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:30:34.069462    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls podName:5c884033-b8a0-4f39-a5b7-d5006599e229 nodeName:}" failed. No retries permitted until 2026-09-12 00:31:06.069448432 +0000 UTC m=+98.273449232 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls") pod "dns-default-p94bw" (UID: "5c884033-b8a0-4f39-a5b7-d5006599e229") : secret "dns-default-metrics-tls" not found
Sep 12 00:30:38.646478 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:38.646367    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q9l42" event={"ID":"4967c7dd-797f-4225-a54d-6b83c6827183","Type":"ContainerStarted","Data":"13475ec1308172b0bdb43fd39ded53b971b74814d29c5631c54e765c67d05ce3"}
Sep 12 00:30:38.648030 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:38.648001    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jlqxr" event={"ID":"e236a93e-7f67-4f7b-bc99-96072320715a","Type":"ContainerStarted","Data":"1587b04d18cf696d5d3e558361ea9087cac61d69713f06d6186427585ecaee7a"}
Sep 12 00:30:38.648155 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:38.648144    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:30:38.661146 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:38.661087    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q9l42" podStartSLOduration=65.857932841 podStartE2EDuration="1m10.661069697s" totalImagesPullingTime="4.803136856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:30:38.660010336 +0000 UTC m=+70.864011216" watchObservedRunningTime="2026-09-12 00:30:38.661069697 +0000 UTC m=+70.865070519"
Sep 12 00:30:38.675608 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:30:38.675507    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jlqxr" podStartSLOduration=65.881295035 podStartE2EDuration="1m10.675484167s" totalImagesPullingTime="4.794189132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:30:38.674648132 +0000 UTC m=+70.878648955" watchObservedRunningTime="2026-09-12 00:30:38.675484167 +0000 UTC m=+70.879484987"
Sep 12 00:31:06.092372 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:06.092319    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:31:06.092846 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:31:06.092488    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:31:06.092846 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:31:06.092560    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls podName:5c884033-b8a0-4f39-a5b7-d5006599e229 nodeName:}" failed. No retries permitted until 2026-09-12 00:32:10.09254366 +0000 UTC m=+162.296544460 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls") pod "dns-default-p94bw" (UID: "5c884033-b8a0-4f39-a5b7-d5006599e229") : secret "dns-default-metrics-tls" not found
Sep 12 00:31:09.423885 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.423844    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd"]
Sep 12 00:31:09.437438 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.437396    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"]
Sep 12 00:31:09.437620 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.437549    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd"
Sep 12 00:31:09.440794 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.440760    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wfnrm\""
Sep 12 00:31:09.440953 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.440811    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 12 00:31:09.440953 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.440854    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 12 00:31:09.440953 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.440758    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 12 00:31:09.441194 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.441178    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 12 00:31:09.446451 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.446425    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd"]
Sep 12 00:31:09.446597 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.446460    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"]
Sep 12 00:31:09.446597 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.446557    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.448928 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.448903    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 12 00:31:09.448928 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.448921    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 12 00:31:09.449066 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.448904    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 12 00:31:09.449066 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.449024    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 12 00:31:09.517651 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.517611    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.517651 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.517657    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-hub\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.517905 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.517730    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.517905 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.517770    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80673e07-d7c6-4c37-ab2e-1a08eb278b32-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d6b69ff6-rg4wd\" (UID: \"80673e07-d7c6-4c37-ab2e-1a08eb278b32\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd"
Sep 12 00:31:09.517905 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.517794    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-ca\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.517905 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.517854    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.517905 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.517893    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4cr4\" (UniqueName: \"kubernetes.io/projected/80673e07-d7c6-4c37-ab2e-1a08eb278b32-kube-api-access-z4cr4\") pod \"managed-serviceaccount-addon-agent-d6b69ff6-rg4wd\" (UID: \"80673e07-d7c6-4c37-ab2e-1a08eb278b32\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd"
Sep 12 00:31:09.518078 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.517915    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqsz9\" (UniqueName: \"kubernetes.io/projected/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-kube-api-access-pqsz9\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.618698 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.618656    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.618698 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.618700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4cr4\" (UniqueName: \"kubernetes.io/projected/80673e07-d7c6-4c37-ab2e-1a08eb278b32-kube-api-access-z4cr4\") pod \"managed-serviceaccount-addon-agent-d6b69ff6-rg4wd\" (UID: \"80673e07-d7c6-4c37-ab2e-1a08eb278b32\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd"
Sep 12 00:31:09.618956 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.618723    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqsz9\" (UniqueName: \"kubernetes.io/projected/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-kube-api-access-pqsz9\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.618956 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.618742    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.618956 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.618923    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-hub\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.619134 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.618998    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.619134 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.619035    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80673e07-d7c6-4c37-ab2e-1a08eb278b32-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d6b69ff6-rg4wd\" (UID: \"80673e07-d7c6-4c37-ab2e-1a08eb278b32\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd"
Sep 12 00:31:09.619134 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.619062    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-ca\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.619496 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.619463    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.621348 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.621326    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.621760 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.621737    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/80673e07-d7c6-4c37-ab2e-1a08eb278b32-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d6b69ff6-rg4wd\" (UID: \"80673e07-d7c6-4c37-ab2e-1a08eb278b32\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd"
Sep 12 00:31:09.621843 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.621759    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-ca\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.621843 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.621811    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-hub\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.622014 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.621998    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.625702 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.625678    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4cr4\" (UniqueName: \"kubernetes.io/projected/80673e07-d7c6-4c37-ab2e-1a08eb278b32-kube-api-access-z4cr4\") pod \"managed-serviceaccount-addon-agent-d6b69ff6-rg4wd\" (UID: \"80673e07-d7c6-4c37-ab2e-1a08eb278b32\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd"
Sep 12 00:31:09.625833 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.625821    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqsz9\" (UniqueName: \"kubernetes.io/projected/0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf-kube-api-access-pqsz9\") pod \"cluster-proxy-proxy-agent-ff596bd-dj9ch\" (UID: \"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:09.652588 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.652532    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jlqxr"
Sep 12 00:31:09.758631 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.758555    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd"
Sep 12 00:31:09.765792 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:09.765753    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:31:10.052826 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:10.052705    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd"]
Sep 12 00:31:10.056379 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:31:10.056342    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80673e07_d7c6_4c37_ab2e_1a08eb278b32.slice/crio-b5e1d7b30bd2f78847ffa02e1dd52c7d7d6db62ea41fbc2ef77b7911872eb403 WatchSource:0}: Error finding container b5e1d7b30bd2f78847ffa02e1dd52c7d7d6db62ea41fbc2ef77b7911872eb403: Status 404 returned error can't find the container with id b5e1d7b30bd2f78847ffa02e1dd52c7d7d6db62ea41fbc2ef77b7911872eb403
Sep 12 00:31:10.078013 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:10.077979    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"]
Sep 12 00:31:10.081083 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:31:10.081051    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0abb5b27_d1bc_42e6_ab0e_985ca62bfbdf.slice/crio-a84afdddb57cc4df2bff22e78be23fcc033515dbfbbb4cca634081a4d0624aac WatchSource:0}: Error finding container a84afdddb57cc4df2bff22e78be23fcc033515dbfbbb4cca634081a4d0624aac: Status 404 returned error can't find the container with id a84afdddb57cc4df2bff22e78be23fcc033515dbfbbb4cca634081a4d0624aac
Sep 12 00:31:10.724619 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:10.724559    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd" event={"ID":"80673e07-d7c6-4c37-ab2e-1a08eb278b32","Type":"ContainerStarted","Data":"b5e1d7b30bd2f78847ffa02e1dd52c7d7d6db62ea41fbc2ef77b7911872eb403"}
Sep 12 00:31:10.725943 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:10.725904    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch" event={"ID":"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf","Type":"ContainerStarted","Data":"a84afdddb57cc4df2bff22e78be23fcc033515dbfbbb4cca634081a4d0624aac"}
Sep 12 00:31:14.740699 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:14.740657    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd" event={"ID":"80673e07-d7c6-4c37-ab2e-1a08eb278b32","Type":"ContainerStarted","Data":"5f954c1c3b6bfb7829804a90983a9e7e6113d588e92e1d2d169896f7f1f4519a"}
Sep 12 00:31:14.742735 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:14.742696    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch" event={"ID":"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf","Type":"ContainerStarted","Data":"2f525d97f0c68e856c38feaa713f63eb3ea5b6f6e07a96ef7449421050d32227"}
Sep 12 00:31:14.742862 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:14.742742    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch" event={"ID":"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf","Type":"ContainerStarted","Data":"e3ff50d538095eb510c576bad20c0b8a479eddb8bca76c30f7763b675e5e782a"}
Sep 12 00:31:14.742862 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:14.742762    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch" event={"ID":"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf","Type":"ContainerStarted","Data":"70275cd62bb349366abcf8dad2379d20a0451a9bc1335740236473c2cae49079"}
Sep 12 00:31:14.755043 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:14.754977    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd" podStartSLOduration=1.513747655 podStartE2EDuration="5.754962105s" totalImagesPullingTime="4.24121445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:31:14.754261328 +0000 UTC m=+106.958262150" watchObservedRunningTime="2026-09-12 00:31:14.754962105 +0000 UTC m=+106.958963171"
Sep 12 00:31:14.770371 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:14.770316    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch" podStartSLOduration=1.5450745019999998 podStartE2EDuration="5.770300192s" totalImagesPullingTime="4.22522569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:31:14.769720966 +0000 UTC m=+106.973721789" watchObservedRunningTime="2026-09-12 00:31:14.770300192 +0000 UTC m=+106.974301014"
Sep 12 00:31:37.122748 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:31:37.122700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:31:37.123255 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:31:37.122855    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 12 00:31:37.123255 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:31:37.122923    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs podName:0a3b6564-fae6-476b-8e7d-003d4beaf870 nodeName:}" failed. No retries permitted until 2026-09-12 00:33:39.122904526 +0000 UTC m=+251.326905345 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs") pod "network-metrics-daemon-qvqwv" (UID: "0a3b6564-fae6-476b-8e7d-003d4beaf870") : secret "metrics-daemon-secret" not found
Sep 12 00:32:05.197409 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:32:05.197353    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-p94bw" podUID="5c884033-b8a0-4f39-a5b7-d5006599e229"
Sep 12 00:32:05.893029 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:32:05.892989    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p94bw"
Sep 12 00:32:06.284474 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:32:06.284431    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:32:10.153446 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:32:10.153406    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:32:10.153965 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:32:10.153518    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:32:10.153965 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:32:10.153593    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls podName:5c884033-b8a0-4f39-a5b7-d5006599e229 nodeName:}" failed. No retries permitted until 2026-09-12 00:34:12.153558616 +0000 UTC m=+284.357559417 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls") pod "dns-default-p94bw" (UID: "5c884033-b8a0-4f39-a5b7-d5006599e229") : secret "dns-default-metrics-tls" not found
Sep 12 00:32:18.268278 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:32:18.268246    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:33:15.096948 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:15.096906    2567 generic.go:356] "Generic (PLEG): container finished" podID="80673e07-d7c6-4c37-ab2e-1a08eb278b32" containerID="5f954c1c3b6bfb7829804a90983a9e7e6113d588e92e1d2d169896f7f1f4519a" exitCode=255
Sep 12 00:33:15.097414 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:15.096970    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd" event={"ID":"80673e07-d7c6-4c37-ab2e-1a08eb278b32","Type":"ContainerDied","Data":"5f954c1c3b6bfb7829804a90983a9e7e6113d588e92e1d2d169896f7f1f4519a"}
Sep 12 00:33:15.097414 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:15.097252    2567 scope.go:117] "RemoveContainer" containerID="5f954c1c3b6bfb7829804a90983a9e7e6113d588e92e1d2d169896f7f1f4519a"
Sep 12 00:33:16.102011 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:16.101964    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d6b69ff6-rg4wd" event={"ID":"80673e07-d7c6-4c37-ab2e-1a08eb278b32","Type":"ContainerStarted","Data":"8d0647c152eb45a436800765d28760cfe49c65f02140c58ba8a30e4ef4e0188b"}
Sep 12 00:33:34.661314 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.661278    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ckmv2"]
Sep 12 00:33:34.663454 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.663428    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:34.670197 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.670166    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 12 00:33:34.670197 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.670187    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 12 00:33:34.670401 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.670220    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 12 00:33:34.670401 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.670264    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 12 00:33:34.671151 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.671133    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hwlcm\""
Sep 12 00:33:34.687340 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.687309    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ckmv2"]
Sep 12 00:33:34.757187 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.757144    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87ebd59e-1ecc-4723-853e-89b54ab190a7-crio-socket\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:34.757187 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.757182    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:34.757399 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.757209    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:34.757399 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.757296    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjnmf\" (UniqueName: \"kubernetes.io/projected/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-api-access-kjnmf\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:34.757399 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.757329    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87ebd59e-1ecc-4723-853e-89b54ab190a7-data-volume\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:34.857797 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.857757    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87ebd59e-1ecc-4723-853e-89b54ab190a7-data-volume\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:34.857867 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.857811    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87ebd59e-1ecc-4723-853e-89b54ab190a7-crio-socket\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:34.857867 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.857831    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:34.857941 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.857853    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:34.857941 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.857906    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87ebd59e-1ecc-4723-853e-89b54ab190a7-crio-socket\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:34.858010 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.857918    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjnmf\" (UniqueName: \"kubernetes.io/projected/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-api-access-kjnmf\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:34.858010 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:34.857972    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:33:34.858073 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:34.858020    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:33:34.858073 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:34.858043    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm podName:87ebd59e-1ecc-4723-853e-89b54ab190a7 nodeName:}" failed. No retries permitted until 2026-09-12 00:33:35.358027317 +0000 UTC m=+247.562028128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ckmv2" (UID: "87ebd59e-1ecc-4723-853e-89b54ab190a7") : configmap "kube-rbac-proxy" not found
Sep 12 00:33:34.858151 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:34.858091    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls podName:87ebd59e-1ecc-4723-853e-89b54ab190a7 nodeName:}" failed. No retries permitted until 2026-09-12 00:33:35.358072705 +0000 UTC m=+247.562073524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ckmv2" (UID: "87ebd59e-1ecc-4723-853e-89b54ab190a7") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:33:34.858201 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.858154    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87ebd59e-1ecc-4723-853e-89b54ab190a7-data-volume\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:34.868075 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:34.868048    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjnmf\" (UniqueName: \"kubernetes.io/projected/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-api-access-kjnmf\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:35.363019 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:35.362963    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:35.363019 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:35.363022    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:35.363255 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:35.363095    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:33:35.363255 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:35.363113    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:33:35.363255 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:35.363176    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm podName:87ebd59e-1ecc-4723-853e-89b54ab190a7 nodeName:}" failed. No retries permitted until 2026-09-12 00:33:36.363156929 +0000 UTC m=+248.567157758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ckmv2" (UID: "87ebd59e-1ecc-4723-853e-89b54ab190a7") : configmap "kube-rbac-proxy" not found
Sep 12 00:33:35.363255 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:35.363199    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls podName:87ebd59e-1ecc-4723-853e-89b54ab190a7 nodeName:}" failed. No retries permitted until 2026-09-12 00:33:36.363186385 +0000 UTC m=+248.567187209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ckmv2" (UID: "87ebd59e-1ecc-4723-853e-89b54ab190a7") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:33:36.370444 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:36.370382    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:36.370444 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:36.370450    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:36.370977 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:36.370524    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:33:36.370977 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:36.370600    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:33:36.370977 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:36.370612    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm podName:87ebd59e-1ecc-4723-853e-89b54ab190a7 nodeName:}" failed. No retries permitted until 2026-09-12 00:33:38.370595793 +0000 UTC m=+250.574596614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ckmv2" (UID: "87ebd59e-1ecc-4723-853e-89b54ab190a7") : configmap "kube-rbac-proxy" not found
Sep 12 00:33:36.370977 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:36.370671    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls podName:87ebd59e-1ecc-4723-853e-89b54ab190a7 nodeName:}" failed. No retries permitted until 2026-09-12 00:33:38.370652698 +0000 UTC m=+250.574653518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ckmv2" (UID: "87ebd59e-1ecc-4723-853e-89b54ab190a7") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:33:37.407418 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:37.407383    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h688x_661a55c6-221b-4cb8-b50b-ff96524e9ac8/dns-node-resolver/0.log"
Sep 12 00:33:38.207979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:38.207949    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7pvhz_0e01af40-0cf2-43d6-83c5-12215a4c0a4f/node-ca/0.log"
Sep 12 00:33:38.387920 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:38.387880    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:38.388123 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:38.387967    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:38.388123 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:38.388039    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:33:38.388123 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:38.388089    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm podName:87ebd59e-1ecc-4723-853e-89b54ab190a7 nodeName:}" failed. No retries permitted until 2026-09-12 00:33:42.388075342 +0000 UTC m=+254.592076141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ckmv2" (UID: "87ebd59e-1ecc-4723-853e-89b54ab190a7") : configmap "kube-rbac-proxy" not found
Sep 12 00:33:38.388123 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:38.388039    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:33:38.388259 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:38.388169    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls podName:87ebd59e-1ecc-4723-853e-89b54ab190a7 nodeName:}" failed. No retries permitted until 2026-09-12 00:33:42.388155234 +0000 UTC m=+254.592156062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ckmv2" (UID: "87ebd59e-1ecc-4723-853e-89b54ab190a7") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:33:39.194909 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:39.194873    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:33:39.195341 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:39.195025    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 12 00:33:39.195341 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:39.195093    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs podName:0a3b6564-fae6-476b-8e7d-003d4beaf870 nodeName:}" failed. No retries permitted until 2026-09-12 00:35:41.195076875 +0000 UTC m=+373.399077694 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs") pod "network-metrics-daemon-qvqwv" (UID: "0a3b6564-fae6-476b-8e7d-003d4beaf870") : secret "metrics-daemon-secret" not found
Sep 12 00:33:39.208403 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:39.208312    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f5zs5_1a25679b-9c33-46d9-a0c5-77b6922c31f6/serve-healthcheck-canary/0.log"
Sep 12 00:33:42.417948 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:42.417910    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:42.418359 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:42.417956    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:42.418359 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:42.418028    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:33:42.418359 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:42.418096    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm podName:87ebd59e-1ecc-4723-853e-89b54ab190a7 nodeName:}" failed. No retries permitted until 2026-09-12 00:33:50.41807959 +0000 UTC m=+262.622080394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ckmv2" (UID: "87ebd59e-1ecc-4723-853e-89b54ab190a7") : configmap "kube-rbac-proxy" not found
Sep 12 00:33:42.418359 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:42.418107    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:33:42.418359 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:33:42.418155    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls podName:87ebd59e-1ecc-4723-853e-89b54ab190a7 nodeName:}" failed. No retries permitted until 2026-09-12 00:33:50.418139431 +0000 UTC m=+262.622140249 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ckmv2" (UID: "87ebd59e-1ecc-4723-853e-89b54ab190a7") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:33:49.767010 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:49.766953    2567 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch" podUID="0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 12 00:33:50.484950 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:50.484852    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:50.484950 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:50.484907    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:50.485407 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:50.485384    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87ebd59e-1ecc-4723-853e-89b54ab190a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:50.487257 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:50.487239    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87ebd59e-1ecc-4723-853e-89b54ab190a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ckmv2\" (UID: \"87ebd59e-1ecc-4723-853e-89b54ab190a7\") " pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:50.572200 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:50.572151    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ckmv2"
Sep 12 00:33:50.840398 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:50.840307    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ckmv2"]
Sep 12 00:33:50.844438 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:33:50.844410    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ebd59e_1ecc_4723_853e_89b54ab190a7.slice/crio-c9f70b75ae3acf36ac0ffb6432c06c2dff0498be3b5a4bef8997f38975f92ff2 WatchSource:0}: Error finding container c9f70b75ae3acf36ac0ffb6432c06c2dff0498be3b5a4bef8997f38975f92ff2: Status 404 returned error can't find the container with id c9f70b75ae3acf36ac0ffb6432c06c2dff0498be3b5a4bef8997f38975f92ff2
Sep 12 00:33:51.208875 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:51.208789    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ckmv2" event={"ID":"87ebd59e-1ecc-4723-853e-89b54ab190a7","Type":"ContainerStarted","Data":"b547864dc86e907f4a9f64cc007787f926d023a7e19e04b3d76f37f4042d04e3"}
Sep 12 00:33:51.208875 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:51.208852    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ckmv2" event={"ID":"87ebd59e-1ecc-4723-853e-89b54ab190a7","Type":"ContainerStarted","Data":"c9f70b75ae3acf36ac0ffb6432c06c2dff0498be3b5a4bef8997f38975f92ff2"}
Sep 12 00:33:52.214297 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:52.214252    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ckmv2" event={"ID":"87ebd59e-1ecc-4723-853e-89b54ab190a7","Type":"ContainerStarted","Data":"4979953e9cdbae332538f33d90898a3bd0e7f371bd729456ab865dae16a65a3b"}
Sep 12 00:33:53.219938 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:53.219906    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ckmv2" event={"ID":"87ebd59e-1ecc-4723-853e-89b54ab190a7","Type":"ContainerStarted","Data":"44c646268759d4c23fdbbcff1270ab95de506113b0c324255ea7a95c3295ac71"}
Sep 12 00:33:53.236630 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:53.236538    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ckmv2" podStartSLOduration=17.319883859 podStartE2EDuration="19.236520151s" totalImagesPullingTime="1.916636292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:33:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:33:53.236355927 +0000 UTC m=+265.440356748" watchObservedRunningTime="2026-09-12 00:33:53.236520151 +0000 UTC m=+265.440520970"
Sep 12 00:33:57.796493 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.796448    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cmtqp"]
Sep 12 00:33:57.798659 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.798642    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cmtqp"
Sep 12 00:33:57.802384 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.802363    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zx5lz\""
Sep 12 00:33:57.802384 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.802374    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 12 00:33:57.810451 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.810426    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cmtqp"]
Sep 12 00:33:57.840890 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.840853    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/411e7923-ac1f-4e50-8ea0-d4119750a83e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cmtqp\" (UID: \"411e7923-ac1f-4e50-8ea0-d4119750a83e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cmtqp"
Sep 12 00:33:57.920764 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.920727    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7db65b44b-pjgst"]
Sep 12 00:33:57.922859 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.922843    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:57.925745 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.925720    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 12 00:33:57.925884 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.925765    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cg9mt\""
Sep 12 00:33:57.925884 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.925795    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 12 00:33:57.925884 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.925795    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 12 00:33:57.930877 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.930855    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 12 00:33:57.938161 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.938139    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7db65b44b-pjgst"]
Sep 12 00:33:57.941429 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.941401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-ca-trust-extracted\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:57.941530 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.941442    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-registry-tls\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:57.941530 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.941489    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-registry-certificates\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:57.941635 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.941618    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-installation-pull-secrets\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:57.941679 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.941649    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvmh\" (UniqueName: \"kubernetes.io/projected/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-kube-api-access-mmvmh\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:57.941746 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.941726    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-image-registry-private-configuration\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:57.941792 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.941767    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-trusted-ca\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:57.941836 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.941814    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-bound-sa-token\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:57.941885 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.941848    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/411e7923-ac1f-4e50-8ea0-d4119750a83e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cmtqp\" (UID: \"411e7923-ac1f-4e50-8ea0-d4119750a83e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cmtqp"
Sep 12 00:33:57.944363 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:57.944341    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/411e7923-ac1f-4e50-8ea0-d4119750a83e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cmtqp\" (UID: \"411e7923-ac1f-4e50-8ea0-d4119750a83e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cmtqp"
Sep 12 00:33:58.042720 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.042679    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-installation-pull-secrets\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.042720 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.042716    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvmh\" (UniqueName: \"kubernetes.io/projected/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-kube-api-access-mmvmh\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.043022 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.042749    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-image-registry-private-configuration\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.043022 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.042770    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-trusted-ca\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.043022 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.042790    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-bound-sa-token\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.043022 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.042827    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-ca-trust-extracted\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.043022 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.042846    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-registry-tls\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.043022 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.042860    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-registry-certificates\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.043348 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.043316    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-ca-trust-extracted\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.043808 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.043786    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-trusted-ca\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.043900 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.043808    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-registry-certificates\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.045678 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.045655    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-image-registry-private-configuration\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.045791 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.045677    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-registry-tls\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.045791 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.045749    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-installation-pull-secrets\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.051689 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.051634    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-bound-sa-token\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.052163 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.052146    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvmh\" (UniqueName: \"kubernetes.io/projected/4bb66d73-4366-4eb7-a858-c353d4d4dcb9-kube-api-access-mmvmh\") pod \"image-registry-7db65b44b-pjgst\" (UID: \"4bb66d73-4366-4eb7-a858-c353d4d4dcb9\") " pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.108090 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.108054    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cmtqp"
Sep 12 00:33:58.232186 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.232140    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:58.377440 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.377272    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cmtqp"]
Sep 12 00:33:58.380298 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:33:58.380264    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411e7923_ac1f_4e50_8ea0_d4119750a83e.slice/crio-5632e5db8fd70893016a20fe8dab0a1d34fc4d9153db991eba33b1743366e954 WatchSource:0}: Error finding container 5632e5db8fd70893016a20fe8dab0a1d34fc4d9153db991eba33b1743366e954: Status 404 returned error can't find the container with id 5632e5db8fd70893016a20fe8dab0a1d34fc4d9153db991eba33b1743366e954
Sep 12 00:33:58.505860 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:58.505829    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7db65b44b-pjgst"]
Sep 12 00:33:58.508607 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:33:58.508557    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb66d73_4366_4eb7_a858_c353d4d4dcb9.slice/crio-b67b1611aa3568da4b41e1ddff5f975e06ff13ecec640ad03c15c8f298c843ea WatchSource:0}: Error finding container b67b1611aa3568da4b41e1ddff5f975e06ff13ecec640ad03c15c8f298c843ea: Status 404 returned error can't find the container with id b67b1611aa3568da4b41e1ddff5f975e06ff13ecec640ad03c15c8f298c843ea
Sep 12 00:33:59.242983 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:59.242945    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db65b44b-pjgst" event={"ID":"4bb66d73-4366-4eb7-a858-c353d4d4dcb9","Type":"ContainerStarted","Data":"e6ad350ccde9adddcb5ab62dc008811c6e576cd540c0bcc55b03ef767abaaa9b"}
Sep 12 00:33:59.243488 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:59.243011    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db65b44b-pjgst" event={"ID":"4bb66d73-4366-4eb7-a858-c353d4d4dcb9","Type":"ContainerStarted","Data":"b67b1611aa3568da4b41e1ddff5f975e06ff13ecec640ad03c15c8f298c843ea"}
Sep 12 00:33:59.243488 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:59.243036    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:33:59.244463 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:59.244430    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cmtqp" event={"ID":"411e7923-ac1f-4e50-8ea0-d4119750a83e","Type":"ContainerStarted","Data":"5632e5db8fd70893016a20fe8dab0a1d34fc4d9153db991eba33b1743366e954"}
Sep 12 00:33:59.261945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:59.261891    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7db65b44b-pjgst" podStartSLOduration=2.261873921 podStartE2EDuration="2.261873921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:33:59.260479643 +0000 UTC m=+271.464480466" watchObservedRunningTime="2026-09-12 00:33:59.261873921 +0000 UTC m=+271.465874744"
Sep 12 00:33:59.766647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:33:59.766607    2567 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch" podUID="0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 12 00:34:00.254239 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:00.254194    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cmtqp" event={"ID":"411e7923-ac1f-4e50-8ea0-d4119750a83e","Type":"ContainerStarted","Data":"e88cdba1c2332efe6c91378d15cf5086f5dcd674afb71dc186a4e49dafdc2b97"}
Sep 12 00:34:00.254705 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:00.254408    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cmtqp"
Sep 12 00:34:00.258911 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:00.258885    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cmtqp"
Sep 12 00:34:00.281631 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:00.281562    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cmtqp" podStartSLOduration=2.269987741 podStartE2EDuration="3.28154571s" totalImagesPullingTime="1.011557969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:34:00.268078271 +0000 UTC m=+272.472079093" watchObservedRunningTime="2026-09-12 00:34:00.28154571 +0000 UTC m=+272.485546531"
Sep 12 00:34:05.078094 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.078056    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jd297"]
Sep 12 00:34:05.081178 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.081157    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.083875 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.083847    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 12 00:34:05.084601 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.084561    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 12 00:34:05.084765 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.084582    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rm5ll\""
Sep 12 00:34:05.084765 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.084585    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 12 00:34:05.084915 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.084587    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 12 00:34:05.084915 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.084611    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 12 00:34:05.084915 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.084663    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 12 00:34:05.098275 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.098240    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-tls\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.098421 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.098283    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ba1acf3-4610-4e04-988d-c716bde57c28-metrics-client-ca\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.098421 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.098314    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.098421 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.098373    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba1acf3-4610-4e04-988d-c716bde57c28-sys\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.098421 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.098399    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ba1acf3-4610-4e04-988d-c716bde57c28-root\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.098617 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.098445    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-wtmp\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.098617 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.098468    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-accelerators-collector-config\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.098617 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.098490    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-textfile\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.098617 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.098508    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxpnn\" (UniqueName: \"kubernetes.io/projected/5ba1acf3-4610-4e04-988d-c716bde57c28-kube-api-access-mxpnn\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.199214 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199171    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-wtmp\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.199214 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199217    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-accelerators-collector-config\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.199482 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199240    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-textfile\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.199482 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199260    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxpnn\" (UniqueName: \"kubernetes.io/projected/5ba1acf3-4610-4e04-988d-c716bde57c28-kube-api-access-mxpnn\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.199482 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199311    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-tls\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.199482 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199344    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ba1acf3-4610-4e04-988d-c716bde57c28-metrics-client-ca\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.199482 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199371    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.199482 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199391    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-wtmp\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.199482 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199401    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba1acf3-4610-4e04-988d-c716bde57c28-sys\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.199482 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199443    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ba1acf3-4610-4e04-988d-c716bde57c28-root\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.199893 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199523    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ba1acf3-4610-4e04-988d-c716bde57c28-root\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.199893 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199559    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba1acf3-4610-4e04-988d-c716bde57c28-sys\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.199893 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199698    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-textfile\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.200026 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199959    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-accelerators-collector-config\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.200026 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.199972    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ba1acf3-4610-4e04-988d-c716bde57c28-metrics-client-ca\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.201750 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.201731    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.201840 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.201823    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ba1acf3-4610-4e04-988d-c716bde57c28-node-exporter-tls\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.206413 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.206391    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxpnn\" (UniqueName: \"kubernetes.io/projected/5ba1acf3-4610-4e04-988d-c716bde57c28-kube-api-access-mxpnn\") pod \"node-exporter-jd297\" (UID: \"5ba1acf3-4610-4e04-988d-c716bde57c28\") " pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.391120 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:05.391031    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jd297"
Sep 12 00:34:05.401272 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:34:05.401243    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba1acf3_4610_4e04_988d_c716bde57c28.slice/crio-3458484089ecc6e6c52908f2bae3b3ba3f341c11aa4c4e2f07471b99759397f5 WatchSource:0}: Error finding container 3458484089ecc6e6c52908f2bae3b3ba3f341c11aa4c4e2f07471b99759397f5: Status 404 returned error can't find the container with id 3458484089ecc6e6c52908f2bae3b3ba3f341c11aa4c4e2f07471b99759397f5
Sep 12 00:34:06.278467 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:06.278407    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jd297" event={"ID":"5ba1acf3-4610-4e04-988d-c716bde57c28","Type":"ContainerStarted","Data":"3458484089ecc6e6c52908f2bae3b3ba3f341c11aa4c4e2f07471b99759397f5"}
Sep 12 00:34:07.283172 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:07.283131    2567 generic.go:356] "Generic (PLEG): container finished" podID="5ba1acf3-4610-4e04-988d-c716bde57c28" containerID="0881eededcc2908ff93530f6d6a8f9cc0d35c6e3f7d0e7ef15049b4f82245fac" exitCode=0
Sep 12 00:34:07.283559 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:07.283218    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jd297" event={"ID":"5ba1acf3-4610-4e04-988d-c716bde57c28","Type":"ContainerDied","Data":"0881eededcc2908ff93530f6d6a8f9cc0d35c6e3f7d0e7ef15049b4f82245fac"}
Sep 12 00:34:08.287923 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:08.287882    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jd297" event={"ID":"5ba1acf3-4610-4e04-988d-c716bde57c28","Type":"ContainerStarted","Data":"39f4596b5f02858c5b5a2f4fbe41e53f7963bc88d49ee71d1deb4640442409f3"}
Sep 12 00:34:08.288318 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:08.287948    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jd297" event={"ID":"5ba1acf3-4610-4e04-988d-c716bde57c28","Type":"ContainerStarted","Data":"cade611cb2112ff327dde1959ca6fd11768ed66219fb01467b5b09d5a4424104"}
Sep 12 00:34:08.306446 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:08.306391    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jd297" podStartSLOduration=2.243972844 podStartE2EDuration="3.306373682s" totalImagesPullingTime="999.57534ms" totalInitContainerRuntime="62.825498ms" isStatefulPod=true podCreationTimestamp="2026-09-12 00:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:34:08.304902412 +0000 UTC m=+280.508903234" watchObservedRunningTime="2026-09-12 00:34:08.306373682 +0000 UTC m=+280.510374503"
Sep 12 00:34:08.893708 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:34:08.893659    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-p94bw" podUID="5c884033-b8a0-4f39-a5b7-d5006599e229"
Sep 12 00:34:09.291158 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:09.291126    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p94bw"
Sep 12 00:34:09.767496 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:09.767450    2567 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch" podUID="0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 12 00:34:09.767680 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:09.767527    2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch"
Sep 12 00:34:09.768038 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:09.768003    2567 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"2f525d97f0c68e856c38feaa713f63eb3ea5b6f6e07a96ef7449421050d32227"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 12 00:34:09.768101 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:09.768057    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch" podUID="0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf" containerName="service-proxy" containerID="cri-o://2f525d97f0c68e856c38feaa713f63eb3ea5b6f6e07a96ef7449421050d32227" gracePeriod=30
Sep 12 00:34:10.297544 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:10.297508    2567 generic.go:356] "Generic (PLEG): container finished" podID="0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf" containerID="2f525d97f0c68e856c38feaa713f63eb3ea5b6f6e07a96ef7449421050d32227" exitCode=2
Sep 12 00:34:10.297960 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:10.297585    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch" event={"ID":"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf","Type":"ContainerDied","Data":"2f525d97f0c68e856c38feaa713f63eb3ea5b6f6e07a96ef7449421050d32227"}
Sep 12 00:34:10.297960 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:10.297623    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ff596bd-dj9ch" event={"ID":"0abb5b27-d1bc-42e6-ab0e-985ca62bfbdf","Type":"ContainerStarted","Data":"05ce7c438c7f1176999891c0bfc144c9dc14f430fc903a2877c1423257574f95"}
Sep 12 00:34:12.250713 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:12.250655    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:34:12.252998 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:12.252974    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c884033-b8a0-4f39-a5b7-d5006599e229-metrics-tls\") pod \"dns-default-p94bw\" (UID: \"5c884033-b8a0-4f39-a5b7-d5006599e229\") " pod="openshift-dns/dns-default-p94bw"
Sep 12 00:34:12.294866 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:12.294837    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h7jvg\""
Sep 12 00:34:12.302628 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:12.302604    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p94bw"
Sep 12 00:34:12.571943 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:12.571864    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p94bw"]
Sep 12 00:34:12.574985 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:34:12.574949    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c884033_b8a0_4f39_a5b7_d5006599e229.slice/crio-c29c71635527c041863d5ce4a1baa8a3fe2ed56bc764541c2b67903e5d22a52f WatchSource:0}: Error finding container c29c71635527c041863d5ce4a1baa8a3fe2ed56bc764541c2b67903e5d22a52f: Status 404 returned error can't find the container with id c29c71635527c041863d5ce4a1baa8a3fe2ed56bc764541c2b67903e5d22a52f
Sep 12 00:34:13.310337 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:13.310295    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p94bw" event={"ID":"5c884033-b8a0-4f39-a5b7-d5006599e229","Type":"ContainerStarted","Data":"c29c71635527c041863d5ce4a1baa8a3fe2ed56bc764541c2b67903e5d22a52f"}
Sep 12 00:34:15.320290 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:15.320255    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p94bw" event={"ID":"5c884033-b8a0-4f39-a5b7-d5006599e229","Type":"ContainerStarted","Data":"45c0337139f5a1db2eb01ebf89036c94d939680a53f3ab470e5601762d85f635"}
Sep 12 00:34:15.320744 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:15.320312    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p94bw" event={"ID":"5c884033-b8a0-4f39-a5b7-d5006599e229","Type":"ContainerStarted","Data":"01909a1a7461a766a4efffcb75b86323eb6942d782311d7a87fd2e8a9ccad333"}
Sep 12 00:34:15.320744 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:15.320370    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p94bw"
Sep 12 00:34:15.337534 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:15.337478    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p94bw" podStartSLOduration=251.523527372 podStartE2EDuration="4m13.337459631s" totalImagesPullingTime="1.813932259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:34:15.335927711 +0000 UTC m=+287.539928532" watchObservedRunningTime="2026-09-12 00:34:15.337459631 +0000 UTC m=+287.541460452"
Sep 12 00:34:20.259124 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:20.259092    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7db65b44b-pjgst"
Sep 12 00:34:21.268783 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:34:21.268726    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-qvqwv" podUID="0a3b6564-fae6-476b-8e7d-003d4beaf870"
Sep 12 00:34:25.329413 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:25.329371    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p94bw"
Sep 12 00:34:28.188171 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:28.188138    2567 kubelet.go:1648] "Image garbage collection succeeded"
Sep 12 00:34:34.267952 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:34:34.267906    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:35:41.219866 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:35:41.219821    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:35:41.222116 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:35:41.222094    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3b6564-fae6-476b-8e7d-003d4beaf870-metrics-certs\") pod \"network-metrics-daemon-qvqwv\" (UID: \"0a3b6564-fae6-476b-8e7d-003d4beaf870\") " pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:35:41.471994 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:35:41.471909    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8bjlg\""
Sep 12 00:35:41.480103 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:35:41.480068    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvqwv"
Sep 12 00:35:41.747020 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:35:41.746927    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvqwv"]
Sep 12 00:35:41.750785 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:35:41.750742    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3b6564_fae6_476b_8e7d_003d4beaf870.slice/crio-9c5254f41bee97577869502402bec5407a6a825f4c70a57c04a8105da722d2cb WatchSource:0}: Error finding container 9c5254f41bee97577869502402bec5407a6a825f4c70a57c04a8105da722d2cb: Status 404 returned error can't find the container with id 9c5254f41bee97577869502402bec5407a6a825f4c70a57c04a8105da722d2cb
Sep 12 00:35:41.752781 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:35:41.752760    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:35:42.608534 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:35:42.608444    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvqwv" event={"ID":"0a3b6564-fae6-476b-8e7d-003d4beaf870","Type":"ContainerStarted","Data":"9c5254f41bee97577869502402bec5407a6a825f4c70a57c04a8105da722d2cb"}
Sep 12 00:35:43.613995 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:35:43.613951    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvqwv" event={"ID":"0a3b6564-fae6-476b-8e7d-003d4beaf870","Type":"ContainerStarted","Data":"c41648794828481051985b01ea98fef8332bbf5956fbfc130ac4aa08447b7cbc"}
Sep 12 00:35:43.614480 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:35:43.614047    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvqwv" event={"ID":"0a3b6564-fae6-476b-8e7d-003d4beaf870","Type":"ContainerStarted","Data":"d11d075edd717372a57655925d7aae9674ef81a52239c0a9ca1556daa0943db1"}
Sep 12 00:35:43.628256 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:35:43.628205    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qvqwv" podStartSLOduration=374.705156693 podStartE2EDuration="6m15.628190516s" totalImagesPullingTime="923.033823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:35:43.627555642 +0000 UTC m=+375.831556463" watchObservedRunningTime="2026-09-12 00:35:43.628190516 +0000 UTC m=+375.832191365"
Sep 12 00:38:36.528305 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.528265    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"]
Sep 12 00:38:36.531207 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.531189    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:36.535244 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.535220    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 12 00:38:36.535244 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.535239    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 12 00:38:36.535441 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.535276    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rkprs\""
Sep 12 00:38:36.535441 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.535241    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 12 00:38:36.535441 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.535223    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:38:36.535441 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.535221    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 12 00:38:36.542927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.542897    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"]
Sep 12 00:38:36.674790 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.674754    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad2b397a-80b9-47ae-b920-7b052d4c49ac-cert\") pod \"lws-controller-manager-7bc7647c9b-4q9nd\" (UID: \"ad2b397a-80b9-47ae-b920-7b052d4c49ac\") " pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:36.674962 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.674804    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64dzr\" (UniqueName: \"kubernetes.io/projected/ad2b397a-80b9-47ae-b920-7b052d4c49ac-kube-api-access-64dzr\") pod \"lws-controller-manager-7bc7647c9b-4q9nd\" (UID: \"ad2b397a-80b9-47ae-b920-7b052d4c49ac\") " pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:36.674962 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.674919    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ad2b397a-80b9-47ae-b920-7b052d4c49ac-manager-config\") pod \"lws-controller-manager-7bc7647c9b-4q9nd\" (UID: \"ad2b397a-80b9-47ae-b920-7b052d4c49ac\") " pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:36.674962 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.674953    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad2b397a-80b9-47ae-b920-7b052d4c49ac-metrics-cert\") pod \"lws-controller-manager-7bc7647c9b-4q9nd\" (UID: \"ad2b397a-80b9-47ae-b920-7b052d4c49ac\") " pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:36.776041 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.775995    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad2b397a-80b9-47ae-b920-7b052d4c49ac-cert\") pod \"lws-controller-manager-7bc7647c9b-4q9nd\" (UID: \"ad2b397a-80b9-47ae-b920-7b052d4c49ac\") " pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:36.776169 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.776068    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64dzr\" (UniqueName: \"kubernetes.io/projected/ad2b397a-80b9-47ae-b920-7b052d4c49ac-kube-api-access-64dzr\") pod \"lws-controller-manager-7bc7647c9b-4q9nd\" (UID: \"ad2b397a-80b9-47ae-b920-7b052d4c49ac\") " pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:36.776169 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.776105    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ad2b397a-80b9-47ae-b920-7b052d4c49ac-manager-config\") pod \"lws-controller-manager-7bc7647c9b-4q9nd\" (UID: \"ad2b397a-80b9-47ae-b920-7b052d4c49ac\") " pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:36.776169 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.776122    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad2b397a-80b9-47ae-b920-7b052d4c49ac-metrics-cert\") pod \"lws-controller-manager-7bc7647c9b-4q9nd\" (UID: \"ad2b397a-80b9-47ae-b920-7b052d4c49ac\") " pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:36.776844 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.776818    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ad2b397a-80b9-47ae-b920-7b052d4c49ac-manager-config\") pod \"lws-controller-manager-7bc7647c9b-4q9nd\" (UID: \"ad2b397a-80b9-47ae-b920-7b052d4c49ac\") " pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:36.778488 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.778436    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad2b397a-80b9-47ae-b920-7b052d4c49ac-cert\") pod \"lws-controller-manager-7bc7647c9b-4q9nd\" (UID: \"ad2b397a-80b9-47ae-b920-7b052d4c49ac\") " pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:36.778640 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.778618    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad2b397a-80b9-47ae-b920-7b052d4c49ac-metrics-cert\") pod \"lws-controller-manager-7bc7647c9b-4q9nd\" (UID: \"ad2b397a-80b9-47ae-b920-7b052d4c49ac\") " pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:36.791367 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.791330    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64dzr\" (UniqueName: \"kubernetes.io/projected/ad2b397a-80b9-47ae-b920-7b052d4c49ac-kube-api-access-64dzr\") pod \"lws-controller-manager-7bc7647c9b-4q9nd\" (UID: \"ad2b397a-80b9-47ae-b920-7b052d4c49ac\") " pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:36.840247 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:36.840209    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:37.122907 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:37.122826    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"]
Sep 12 00:38:37.126085 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:38:37.126060    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2b397a_80b9_47ae_b920_7b052d4c49ac.slice/crio-1163085f7b4cd4f9c01e673cf21ea033893268fa541d8d7933db46e29be8ba70 WatchSource:0}: Error finding container 1163085f7b4cd4f9c01e673cf21ea033893268fa541d8d7933db46e29be8ba70: Status 404 returned error can't find the container with id 1163085f7b4cd4f9c01e673cf21ea033893268fa541d8d7933db46e29be8ba70
Sep 12 00:38:37.195360 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:37.195328    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd" event={"ID":"ad2b397a-80b9-47ae-b920-7b052d4c49ac","Type":"ContainerStarted","Data":"1163085f7b4cd4f9c01e673cf21ea033893268fa541d8d7933db46e29be8ba70"}
Sep 12 00:38:40.212752 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:40.212704    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd" event={"ID":"ad2b397a-80b9-47ae-b920-7b052d4c49ac","Type":"ContainerStarted","Data":"ee78ce457c9b94ccf07a17202e6cf752860b65c69c4c03e02addc15baf9f016d"}
Sep 12 00:38:40.213283 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:40.212802    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:40.232932 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:40.232875    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd" podStartSLOduration=1.9404377739999998 podStartE2EDuration="4.232859563s" totalImagesPullingTime="2.292421789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:38:40.231504825 +0000 UTC m=+552.435505651" watchObservedRunningTime="2026-09-12 00:38:40.232859563 +0000 UTC m=+552.436860384"
Sep 12 00:38:41.900952 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:41.900916    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"]
Sep 12 00:38:41.904111 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:41.904094    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:41.912068 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:41.912043    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 12 00:38:41.912639 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:41.912620    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 12 00:38:41.914953 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:41.914934    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 12 00:38:41.915117 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:41.915101    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-g56gz\""
Sep 12 00:38:41.916270 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:41.916255    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 12 00:38:41.943884 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:41.943859    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"]
Sep 12 00:38:42.018248 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.018209    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m89b\" (UniqueName: \"kubernetes.io/projected/db0f42de-90a2-4018-be00-309da0fd271c-kube-api-access-4m89b\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.018248 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.018252    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db0f42de-90a2-4018-be00-309da0fd271c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.018463 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.018274    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db0f42de-90a2-4018-be00-309da0fd271c-manifests\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.018463 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.018340    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db0f42de-90a2-4018-be00-309da0fd271c-tmp\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.018463 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.018395    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db0f42de-90a2-4018-be00-309da0fd271c-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.119681 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.119647    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m89b\" (UniqueName: \"kubernetes.io/projected/db0f42de-90a2-4018-be00-309da0fd271c-kube-api-access-4m89b\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.119787 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.119695    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db0f42de-90a2-4018-be00-309da0fd271c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.119787 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.119713    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db0f42de-90a2-4018-be00-309da0fd271c-manifests\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.119787 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.119736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db0f42de-90a2-4018-be00-309da0fd271c-tmp\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.119928 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.119862    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db0f42de-90a2-4018-be00-309da0fd271c-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.120229 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.120201    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db0f42de-90a2-4018-be00-309da0fd271c-manifests\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.121970 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.121949    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db0f42de-90a2-4018-be00-309da0fd271c-tmp\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.122470 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.122449    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db0f42de-90a2-4018-be00-309da0fd271c-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.122534 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.122507    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db0f42de-90a2-4018-be00-309da0fd271c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.133910 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.133888    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m89b\" (UniqueName: \"kubernetes.io/projected/db0f42de-90a2-4018-be00-309da0fd271c-kube-api-access-4m89b\") pod \"opendatahub-operator-controller-manager-67b7744c54-mk6xw\" (UID: \"db0f42de-90a2-4018-be00-309da0fd271c\") " pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.213876 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.213790    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:42.493941 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:42.493908    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"]
Sep 12 00:38:42.497338 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:38:42.497303    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0f42de_90a2_4018_be00_309da0fd271c.slice/crio-2049b6bf1bc0d120e8a5fad54c404a11fc00b7ab7efd5f8a5641018b1b6d01a3 WatchSource:0}: Error finding container 2049b6bf1bc0d120e8a5fad54c404a11fc00b7ab7efd5f8a5641018b1b6d01a3: Status 404 returned error can't find the container with id 2049b6bf1bc0d120e8a5fad54c404a11fc00b7ab7efd5f8a5641018b1b6d01a3
Sep 12 00:38:43.224922 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:43.224882    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw" event={"ID":"db0f42de-90a2-4018-be00-309da0fd271c","Type":"ContainerStarted","Data":"2049b6bf1bc0d120e8a5fad54c404a11fc00b7ab7efd5f8a5641018b1b6d01a3"}
Sep 12 00:38:46.238478 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:46.238433    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw" event={"ID":"db0f42de-90a2-4018-be00-309da0fd271c","Type":"ContainerStarted","Data":"e0432f3710a2ecde5ef534feb9a56c8e798ed02d5a56118f1d3522da009fba0c"}
Sep 12 00:38:47.245626 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:47.245587    2567 generic.go:356] "Generic (PLEG): container finished" podID="db0f42de-90a2-4018-be00-309da0fd271c" containerID="e0432f3710a2ecde5ef534feb9a56c8e798ed02d5a56118f1d3522da009fba0c" exitCode=0
Sep 12 00:38:47.246003 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:47.245674    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw" event={"ID":"db0f42de-90a2-4018-be00-309da0fd271c","Type":"ContainerDied","Data":"e0432f3710a2ecde5ef534feb9a56c8e798ed02d5a56118f1d3522da009fba0c"}
Sep 12 00:38:48.251708 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:48.251675    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw" event={"ID":"db0f42de-90a2-4018-be00-309da0fd271c","Type":"ContainerStarted","Data":"02b623320a2b255d369dfcd4040eb4c81fc51cb05b1fc56bda0ae8d71c91a429"}
Sep 12 00:38:48.252111 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:48.251810    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:38:48.275616 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:48.275555    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw" podStartSLOduration=2.952864824 podStartE2EDuration="7.275539257s" totalImagesPullingTime="3.23558578s" totalInitContainerRuntime="1.087088653s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:38:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:38:48.274333125 +0000 UTC m=+560.478333941" watchObservedRunningTime="2026-09-12 00:38:48.275539257 +0000 UTC m=+560.479540078"
Sep 12 00:38:51.218913 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:51.218879    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7bc7647c9b-4q9nd"
Sep 12 00:38:59.257949 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:38:59.257917    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67b7744c54-mk6xw"
Sep 12 00:39:08.789152 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.789119    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"]
Sep 12 00:39:08.797551 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.797531    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.800933 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.800900    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 12 00:39:08.800933 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.800924    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 12 00:39:08.802060 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.802043    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kdt4f\""
Sep 12 00:39:08.802129 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.802060    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 12 00:39:08.810482 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.810460    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"]
Sep 12 00:39:08.812489 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.812465    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.812602 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.812508    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.812652 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.812597    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwk5z\" (UniqueName: \"kubernetes.io/projected/d5daf33e-366a-458a-9006-97e0e974afbb-kube-api-access-dwk5z\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.812652 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.812630    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.812652 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.812648    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5daf33e-366a-458a-9006-97e0e974afbb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.812779 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.812668    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5daf33e-366a-458a-9006-97e0e974afbb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.812779 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.812749    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.812841 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.812790    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.812841 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.812830    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5daf33e-366a-458a-9006-97e0e974afbb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.913730 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.913691    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.913730 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.913732    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.913951 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.913767    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwk5z\" (UniqueName: \"kubernetes.io/projected/d5daf33e-366a-458a-9006-97e0e974afbb-kube-api-access-dwk5z\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.913951 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.913892    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.913951 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.913929    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5daf33e-366a-458a-9006-97e0e974afbb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.913951 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.913946    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5daf33e-366a-458a-9006-97e0e974afbb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.914136 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.914018    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.914136 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.914082    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.914136 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.914117    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5daf33e-366a-458a-9006-97e0e974afbb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.914287 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.914216    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.914347 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.914315    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.914477 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.914456    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.914538 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.914488    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.914759 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.914741    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5daf33e-366a-458a-9006-97e0e974afbb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.916325 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.916303    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5daf33e-366a-458a-9006-97e0e974afbb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.916530 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.916515    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5daf33e-366a-458a-9006-97e0e974afbb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.923982 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.923960    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwk5z\" (UniqueName: \"kubernetes.io/projected/d5daf33e-366a-458a-9006-97e0e974afbb-kube-api-access-dwk5z\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:08.926998 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:08.926976    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5daf33e-366a-458a-9006-97e0e974afbb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw\" (UID: \"d5daf33e-366a-458a-9006-97e0e974afbb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:09.107264 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:09.107185    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:09.388739 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:09.388714    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"]
Sep 12 00:39:09.391092 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:39:09.391063    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5daf33e_366a_458a_9006_97e0e974afbb.slice/crio-436b9124c2153e2c1fee705c7c94591ca0ae30018cda7fe55b75ed04b39c115c WatchSource:0}: Error finding container 436b9124c2153e2c1fee705c7c94591ca0ae30018cda7fe55b75ed04b39c115c: Status 404 returned error can't find the container with id 436b9124c2153e2c1fee705c7c94591ca0ae30018cda7fe55b75ed04b39c115c
Sep 12 00:39:10.334052 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:10.334016    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw" event={"ID":"d5daf33e-366a-458a-9006-97e0e974afbb","Type":"ContainerStarted","Data":"436b9124c2153e2c1fee705c7c94591ca0ae30018cda7fe55b75ed04b39c115c"}
Sep 12 00:39:12.054676 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.054637    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-drtcs"]
Sep 12 00:39:12.059882 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.059857    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-drtcs"
Sep 12 00:39:12.062781 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.062752    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 12 00:39:12.062918 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.062818    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-kxs49\""
Sep 12 00:39:12.071089 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.071042    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-drtcs"]
Sep 12 00:39:12.141684 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.141644    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ce16dde-9fbf-4b0a-8148-6ff5955bf039-cert\") pod \"odh-model-controller-7998c67dc4-drtcs\" (UID: \"7ce16dde-9fbf-4b0a-8148-6ff5955bf039\") " pod="opendatahub/odh-model-controller-7998c67dc4-drtcs"
Sep 12 00:39:12.141851 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.141771    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48bcj\" (UniqueName: \"kubernetes.io/projected/7ce16dde-9fbf-4b0a-8148-6ff5955bf039-kube-api-access-48bcj\") pod \"odh-model-controller-7998c67dc4-drtcs\" (UID: \"7ce16dde-9fbf-4b0a-8148-6ff5955bf039\") " pod="opendatahub/odh-model-controller-7998c67dc4-drtcs"
Sep 12 00:39:12.242347 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.242303    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bcj\" (UniqueName: \"kubernetes.io/projected/7ce16dde-9fbf-4b0a-8148-6ff5955bf039-kube-api-access-48bcj\") pod \"odh-model-controller-7998c67dc4-drtcs\" (UID: \"7ce16dde-9fbf-4b0a-8148-6ff5955bf039\") " pod="opendatahub/odh-model-controller-7998c67dc4-drtcs"
Sep 12 00:39:12.242527 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.242391    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ce16dde-9fbf-4b0a-8148-6ff5955bf039-cert\") pod \"odh-model-controller-7998c67dc4-drtcs\" (UID: \"7ce16dde-9fbf-4b0a-8148-6ff5955bf039\") " pod="opendatahub/odh-model-controller-7998c67dc4-drtcs"
Sep 12 00:39:12.242527 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:39:12.242521    2567 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 12 00:39:12.242635 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:39:12.242613    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ce16dde-9fbf-4b0a-8148-6ff5955bf039-cert podName:7ce16dde-9fbf-4b0a-8148-6ff5955bf039 nodeName:}" failed. No retries permitted until 2026-09-12 00:39:12.742596187 +0000 UTC m=+584.946596988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7ce16dde-9fbf-4b0a-8148-6ff5955bf039-cert") pod "odh-model-controller-7998c67dc4-drtcs" (UID: "7ce16dde-9fbf-4b0a-8148-6ff5955bf039") : secret "odh-model-controller-webhook-cert" not found
Sep 12 00:39:12.252639 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.252607    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48bcj\" (UniqueName: \"kubernetes.io/projected/7ce16dde-9fbf-4b0a-8148-6ff5955bf039-kube-api-access-48bcj\") pod \"odh-model-controller-7998c67dc4-drtcs\" (UID: \"7ce16dde-9fbf-4b0a-8148-6ff5955bf039\") " pod="opendatahub/odh-model-controller-7998c67dc4-drtcs"
Sep 12 00:39:12.637871 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.637835    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-j7qcq"]
Sep 12 00:39:12.640789 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.640770    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-j7qcq"
Sep 12 00:39:12.645179 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.645159    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 12 00:39:12.661069 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.661045    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-j7qcq"]
Sep 12 00:39:12.694848 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.694808    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:39:12.694938 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.694882    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:39:12.694938 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.694911    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:39:12.746944 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.746919    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltvlg\" (UniqueName: \"kubernetes.io/projected/c1e9be4b-762a-4190-9cd4-6b4d65505de6-kube-api-access-ltvlg\") pod \"model-serving-api-54d89bd5fc-j7qcq\" (UID: \"c1e9be4b-762a-4190-9cd4-6b4d65505de6\") " pod="opendatahub/model-serving-api-54d89bd5fc-j7qcq"
Sep 12 00:39:12.747069 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.746957    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ce16dde-9fbf-4b0a-8148-6ff5955bf039-cert\") pod \"odh-model-controller-7998c67dc4-drtcs\" (UID: \"7ce16dde-9fbf-4b0a-8148-6ff5955bf039\") " pod="opendatahub/odh-model-controller-7998c67dc4-drtcs"
Sep 12 00:39:12.747069 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.746990    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e9be4b-762a-4190-9cd4-6b4d65505de6-tls-certs\") pod \"model-serving-api-54d89bd5fc-j7qcq\" (UID: \"c1e9be4b-762a-4190-9cd4-6b4d65505de6\") " pod="opendatahub/model-serving-api-54d89bd5fc-j7qcq"
Sep 12 00:39:12.749195 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.749173    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ce16dde-9fbf-4b0a-8148-6ff5955bf039-cert\") pod \"odh-model-controller-7998c67dc4-drtcs\" (UID: \"7ce16dde-9fbf-4b0a-8148-6ff5955bf039\") " pod="opendatahub/odh-model-controller-7998c67dc4-drtcs"
Sep 12 00:39:12.848077 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.848039    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e9be4b-762a-4190-9cd4-6b4d65505de6-tls-certs\") pod \"model-serving-api-54d89bd5fc-j7qcq\" (UID: \"c1e9be4b-762a-4190-9cd4-6b4d65505de6\") " pod="opendatahub/model-serving-api-54d89bd5fc-j7qcq"
Sep 12 00:39:12.848250 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.848223    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltvlg\" (UniqueName: \"kubernetes.io/projected/c1e9be4b-762a-4190-9cd4-6b4d65505de6-kube-api-access-ltvlg\") pod \"model-serving-api-54d89bd5fc-j7qcq\" (UID: \"c1e9be4b-762a-4190-9cd4-6b4d65505de6\") " pod="opendatahub/model-serving-api-54d89bd5fc-j7qcq"
Sep 12 00:39:12.850508 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.850487    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e9be4b-762a-4190-9cd4-6b4d65505de6-tls-certs\") pod \"model-serving-api-54d89bd5fc-j7qcq\" (UID: \"c1e9be4b-762a-4190-9cd4-6b4d65505de6\") " pod="opendatahub/model-serving-api-54d89bd5fc-j7qcq"
Sep 12 00:39:12.858081 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.858048    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltvlg\" (UniqueName: \"kubernetes.io/projected/c1e9be4b-762a-4190-9cd4-6b4d65505de6-kube-api-access-ltvlg\") pod \"model-serving-api-54d89bd5fc-j7qcq\" (UID: \"c1e9be4b-762a-4190-9cd4-6b4d65505de6\") " pod="opendatahub/model-serving-api-54d89bd5fc-j7qcq"
Sep 12 00:39:12.950762 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.950669    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-j7qcq"
Sep 12 00:39:12.973754 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:12.973720    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-drtcs"
Sep 12 00:39:13.236173 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:13.236138    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-j7qcq"]
Sep 12 00:39:13.239317 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:39:13.239282    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e9be4b_762a_4190_9cd4_6b4d65505de6.slice/crio-49e2a1cab316a6f7775e758e80fc79181e43323bfb844348605cd595539183ca WatchSource:0}: Error finding container 49e2a1cab316a6f7775e758e80fc79181e43323bfb844348605cd595539183ca: Status 404 returned error can't find the container with id 49e2a1cab316a6f7775e758e80fc79181e43323bfb844348605cd595539183ca
Sep 12 00:39:13.268147 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:13.268120    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-drtcs"]
Sep 12 00:39:13.271420 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:39:13.271394    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce16dde_9fbf_4b0a_8148_6ff5955bf039.slice/crio-cb8b72ae36e7958b37b4165a61913fb263aaa8f3324c4be4d5dcb09da56a67af WatchSource:0}: Error finding container cb8b72ae36e7958b37b4165a61913fb263aaa8f3324c4be4d5dcb09da56a67af: Status 404 returned error can't find the container with id cb8b72ae36e7958b37b4165a61913fb263aaa8f3324c4be4d5dcb09da56a67af
Sep 12 00:39:13.346874 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:13.346840    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw" event={"ID":"d5daf33e-366a-458a-9006-97e0e974afbb","Type":"ContainerStarted","Data":"65d9a95602ef51a886d635b9143d6d84f7677fec603a09d3edd7f051f5197559"}
Sep 12 00:39:13.348073 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:13.348043    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-drtcs" event={"ID":"7ce16dde-9fbf-4b0a-8148-6ff5955bf039","Type":"ContainerStarted","Data":"cb8b72ae36e7958b37b4165a61913fb263aaa8f3324c4be4d5dcb09da56a67af"}
Sep 12 00:39:13.349143 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:13.349119    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-j7qcq" event={"ID":"c1e9be4b-762a-4190-9cd4-6b4d65505de6","Type":"ContainerStarted","Data":"49e2a1cab316a6f7775e758e80fc79181e43323bfb844348605cd595539183ca"}
Sep 12 00:39:13.383162 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:13.383106    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw" podStartSLOduration=2.081684861 podStartE2EDuration="5.383088606s" totalImagesPullingTime="3.301403745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:39:13.382517658 +0000 UTC m=+585.586518491" watchObservedRunningTime="2026-09-12 00:39:13.383088606 +0000 UTC m=+585.587089427"
Sep 12 00:39:14.108321 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:14.108281    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:14.114603 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:14.114547    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:14.356683 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:14.356648    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:14.357956 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:14.357931    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw"
Sep 12 00:39:17.373309 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:17.373261    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-drtcs" event={"ID":"7ce16dde-9fbf-4b0a-8148-6ff5955bf039","Type":"ContainerStarted","Data":"4c5cd4a87ef390ddbc7e4a6d48c23334d941ecd82621b7fcab1d60fa6c4b5eff"}
Sep 12 00:39:17.373833 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:17.373341    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-drtcs"
Sep 12 00:39:17.375413 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:17.375381    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-j7qcq" event={"ID":"c1e9be4b-762a-4190-9cd4-6b4d65505de6","Type":"ContainerStarted","Data":"83fe7396eaf284d9d08e6ba4828c05d9103155787434b891a183698d4326f765"}
Sep 12 00:39:17.375628 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:17.375520    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-j7qcq"
Sep 12 00:39:17.466632 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:17.466564    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-drtcs" podStartSLOduration=1.742824387 podStartE2EDuration="5.466544472s" totalImagesPullingTime="3.723720085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:39:17.421825247 +0000 UTC m=+589.625826067" watchObservedRunningTime="2026-09-12 00:39:17.466544472 +0000 UTC m=+589.670545324"
Sep 12 00:39:17.466817 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:17.466665    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-j7qcq" podStartSLOduration=1.757960162 podStartE2EDuration="5.466660404s" totalImagesPullingTime="3.708700242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:39:17.462742596 +0000 UTC m=+589.666743418" watchObservedRunningTime="2026-09-12 00:39:17.466660404 +0000 UTC m=+589.670661225"
Sep 12 00:39:21.956624 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:21.956564    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"]
Sep 12 00:39:21.960027 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:21.960002    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"
Sep 12 00:39:21.963444 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:21.963420    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gntmw\""
Sep 12 00:39:21.963618 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:21.963595    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 12 00:39:21.981585 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:21.981544    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"]
Sep 12 00:39:22.023900 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:22.023861    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc2sh\" (UniqueName: \"kubernetes.io/projected/482daf06-10b6-4385-95eb-2895bce457fd-kube-api-access-fc2sh\") pod \"kserve-controller-manager-d6f4b65b8-2xddl\" (UID: \"482daf06-10b6-4385-95eb-2895bce457fd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"
Sep 12 00:39:22.024094 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:22.023951    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/482daf06-10b6-4385-95eb-2895bce457fd-cert\") pod \"kserve-controller-manager-d6f4b65b8-2xddl\" (UID: \"482daf06-10b6-4385-95eb-2895bce457fd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"
Sep 12 00:39:22.125302 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:22.125247    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc2sh\" (UniqueName: \"kubernetes.io/projected/482daf06-10b6-4385-95eb-2895bce457fd-kube-api-access-fc2sh\") pod \"kserve-controller-manager-d6f4b65b8-2xddl\" (UID: \"482daf06-10b6-4385-95eb-2895bce457fd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"
Sep 12 00:39:22.125492 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:22.125320    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/482daf06-10b6-4385-95eb-2895bce457fd-cert\") pod \"kserve-controller-manager-d6f4b65b8-2xddl\" (UID: \"482daf06-10b6-4385-95eb-2895bce457fd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"
Sep 12 00:39:22.125492 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:39:22.125413    2567 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 12 00:39:22.125492 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:39:22.125469    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/482daf06-10b6-4385-95eb-2895bce457fd-cert podName:482daf06-10b6-4385-95eb-2895bce457fd nodeName:}" failed. No retries permitted until 2026-09-12 00:39:22.625452562 +0000 UTC m=+594.829453361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/482daf06-10b6-4385-95eb-2895bce457fd-cert") pod "kserve-controller-manager-d6f4b65b8-2xddl" (UID: "482daf06-10b6-4385-95eb-2895bce457fd") : secret "kserve-webhook-server-cert" not found
Sep 12 00:39:22.137372 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:22.137339    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc2sh\" (UniqueName: \"kubernetes.io/projected/482daf06-10b6-4385-95eb-2895bce457fd-kube-api-access-fc2sh\") pod \"kserve-controller-manager-d6f4b65b8-2xddl\" (UID: \"482daf06-10b6-4385-95eb-2895bce457fd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"
Sep 12 00:39:22.630010 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:22.629970    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/482daf06-10b6-4385-95eb-2895bce457fd-cert\") pod \"kserve-controller-manager-d6f4b65b8-2xddl\" (UID: \"482daf06-10b6-4385-95eb-2895bce457fd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"
Sep 12 00:39:22.632363 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:22.632344    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/482daf06-10b6-4385-95eb-2895bce457fd-cert\") pod \"kserve-controller-manager-d6f4b65b8-2xddl\" (UID: \"482daf06-10b6-4385-95eb-2895bce457fd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"
Sep 12 00:39:22.870157 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:22.870119    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"
Sep 12 00:39:23.140332 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:23.140292    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"]
Sep 12 00:39:23.145102 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:39:23.145073    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482daf06_10b6_4385_95eb_2895bce457fd.slice/crio-acde3f9d5f80ff4a8bf14061843d4424da137b5ff6ba8f2b83ba3ab76dbe3571 WatchSource:0}: Error finding container acde3f9d5f80ff4a8bf14061843d4424da137b5ff6ba8f2b83ba3ab76dbe3571: Status 404 returned error can't find the container with id acde3f9d5f80ff4a8bf14061843d4424da137b5ff6ba8f2b83ba3ab76dbe3571
Sep 12 00:39:23.403019 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:23.402982    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl" event={"ID":"482daf06-10b6-4385-95eb-2895bce457fd","Type":"ContainerStarted","Data":"acde3f9d5f80ff4a8bf14061843d4424da137b5ff6ba8f2b83ba3ab76dbe3571"}
Sep 12 00:39:25.411995 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:25.411957    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl" event={"ID":"482daf06-10b6-4385-95eb-2895bce457fd","Type":"ContainerStarted","Data":"f08c5c4213e6ce69231d5a97518fe5c6cb72050fd8d504698e2fcaaed029b44a"}
Sep 12 00:39:25.412481 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:25.412021    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"
Sep 12 00:39:25.430678 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:25.430624    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl" podStartSLOduration=2.308844132 podStartE2EDuration="4.430606042s" totalImagesPullingTime="2.12176191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:39:25.429884838 +0000 UTC m=+597.633885659" watchObservedRunningTime="2026-09-12 00:39:25.430606042 +0000 UTC m=+597.634606863"
Sep 12 00:39:28.382599 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:28.382553    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-drtcs"
Sep 12 00:39:28.384313 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:28.384288    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-j7qcq"
Sep 12 00:39:56.421472 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:39:56.421389    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2xddl"
Sep 12 00:40:34.933154 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:34.933114    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"]
Sep 12 00:40:34.937421 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:34.937380    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"
Sep 12 00:40:34.940351 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:34.940317    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-w5lxg\""
Sep 12 00:40:34.940499 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:34.940351    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 12 00:40:34.941299 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:34.941282    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 12 00:40:34.946266 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:34.946243    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"]
Sep 12 00:40:35.013161 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:35.013121    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5b8j\" (UniqueName: \"kubernetes.io/projected/6116dfc0-f305-4d27-81a1-5fc2e9bb47a4-kube-api-access-g5b8j\") pod \"limitador-operator-controller-manager-74495c69f8-rqsd2\" (UID: \"6116dfc0-f305-4d27-81a1-5fc2e9bb47a4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"
Sep 12 00:40:35.114189 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:35.114151    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5b8j\" (UniqueName: \"kubernetes.io/projected/6116dfc0-f305-4d27-81a1-5fc2e9bb47a4-kube-api-access-g5b8j\") pod \"limitador-operator-controller-manager-74495c69f8-rqsd2\" (UID: \"6116dfc0-f305-4d27-81a1-5fc2e9bb47a4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"
Sep 12 00:40:35.127459 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:35.127432    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5b8j\" (UniqueName: \"kubernetes.io/projected/6116dfc0-f305-4d27-81a1-5fc2e9bb47a4-kube-api-access-g5b8j\") pod \"limitador-operator-controller-manager-74495c69f8-rqsd2\" (UID: \"6116dfc0-f305-4d27-81a1-5fc2e9bb47a4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"
Sep 12 00:40:35.248721 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:35.248679    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"
Sep 12 00:40:35.536538 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:35.536462    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"]
Sep 12 00:40:35.540121 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:40:35.540094    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6116dfc0_f305_4d27_81a1_5fc2e9bb47a4.slice/crio-30c220a2a9101b9d6716ecdfaed3c10239769ac3c27ff6661b4058849bac2958 WatchSource:0}: Error finding container 30c220a2a9101b9d6716ecdfaed3c10239769ac3c27ff6661b4058849bac2958: Status 404 returned error can't find the container with id 30c220a2a9101b9d6716ecdfaed3c10239769ac3c27ff6661b4058849bac2958
Sep 12 00:40:35.677235 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:35.677196    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2" event={"ID":"6116dfc0-f305-4d27-81a1-5fc2e9bb47a4","Type":"ContainerStarted","Data":"30c220a2a9101b9d6716ecdfaed3c10239769ac3c27ff6661b4058849bac2958"}
Sep 12 00:40:38.694821 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:38.694782    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2" event={"ID":"6116dfc0-f305-4d27-81a1-5fc2e9bb47a4","Type":"ContainerStarted","Data":"6471cfd8465966686eed4bfdb20c056c4e244fc3e811542f1e8178be828c8464"}
Sep 12 00:40:38.695233 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:38.694933    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"
Sep 12 00:40:38.712369 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:38.712312    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2" podStartSLOduration=2.085986843 podStartE2EDuration="4.712295177s" totalImagesPullingTime="2.626308334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:40:38.71095199 +0000 UTC m=+670.914952839" watchObservedRunningTime="2026-09-12 00:40:38.712295177 +0000 UTC m=+670.916296004"
Sep 12 00:40:39.896295 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:39.896258    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"]
Sep 12 00:40:39.899621 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:39.899599    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"
Sep 12 00:40:39.901929 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:39.901901    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 12 00:40:39.902051 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:39.901976    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 12 00:40:39.902051 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:39.902011    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mhmr6\""
Sep 12 00:40:39.909358 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:39.909331    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"]
Sep 12 00:40:40.056653 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:40.056611    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9ffx\" (UniqueName: \"kubernetes.io/projected/a6132da6-fa63-4f8b-a6ac-8022da8818b3-kube-api-access-r9ffx\") pod \"kuadrant-console-plugin-fbdd4f445-7v8nj\" (UID: \"a6132da6-fa63-4f8b-a6ac-8022da8818b3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"
Sep 12 00:40:40.056829 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:40.056706    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6132da6-fa63-4f8b-a6ac-8022da8818b3-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7v8nj\" (UID: \"a6132da6-fa63-4f8b-a6ac-8022da8818b3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"
Sep 12 00:40:40.056829 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:40.056751    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a6132da6-fa63-4f8b-a6ac-8022da8818b3-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7v8nj\" (UID: \"a6132da6-fa63-4f8b-a6ac-8022da8818b3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"
Sep 12 00:40:40.157159 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:40.157069    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6132da6-fa63-4f8b-a6ac-8022da8818b3-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7v8nj\" (UID: \"a6132da6-fa63-4f8b-a6ac-8022da8818b3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"
Sep 12 00:40:40.157159 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:40.157122    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a6132da6-fa63-4f8b-a6ac-8022da8818b3-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7v8nj\" (UID: \"a6132da6-fa63-4f8b-a6ac-8022da8818b3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"
Sep 12 00:40:40.157347 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:40.157185    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9ffx\" (UniqueName: \"kubernetes.io/projected/a6132da6-fa63-4f8b-a6ac-8022da8818b3-kube-api-access-r9ffx\") pod \"kuadrant-console-plugin-fbdd4f445-7v8nj\" (UID: \"a6132da6-fa63-4f8b-a6ac-8022da8818b3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"
Sep 12 00:40:40.157851 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:40.157826    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a6132da6-fa63-4f8b-a6ac-8022da8818b3-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7v8nj\" (UID: \"a6132da6-fa63-4f8b-a6ac-8022da8818b3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"
Sep 12 00:40:40.159548 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:40.159518    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6132da6-fa63-4f8b-a6ac-8022da8818b3-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7v8nj\" (UID: \"a6132da6-fa63-4f8b-a6ac-8022da8818b3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"
Sep 12 00:40:40.165699 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:40.165677    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9ffx\" (UniqueName: \"kubernetes.io/projected/a6132da6-fa63-4f8b-a6ac-8022da8818b3-kube-api-access-r9ffx\") pod \"kuadrant-console-plugin-fbdd4f445-7v8nj\" (UID: \"a6132da6-fa63-4f8b-a6ac-8022da8818b3\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"
Sep 12 00:40:40.208655 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:40.208621    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"
Sep 12 00:40:40.483687 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:40.483550    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj"]
Sep 12 00:40:40.486361 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:40:40.486332    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6132da6_fa63_4f8b_a6ac_8022da8818b3.slice/crio-21e7851c05c204693ce50b212870bba02b6b2d4e49c16905deb768c7a898679d WatchSource:0}: Error finding container 21e7851c05c204693ce50b212870bba02b6b2d4e49c16905deb768c7a898679d: Status 404 returned error can't find the container with id 21e7851c05c204693ce50b212870bba02b6b2d4e49c16905deb768c7a898679d
Sep 12 00:40:40.703977 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:40.703939    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj" event={"ID":"a6132da6-fa63-4f8b-a6ac-8022da8818b3","Type":"ContainerStarted","Data":"21e7851c05c204693ce50b212870bba02b6b2d4e49c16905deb768c7a898679d"}
Sep 12 00:40:45.729289 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:45.729243    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj" event={"ID":"a6132da6-fa63-4f8b-a6ac-8022da8818b3","Type":"ContainerStarted","Data":"19b4e4f8c989c108f8657322b0f8762dab8964fed95e76efa9ae99a8ace08eb9"}
Sep 12 00:40:45.745784 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:45.745724    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7v8nj" podStartSLOduration=2.251498841 podStartE2EDuration="6.74570619s" totalImagesPullingTime="4.494207349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:40:45.744283128 +0000 UTC m=+677.948283951" watchObservedRunningTime="2026-09-12 00:40:45.74570619 +0000 UTC m=+677.949707012"
Sep 12 00:40:49.701628 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:49.701592    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"
Sep 12 00:40:50.548042 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.547978    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"]
Sep 12 00:40:50.548685 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.548636    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2" podUID="6116dfc0-f305-4d27-81a1-5fc2e9bb47a4" containerName="manager" containerID="cri-o://6471cfd8465966686eed4bfdb20c056c4e244fc3e811542f1e8178be828c8464" gracePeriod=2
Sep 12 00:40:50.558102 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.558066    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"]
Sep 12 00:40:50.569889 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.569860    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l9g84"]
Sep 12 00:40:50.570156 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.570143    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6116dfc0-f305-4d27-81a1-5fc2e9bb47a4" containerName="manager"
Sep 12 00:40:50.570156 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.570157    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6116dfc0-f305-4d27-81a1-5fc2e9bb47a4" containerName="manager"
Sep 12 00:40:50.570238 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.570207    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6116dfc0-f305-4d27-81a1-5fc2e9bb47a4" containerName="manager"
Sep 12 00:40:50.573323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.573301    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l9g84"
Sep 12 00:40:50.583780 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.583743    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l9g84"]
Sep 12 00:40:50.599216 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:40:50.599170    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rqsd2\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="6116dfc0-f305-4d27-81a1-5fc2e9bb47a4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"
Sep 12 00:40:50.751055 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.751018    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrw8m\" (UniqueName: \"kubernetes.io/projected/e5db7ffa-9634-4653-af5a-ef87004060fd-kube-api-access-jrw8m\") pod \"limitador-operator-controller-manager-74495c69f8-l9g84\" (UID: \"e5db7ffa-9634-4653-af5a-ef87004060fd\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l9g84"
Sep 12 00:40:50.754655 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.754626    2567 generic.go:356] "Generic (PLEG): container finished" podID="6116dfc0-f305-4d27-81a1-5fc2e9bb47a4" containerID="6471cfd8465966686eed4bfdb20c056c4e244fc3e811542f1e8178be828c8464" exitCode=0
Sep 12 00:40:50.788277 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.788245    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"
Sep 12 00:40:50.790753 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:40:50.790724    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rqsd2\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="6116dfc0-f305-4d27-81a1-5fc2e9bb47a4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"
Sep 12 00:40:50.852407 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.852312    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrw8m\" (UniqueName: \"kubernetes.io/projected/e5db7ffa-9634-4653-af5a-ef87004060fd-kube-api-access-jrw8m\") pod \"limitador-operator-controller-manager-74495c69f8-l9g84\" (UID: \"e5db7ffa-9634-4653-af5a-ef87004060fd\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l9g84"
Sep 12 00:40:50.879626 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.879564    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrw8m\" (UniqueName: \"kubernetes.io/projected/e5db7ffa-9634-4653-af5a-ef87004060fd-kube-api-access-jrw8m\") pod \"limitador-operator-controller-manager-74495c69f8-l9g84\" (UID: \"e5db7ffa-9634-4653-af5a-ef87004060fd\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l9g84"
Sep 12 00:40:50.914203 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.914167    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l9g84"
Sep 12 00:40:50.952895 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.952854    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5b8j\" (UniqueName: \"kubernetes.io/projected/6116dfc0-f305-4d27-81a1-5fc2e9bb47a4-kube-api-access-g5b8j\") pod \"6116dfc0-f305-4d27-81a1-5fc2e9bb47a4\" (UID: \"6116dfc0-f305-4d27-81a1-5fc2e9bb47a4\") "
Sep 12 00:40:50.955021 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:50.954991    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6116dfc0-f305-4d27-81a1-5fc2e9bb47a4-kube-api-access-g5b8j" (OuterVolumeSpecName: "kube-api-access-g5b8j") pod "6116dfc0-f305-4d27-81a1-5fc2e9bb47a4" (UID: "6116dfc0-f305-4d27-81a1-5fc2e9bb47a4"). InnerVolumeSpecName "kube-api-access-g5b8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:40:51.054517 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:51.054458    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5b8j\" (UniqueName: \"kubernetes.io/projected/6116dfc0-f305-4d27-81a1-5fc2e9bb47a4-kube-api-access-g5b8j\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:40:51.193154 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:51.193075    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l9g84"]
Sep 12 00:40:51.196096 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:40:51.196064    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5db7ffa_9634_4653_af5a_ef87004060fd.slice/crio-c503bfd0ffdbe359693e5d325223707775cbb54b45ca528bb55bc43a04baa404 WatchSource:0}: Error finding container c503bfd0ffdbe359693e5d325223707775cbb54b45ca528bb55bc43a04baa404: Status 404 returned error can't find the container with id c503bfd0ffdbe359693e5d325223707775cbb54b45ca528bb55bc43a04baa404
Sep 12 00:40:51.760932 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:51.760881    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l9g84" event={"ID":"e5db7ffa-9634-4653-af5a-ef87004060fd","Type":"ContainerStarted","Data":"8b33a6036e74421fa357ff65a4a0d4d883f6abdc42ae47e16a76987396b920d1"}
Sep 12 00:40:51.761376 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:51.760944    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l9g84" event={"ID":"e5db7ffa-9634-4653-af5a-ef87004060fd","Type":"ContainerStarted","Data":"c503bfd0ffdbe359693e5d325223707775cbb54b45ca528bb55bc43a04baa404"}
Sep 12 00:40:51.761376 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:51.760973    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l9g84"
Sep 12 00:40:51.762273 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:51.762252    2567 scope.go:117] "RemoveContainer" containerID="6471cfd8465966686eed4bfdb20c056c4e244fc3e811542f1e8178be828c8464"
Sep 12 00:40:51.762342 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:51.762275    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"
Sep 12 00:40:51.763191 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:40:51.763155    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rqsd2\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="6116dfc0-f305-4d27-81a1-5fc2e9bb47a4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"
Sep 12 00:40:51.784810 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:51.784748    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l9g84" podStartSLOduration=1.7847273179999998 podStartE2EDuration="1.784727318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:40:51.783819777 +0000 UTC m=+683.987820600" watchObservedRunningTime="2026-09-12 00:40:51.784727318 +0000 UTC m=+683.988728140"
Sep 12 00:40:51.786023 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:40:51.786000    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rqsd2\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="6116dfc0-f305-4d27-81a1-5fc2e9bb47a4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rqsd2"
Sep 12 00:40:52.272795 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:40:52.272756    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6116dfc0-f305-4d27-81a1-5fc2e9bb47a4" path="/var/lib/kubelet/pods/6116dfc0-f305-4d27-81a1-5fc2e9bb47a4/volumes"
Sep 12 00:41:02.770655 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:02.770619    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l9g84"
Sep 12 00:41:18.850180 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:18.850140    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"]
Sep 12 00:41:18.870022 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:18.869989    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"]
Sep 12 00:41:18.870194 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:18.870118    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:18.872715 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:18.872691    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kb54n\""
Sep 12 00:41:18.967363 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:18.967314    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:18.967363 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:18.967359    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:18.967647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:18.967384    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:18.967647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:18.967402    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:18.967647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:18.967485    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:18.967647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:18.967524    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:18.967647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:18.967594    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bpnb\" (UniqueName: \"kubernetes.io/projected/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-kube-api-access-7bpnb\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:18.967647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:18.967632    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:18.967862 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:18.967662    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.068739 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.068695    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bpnb\" (UniqueName: \"kubernetes.io/projected/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-kube-api-access-7bpnb\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.068949 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.068751    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.068949 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.068879    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.068949 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.068942    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.069108 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.068960    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.069108 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.068986    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.069108 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.069002    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.069108 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.069028    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.069108 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.069050    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.069427 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.069404    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.069517 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.069484    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.069589 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.069518    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.069809 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.069789    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.069855 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.069809    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.071208 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.071180    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.071593 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.071546    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.077164 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.077140    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.077299 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.077276    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bpnb\" (UniqueName: \"kubernetes.io/projected/e8f16670-c284-4e2e-b6b4-fe54a025c6a9-kube-api-access-7bpnb\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cvpvm\" (UID: \"e8f16670-c284-4e2e-b6b4-fe54a025c6a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.181173 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.181075    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:19.452930 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.452743    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"]
Sep 12 00:41:19.455662 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:41:19.455629    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f16670_c284_4e2e_b6b4_fe54a025c6a9.slice/crio-1306c99c8836df8fe9eb6afde963e5ac8f7fe7eda9825c54f1ac664a6b12ebde WatchSource:0}: Error finding container 1306c99c8836df8fe9eb6afde963e5ac8f7fe7eda9825c54f1ac664a6b12ebde: Status 404 returned error can't find the container with id 1306c99c8836df8fe9eb6afde963e5ac8f7fe7eda9825c54f1ac664a6b12ebde
Sep 12 00:41:19.457809 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.457773    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:41:19.457900 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.457856    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:41:19.457900 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.457890    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:41:19.884565 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.884524    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm" event={"ID":"e8f16670-c284-4e2e-b6b4-fe54a025c6a9","Type":"ContainerStarted","Data":"c9d1b5183e14dff84cd2737922d5ec7396ff9ede3c391319b7dd639fc3ccf2c4"}
Sep 12 00:41:19.885028 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.884606    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm" event={"ID":"e8f16670-c284-4e2e-b6b4-fe54a025c6a9","Type":"ContainerStarted","Data":"1306c99c8836df8fe9eb6afde963e5ac8f7fe7eda9825c54f1ac664a6b12ebde"}
Sep 12 00:41:19.904058 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:19.903997    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm" podStartSLOduration=1.9039814750000001 podStartE2EDuration="1.903981475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:41:19.90255058 +0000 UTC m=+712.106551402" watchObservedRunningTime="2026-09-12 00:41:19.903981475 +0000 UTC m=+712.107982343"
Sep 12 00:41:20.181642 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:20.181536    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:20.186364 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:20.186333    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:20.888326 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:20.888285    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:20.889217 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:20.889187    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:41:36.289730 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.289691    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8b8p6"]
Sep 12 00:41:36.293850 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.293829    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8b8p6"
Sep 12 00:41:36.296247 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.296218    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rf82r\""
Sep 12 00:41:36.299695 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.299664    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8b8p6"]
Sep 12 00:41:36.400364 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.400324    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-qlkcp"]
Sep 12 00:41:36.403421 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.403400    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qlkcp"
Sep 12 00:41:36.410859 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.410832    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qlkcp"]
Sep 12 00:41:36.414367 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.414338    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82l7v\" (UniqueName: \"kubernetes.io/projected/d19af9c3-d6ad-4906-8c6c-6387e9e5462e-kube-api-access-82l7v\") pod \"authorino-7bcfd8f9cd-8b8p6\" (UID: \"d19af9c3-d6ad-4906-8c6c-6387e9e5462e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8b8p6"
Sep 12 00:41:36.515311 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.515273    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6bh9\" (UniqueName: \"kubernetes.io/projected/6d2386a3-9b18-48b8-854f-f258feb412b7-kube-api-access-g6bh9\") pod \"authorino-59c974bd48-qlkcp\" (UID: \"6d2386a3-9b18-48b8-854f-f258feb412b7\") " pod="kuadrant-system/authorino-59c974bd48-qlkcp"
Sep 12 00:41:36.515490 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.515343    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82l7v\" (UniqueName: \"kubernetes.io/projected/d19af9c3-d6ad-4906-8c6c-6387e9e5462e-kube-api-access-82l7v\") pod \"authorino-7bcfd8f9cd-8b8p6\" (UID: \"d19af9c3-d6ad-4906-8c6c-6387e9e5462e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8b8p6"
Sep 12 00:41:36.523320 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.523293    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82l7v\" (UniqueName: \"kubernetes.io/projected/d19af9c3-d6ad-4906-8c6c-6387e9e5462e-kube-api-access-82l7v\") pod \"authorino-7bcfd8f9cd-8b8p6\" (UID: \"d19af9c3-d6ad-4906-8c6c-6387e9e5462e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8b8p6"
Sep 12 00:41:36.604205 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.604120    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8b8p6"
Sep 12 00:41:36.616431 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.616398    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6bh9\" (UniqueName: \"kubernetes.io/projected/6d2386a3-9b18-48b8-854f-f258feb412b7-kube-api-access-g6bh9\") pod \"authorino-59c974bd48-qlkcp\" (UID: \"6d2386a3-9b18-48b8-854f-f258feb412b7\") " pod="kuadrant-system/authorino-59c974bd48-qlkcp"
Sep 12 00:41:36.625549 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.625485    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6bh9\" (UniqueName: \"kubernetes.io/projected/6d2386a3-9b18-48b8-854f-f258feb412b7-kube-api-access-g6bh9\") pod \"authorino-59c974bd48-qlkcp\" (UID: \"6d2386a3-9b18-48b8-854f-f258feb412b7\") " pod="kuadrant-system/authorino-59c974bd48-qlkcp"
Sep 12 00:41:36.712880 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.712839    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qlkcp"
Sep 12 00:41:36.874228 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.874149    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8b8p6"]
Sep 12 00:41:36.877485 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:41:36.877456    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd19af9c3_d6ad_4906_8c6c_6387e9e5462e.slice/crio-483ba621c3f134fdbd2c4c7235987a79f7e39ab2dabd9b3ded7f1f0189a36502 WatchSource:0}: Error finding container 483ba621c3f134fdbd2c4c7235987a79f7e39ab2dabd9b3ded7f1f0189a36502: Status 404 returned error can't find the container with id 483ba621c3f134fdbd2c4c7235987a79f7e39ab2dabd9b3ded7f1f0189a36502
Sep 12 00:41:36.878779 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.878761    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:41:36.955427 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.955381    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8b8p6" event={"ID":"d19af9c3-d6ad-4906-8c6c-6387e9e5462e","Type":"ContainerStarted","Data":"483ba621c3f134fdbd2c4c7235987a79f7e39ab2dabd9b3ded7f1f0189a36502"}
Sep 12 00:41:36.979830 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:36.979795    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qlkcp"]
Sep 12 00:41:36.982586 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:41:36.982550    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d2386a3_9b18_48b8_854f_f258feb412b7.slice/crio-4993c01c5bee8240634911fa65644ccbd126bea607230e20d26c8ee82a6d2512 WatchSource:0}: Error finding container 4993c01c5bee8240634911fa65644ccbd126bea607230e20d26c8ee82a6d2512: Status 404 returned error can't find the container with id 4993c01c5bee8240634911fa65644ccbd126bea607230e20d26c8ee82a6d2512
Sep 12 00:41:37.963754 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:37.963708    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qlkcp" event={"ID":"6d2386a3-9b18-48b8-854f-f258feb412b7","Type":"ContainerStarted","Data":"4993c01c5bee8240634911fa65644ccbd126bea607230e20d26c8ee82a6d2512"}
Sep 12 00:41:38.969336 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:38.969294    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8b8p6" event={"ID":"d19af9c3-d6ad-4906-8c6c-6387e9e5462e","Type":"ContainerStarted","Data":"665807df33a6b276f495a7bf8ca518b2471f9ce0ad9f52e2497598e3893b482c"}
Sep 12 00:41:38.971001 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:38.970971    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qlkcp" event={"ID":"6d2386a3-9b18-48b8-854f-f258feb412b7","Type":"ContainerStarted","Data":"1a82935fa680041af396d92b7e2238365129aceed24214332d0140fd8934e3f1"}
Sep 12 00:41:38.982943 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:38.982882    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-8b8p6" podStartSLOduration=1.091587344 podStartE2EDuration="2.982862617s" totalImagesPullingTime="1.891275273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:41:38.982293257 +0000 UTC m=+731.186294078" watchObservedRunningTime="2026-09-12 00:41:38.982862617 +0000 UTC m=+731.186863439"
Sep 12 00:41:39.000247 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:39.000167    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-qlkcp" podStartSLOduration=1.209431452 podStartE2EDuration="3.000140125s" totalImagesPullingTime="1.790708673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:41:38.997774878 +0000 UTC m=+731.201775703" watchObservedRunningTime="2026-09-12 00:41:39.000140125 +0000 UTC m=+731.204140968"
Sep 12 00:41:39.021775 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:39.021689    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8b8p6"]
Sep 12 00:41:40.979123 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:40.979069    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-8b8p6" podUID="d19af9c3-d6ad-4906-8c6c-6387e9e5462e" containerName="authorino" containerID="cri-o://665807df33a6b276f495a7bf8ca518b2471f9ce0ad9f52e2497598e3893b482c" gracePeriod=30
Sep 12 00:41:41.238924 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:41.238847    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8b8p6"
Sep 12 00:41:41.354513 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:41.354472    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82l7v\" (UniqueName: \"kubernetes.io/projected/d19af9c3-d6ad-4906-8c6c-6387e9e5462e-kube-api-access-82l7v\") pod \"d19af9c3-d6ad-4906-8c6c-6387e9e5462e\" (UID: \"d19af9c3-d6ad-4906-8c6c-6387e9e5462e\") "
Sep 12 00:41:41.356634 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:41.356596    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19af9c3-d6ad-4906-8c6c-6387e9e5462e-kube-api-access-82l7v" (OuterVolumeSpecName: "kube-api-access-82l7v") pod "d19af9c3-d6ad-4906-8c6c-6387e9e5462e" (UID: "d19af9c3-d6ad-4906-8c6c-6387e9e5462e"). InnerVolumeSpecName "kube-api-access-82l7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:41:41.456041 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:41.455994    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82l7v\" (UniqueName: \"kubernetes.io/projected/d19af9c3-d6ad-4906-8c6c-6387e9e5462e-kube-api-access-82l7v\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:41:41.984591 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:41.984531    2567 generic.go:356] "Generic (PLEG): container finished" podID="d19af9c3-d6ad-4906-8c6c-6387e9e5462e" containerID="665807df33a6b276f495a7bf8ca518b2471f9ce0ad9f52e2497598e3893b482c" exitCode=0
Sep 12 00:41:41.985018 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:41.984598    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8b8p6"
Sep 12 00:41:41.985018 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:41.984615    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8b8p6" event={"ID":"d19af9c3-d6ad-4906-8c6c-6387e9e5462e","Type":"ContainerDied","Data":"665807df33a6b276f495a7bf8ca518b2471f9ce0ad9f52e2497598e3893b482c"}
Sep 12 00:41:41.985018 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:41.984663    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8b8p6" event={"ID":"d19af9c3-d6ad-4906-8c6c-6387e9e5462e","Type":"ContainerDied","Data":"483ba621c3f134fdbd2c4c7235987a79f7e39ab2dabd9b3ded7f1f0189a36502"}
Sep 12 00:41:41.985018 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:41.984681    2567 scope.go:117] "RemoveContainer" containerID="665807df33a6b276f495a7bf8ca518b2471f9ce0ad9f52e2497598e3893b482c"
Sep 12 00:41:41.993035 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:41.993015    2567 scope.go:117] "RemoveContainer" containerID="665807df33a6b276f495a7bf8ca518b2471f9ce0ad9f52e2497598e3893b482c"
Sep 12 00:41:41.993381 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:41:41.993356    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665807df33a6b276f495a7bf8ca518b2471f9ce0ad9f52e2497598e3893b482c\": container with ID starting with 665807df33a6b276f495a7bf8ca518b2471f9ce0ad9f52e2497598e3893b482c not found: ID does not exist" containerID="665807df33a6b276f495a7bf8ca518b2471f9ce0ad9f52e2497598e3893b482c"
Sep 12 00:41:41.993442 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:41.993395    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665807df33a6b276f495a7bf8ca518b2471f9ce0ad9f52e2497598e3893b482c"} err="failed to get container status \"665807df33a6b276f495a7bf8ca518b2471f9ce0ad9f52e2497598e3893b482c\": rpc error: code = NotFound desc = could not find container \"665807df33a6b276f495a7bf8ca518b2471f9ce0ad9f52e2497598e3893b482c\": container with ID starting with 665807df33a6b276f495a7bf8ca518b2471f9ce0ad9f52e2497598e3893b482c not found: ID does not exist"
Sep 12 00:41:42.004713 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:42.004678    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8b8p6"]
Sep 12 00:41:42.007621 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:42.007565    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8b8p6"]
Sep 12 00:41:42.273317 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:41:42.273281    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d19af9c3-d6ad-4906-8c6c-6387e9e5462e" path="/var/lib/kubelet/pods/d19af9c3-d6ad-4906-8c6c-6387e9e5462e/volumes"
Sep 12 00:42:05.185102 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.185063    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-ftk6p"]
Sep 12 00:42:05.185646 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.185369    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d19af9c3-d6ad-4906-8c6c-6387e9e5462e" containerName="authorino"
Sep 12 00:42:05.185646 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.185379    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d19af9c3-d6ad-4906-8c6c-6387e9e5462e" containerName="authorino"
Sep 12 00:42:05.185646 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.185425    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d19af9c3-d6ad-4906-8c6c-6387e9e5462e" containerName="authorino"
Sep 12 00:42:05.188150 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.188132    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ftk6p"
Sep 12 00:42:05.195014 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.194986    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-ftk6p"]
Sep 12 00:42:05.250882 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.250846    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ftk6p"]
Sep 12 00:42:05.251123 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:42:05.251094    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-r7nst], unattached volumes=[], failed to process volumes=[kube-api-access-r7nst]: context canceled" pod="kuadrant-system/authorino-557f876f69-ftk6p" podUID="13771957-b24a-4aa8-ac55-eb173bf270bc"
Sep 12 00:42:05.260810 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.260781    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7nst\" (UniqueName: \"kubernetes.io/projected/13771957-b24a-4aa8-ac55-eb173bf270bc-kube-api-access-r7nst\") pod \"authorino-557f876f69-ftk6p\" (UID: \"13771957-b24a-4aa8-ac55-eb173bf270bc\") " pod="kuadrant-system/authorino-557f876f69-ftk6p"
Sep 12 00:42:05.274238 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.274202    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ctvv9"]
Sep 12 00:42:05.277422 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.277405    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ctvv9"
Sep 12 00:42:05.279780 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.279759    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 12 00:42:05.286434 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.286411    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ctvv9"]
Sep 12 00:42:05.361890 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.361850    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65a5f76e-80da-4bdc-a259-494563c97775-tls-cert\") pod \"authorino-6d4dfc649f-ctvv9\" (UID: \"65a5f76e-80da-4bdc-a259-494563c97775\") " pod="kuadrant-system/authorino-6d4dfc649f-ctvv9"
Sep 12 00:42:05.362066 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.361913    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dthqh\" (UniqueName: \"kubernetes.io/projected/65a5f76e-80da-4bdc-a259-494563c97775-kube-api-access-dthqh\") pod \"authorino-6d4dfc649f-ctvv9\" (UID: \"65a5f76e-80da-4bdc-a259-494563c97775\") " pod="kuadrant-system/authorino-6d4dfc649f-ctvv9"
Sep 12 00:42:05.362066 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.361965    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nst\" (UniqueName: \"kubernetes.io/projected/13771957-b24a-4aa8-ac55-eb173bf270bc-kube-api-access-r7nst\") pod \"authorino-557f876f69-ftk6p\" (UID: \"13771957-b24a-4aa8-ac55-eb173bf270bc\") " pod="kuadrant-system/authorino-557f876f69-ftk6p"
Sep 12 00:42:05.370012 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.369981    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nst\" (UniqueName: \"kubernetes.io/projected/13771957-b24a-4aa8-ac55-eb173bf270bc-kube-api-access-r7nst\") pod \"authorino-557f876f69-ftk6p\" (UID: \"13771957-b24a-4aa8-ac55-eb173bf270bc\") " pod="kuadrant-system/authorino-557f876f69-ftk6p"
Sep 12 00:42:05.427470 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.427429    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ctvv9"]
Sep 12 00:42:05.427752 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:42:05.427732    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dthqh tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-ctvv9" podUID="65a5f76e-80da-4bdc-a259-494563c97775"
Sep 12 00:42:05.458266 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.458178    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b67b89f7f-qj4t8"]
Sep 12 00:42:05.461625 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.461605    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b67b89f7f-qj4t8"
Sep 12 00:42:05.462799 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.462773    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dthqh\" (UniqueName: \"kubernetes.io/projected/65a5f76e-80da-4bdc-a259-494563c97775-kube-api-access-dthqh\") pod \"authorino-6d4dfc649f-ctvv9\" (UID: \"65a5f76e-80da-4bdc-a259-494563c97775\") " pod="kuadrant-system/authorino-6d4dfc649f-ctvv9"
Sep 12 00:42:05.462922 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.462863    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65a5f76e-80da-4bdc-a259-494563c97775-tls-cert\") pod \"authorino-6d4dfc649f-ctvv9\" (UID: \"65a5f76e-80da-4bdc-a259-494563c97775\") " pod="kuadrant-system/authorino-6d4dfc649f-ctvv9"
Sep 12 00:42:05.465294 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.465265    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65a5f76e-80da-4bdc-a259-494563c97775-tls-cert\") pod \"authorino-6d4dfc649f-ctvv9\" (UID: \"65a5f76e-80da-4bdc-a259-494563c97775\") " pod="kuadrant-system/authorino-6d4dfc649f-ctvv9"
Sep 12 00:42:05.474217 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.473970    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dthqh\" (UniqueName: \"kubernetes.io/projected/65a5f76e-80da-4bdc-a259-494563c97775-kube-api-access-dthqh\") pod \"authorino-6d4dfc649f-ctvv9\" (UID: \"65a5f76e-80da-4bdc-a259-494563c97775\") " pod="kuadrant-system/authorino-6d4dfc649f-ctvv9"
Sep 12 00:42:05.476989 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.476959    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b67b89f7f-qj4t8"]
Sep 12 00:42:05.563870 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.563830    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/856b7345-50af-4e17-b908-0220753f59fa-tls-cert\") pod \"authorino-7b67b89f7f-qj4t8\" (UID: \"856b7345-50af-4e17-b908-0220753f59fa\") " pod="kuadrant-system/authorino-7b67b89f7f-qj4t8"
Sep 12 00:42:05.563870 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.563877    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-857p5\" (UniqueName: \"kubernetes.io/projected/856b7345-50af-4e17-b908-0220753f59fa-kube-api-access-857p5\") pod \"authorino-7b67b89f7f-qj4t8\" (UID: \"856b7345-50af-4e17-b908-0220753f59fa\") " pod="kuadrant-system/authorino-7b67b89f7f-qj4t8"
Sep 12 00:42:05.664527 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.664476    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/856b7345-50af-4e17-b908-0220753f59fa-tls-cert\") pod \"authorino-7b67b89f7f-qj4t8\" (UID: \"856b7345-50af-4e17-b908-0220753f59fa\") " pod="kuadrant-system/authorino-7b67b89f7f-qj4t8"
Sep 12 00:42:05.664527 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.664533    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-857p5\" (UniqueName: \"kubernetes.io/projected/856b7345-50af-4e17-b908-0220753f59fa-kube-api-access-857p5\") pod \"authorino-7b67b89f7f-qj4t8\" (UID: \"856b7345-50af-4e17-b908-0220753f59fa\") " pod="kuadrant-system/authorino-7b67b89f7f-qj4t8"
Sep 12 00:42:05.666947 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.666915    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/856b7345-50af-4e17-b908-0220753f59fa-tls-cert\") pod \"authorino-7b67b89f7f-qj4t8\" (UID: \"856b7345-50af-4e17-b908-0220753f59fa\") " pod="kuadrant-system/authorino-7b67b89f7f-qj4t8"
Sep 12 00:42:05.672015 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.671982    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-857p5\" (UniqueName: \"kubernetes.io/projected/856b7345-50af-4e17-b908-0220753f59fa-kube-api-access-857p5\") pod \"authorino-7b67b89f7f-qj4t8\" (UID: \"856b7345-50af-4e17-b908-0220753f59fa\") " pod="kuadrant-system/authorino-7b67b89f7f-qj4t8"
Sep 12 00:42:05.788063 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:05.788029    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b67b89f7f-qj4t8"
Sep 12 00:42:06.056800 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.056710    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b67b89f7f-qj4t8"]
Sep 12 00:42:06.060585 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:42:06.060545    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod856b7345_50af_4e17_b908_0220753f59fa.slice/crio-c4b545646eaf115c66d68a1f85fa87110e7eecca16116957386d70ee00fed1a3 WatchSource:0}: Error finding container c4b545646eaf115c66d68a1f85fa87110e7eecca16116957386d70ee00fed1a3: Status 404 returned error can't find the container with id c4b545646eaf115c66d68a1f85fa87110e7eecca16116957386d70ee00fed1a3
Sep 12 00:42:06.086230 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.086189    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b67b89f7f-qj4t8" event={"ID":"856b7345-50af-4e17-b908-0220753f59fa","Type":"ContainerStarted","Data":"c4b545646eaf115c66d68a1f85fa87110e7eecca16116957386d70ee00fed1a3"}
Sep 12 00:42:06.086230 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.086220    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ctvv9"
Sep 12 00:42:06.086450 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.086239    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ftk6p"
Sep 12 00:42:06.094681 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.094649    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ctvv9"
Sep 12 00:42:06.096683 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.096666    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ftk6p"
Sep 12 00:42:06.169053 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.169017    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dthqh\" (UniqueName: \"kubernetes.io/projected/65a5f76e-80da-4bdc-a259-494563c97775-kube-api-access-dthqh\") pod \"65a5f76e-80da-4bdc-a259-494563c97775\" (UID: \"65a5f76e-80da-4bdc-a259-494563c97775\") "
Sep 12 00:42:06.169258 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.169062    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65a5f76e-80da-4bdc-a259-494563c97775-tls-cert\") pod \"65a5f76e-80da-4bdc-a259-494563c97775\" (UID: \"65a5f76e-80da-4bdc-a259-494563c97775\") "
Sep 12 00:42:06.169258 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.169094    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7nst\" (UniqueName: \"kubernetes.io/projected/13771957-b24a-4aa8-ac55-eb173bf270bc-kube-api-access-r7nst\") pod \"13771957-b24a-4aa8-ac55-eb173bf270bc\" (UID: \"13771957-b24a-4aa8-ac55-eb173bf270bc\") "
Sep 12 00:42:06.171305 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.171266    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a5f76e-80da-4bdc-a259-494563c97775-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "65a5f76e-80da-4bdc-a259-494563c97775" (UID: "65a5f76e-80da-4bdc-a259-494563c97775"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:42:06.171413 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.171309    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13771957-b24a-4aa8-ac55-eb173bf270bc-kube-api-access-r7nst" (OuterVolumeSpecName: "kube-api-access-r7nst") pod "13771957-b24a-4aa8-ac55-eb173bf270bc" (UID: "13771957-b24a-4aa8-ac55-eb173bf270bc"). InnerVolumeSpecName "kube-api-access-r7nst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:42:06.171413 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.171325    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a5f76e-80da-4bdc-a259-494563c97775-kube-api-access-dthqh" (OuterVolumeSpecName: "kube-api-access-dthqh") pod "65a5f76e-80da-4bdc-a259-494563c97775" (UID: "65a5f76e-80da-4bdc-a259-494563c97775"). InnerVolumeSpecName "kube-api-access-dthqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:42:06.269750 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.269719    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dthqh\" (UniqueName: \"kubernetes.io/projected/65a5f76e-80da-4bdc-a259-494563c97775-kube-api-access-dthqh\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:42:06.269750 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.269747    2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65a5f76e-80da-4bdc-a259-494563c97775-tls-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:42:06.269750 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:06.269759    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7nst\" (UniqueName: \"kubernetes.io/projected/13771957-b24a-4aa8-ac55-eb173bf270bc-kube-api-access-r7nst\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:42:07.091456 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.091409    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b67b89f7f-qj4t8" event={"ID":"856b7345-50af-4e17-b908-0220753f59fa","Type":"ContainerStarted","Data":"63e0062827de5268e1cae2b708fd0fe31c33cf3b86ef84cf36ef2e26889b1ba9"}
Sep 12 00:42:07.091694 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.091424    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ftk6p"
Sep 12 00:42:07.091694 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.091430    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ctvv9"
Sep 12 00:42:07.106644 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.106590    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b67b89f7f-qj4t8" podStartSLOduration=1.618166193 podStartE2EDuration="2.106560788s" totalImagesPullingTime="488.394595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:42:07.10519382 +0000 UTC m=+759.309194642" watchObservedRunningTime="2026-09-12 00:42:07.106560788 +0000 UTC m=+759.310561613"
Sep 12 00:42:07.129171 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.129128    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ftk6p"]
Sep 12 00:42:07.135324 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.135289    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-ftk6p"]
Sep 12 00:42:07.137509 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.137482    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qlkcp"]
Sep 12 00:42:07.137729 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.137703    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-qlkcp" podUID="6d2386a3-9b18-48b8-854f-f258feb412b7" containerName="authorino" containerID="cri-o://1a82935fa680041af396d92b7e2238365129aceed24214332d0140fd8934e3f1" gracePeriod=30
Sep 12 00:42:07.155202 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.155171    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ctvv9"]
Sep 12 00:42:07.161451 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.161421    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ctvv9"]
Sep 12 00:42:07.367751 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.367725    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qlkcp"
Sep 12 00:42:07.379239 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.379213    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6bh9\" (UniqueName: \"kubernetes.io/projected/6d2386a3-9b18-48b8-854f-f258feb412b7-kube-api-access-g6bh9\") pod \"6d2386a3-9b18-48b8-854f-f258feb412b7\" (UID: \"6d2386a3-9b18-48b8-854f-f258feb412b7\") "
Sep 12 00:42:07.381540 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.381485    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2386a3-9b18-48b8-854f-f258feb412b7-kube-api-access-g6bh9" (OuterVolumeSpecName: "kube-api-access-g6bh9") pod "6d2386a3-9b18-48b8-854f-f258feb412b7" (UID: "6d2386a3-9b18-48b8-854f-f258feb412b7"). InnerVolumeSpecName "kube-api-access-g6bh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:42:07.480149 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:07.480098    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6bh9\" (UniqueName: \"kubernetes.io/projected/6d2386a3-9b18-48b8-854f-f258feb412b7-kube-api-access-g6bh9\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:42:08.097338 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:08.097297    2567 generic.go:356] "Generic (PLEG): container finished" podID="6d2386a3-9b18-48b8-854f-f258feb412b7" containerID="1a82935fa680041af396d92b7e2238365129aceed24214332d0140fd8934e3f1" exitCode=0
Sep 12 00:42:08.097547 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:08.097350    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qlkcp"
Sep 12 00:42:08.097547 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:08.097391    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qlkcp" event={"ID":"6d2386a3-9b18-48b8-854f-f258feb412b7","Type":"ContainerDied","Data":"1a82935fa680041af396d92b7e2238365129aceed24214332d0140fd8934e3f1"}
Sep 12 00:42:08.097547 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:08.097439    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qlkcp" event={"ID":"6d2386a3-9b18-48b8-854f-f258feb412b7","Type":"ContainerDied","Data":"4993c01c5bee8240634911fa65644ccbd126bea607230e20d26c8ee82a6d2512"}
Sep 12 00:42:08.097547 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:08.097465    2567 scope.go:117] "RemoveContainer" containerID="1a82935fa680041af396d92b7e2238365129aceed24214332d0140fd8934e3f1"
Sep 12 00:42:08.106323 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:08.106300    2567 scope.go:117] "RemoveContainer" containerID="1a82935fa680041af396d92b7e2238365129aceed24214332d0140fd8934e3f1"
Sep 12 00:42:08.106666 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:42:08.106645    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a82935fa680041af396d92b7e2238365129aceed24214332d0140fd8934e3f1\": container with ID starting with 1a82935fa680041af396d92b7e2238365129aceed24214332d0140fd8934e3f1 not found: ID does not exist" containerID="1a82935fa680041af396d92b7e2238365129aceed24214332d0140fd8934e3f1"
Sep 12 00:42:08.106737 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:08.106680    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a82935fa680041af396d92b7e2238365129aceed24214332d0140fd8934e3f1"} err="failed to get container status \"1a82935fa680041af396d92b7e2238365129aceed24214332d0140fd8934e3f1\": rpc error: code = NotFound desc = could not find container \"1a82935fa680041af396d92b7e2238365129aceed24214332d0140fd8934e3f1\": container with ID starting with 1a82935fa680041af396d92b7e2238365129aceed24214332d0140fd8934e3f1 not found: ID does not exist"
Sep 12 00:42:08.117644 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:08.117611    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qlkcp"]
Sep 12 00:42:08.121336 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:08.121306    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qlkcp"]
Sep 12 00:42:08.272818 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:08.272785    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13771957-b24a-4aa8-ac55-eb173bf270bc" path="/var/lib/kubelet/pods/13771957-b24a-4aa8-ac55-eb173bf270bc/volumes"
Sep 12 00:42:08.273039 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:08.273028    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65a5f76e-80da-4bdc-a259-494563c97775" path="/var/lib/kubelet/pods/65a5f76e-80da-4bdc-a259-494563c97775/volumes"
Sep 12 00:42:08.273216 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:08.273206    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d2386a3-9b18-48b8-854f-f258feb412b7" path="/var/lib/kubelet/pods/6d2386a3-9b18-48b8-854f-f258feb412b7/volumes"
Sep 12 00:42:59.321236 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:59.321199    2567 generic.go:356] "Generic (PLEG): container finished" podID="e8f16670-c284-4e2e-b6b4-fe54a025c6a9" containerID="c9d1b5183e14dff84cd2737922d5ec7396ff9ede3c391319b7dd639fc3ccf2c4" exitCode=0
Sep 12 00:42:59.321712 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:59.321273    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm" event={"ID":"e8f16670-c284-4e2e-b6b4-fe54a025c6a9","Type":"ContainerDied","Data":"c9d1b5183e14dff84cd2737922d5ec7396ff9ede3c391319b7dd639fc3ccf2c4"}
Sep 12 00:42:59.321758 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:59.321734    2567 scope.go:117] "RemoveContainer" containerID="c9d1b5183e14dff84cd2737922d5ec7396ff9ede3c391319b7dd639fc3ccf2c4"
Sep 12 00:42:59.322825 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:59.322796    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:42:59.322900 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:59.322869    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:42:59.322934 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:42:59.322913    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:43:00.327406 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:00.327366    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm" event={"ID":"e8f16670-c284-4e2e-b6b4-fe54a025c6a9","Type":"ContainerStarted","Data":"3936e583fb9d256a1babe36a79b3c4e8aa483fdb1f43be555691c718ce769310"}
Sep 12 00:43:01.181801 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:01.181753    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:43:01.182008 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:01.181984    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cvpvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 12 00:43:01.182086 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:01.182064    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm" podUID="e8f16670-c284-4e2e-b6b4-fe54a025c6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 12 00:43:02.182476 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:02.182400    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cvpvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 12 00:43:02.182942 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:02.182481    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm" podUID="e8f16670-c284-4e2e-b6b4-fe54a025c6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 12 00:43:03.182555 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:03.182513    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cvpvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 12 00:43:03.183014 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:03.182602    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm" podUID="e8f16670-c284-4e2e-b6b4-fe54a025c6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 12 00:43:04.182511 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:04.182471    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cvpvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 12 00:43:04.182734 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:04.182532    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm" podUID="e8f16670-c284-4e2e-b6b4-fe54a025c6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 12 00:43:05.182263 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:05.182222    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cvpvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 12 00:43:05.182457 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:05.182280    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm" podUID="e8f16670-c284-4e2e-b6b4-fe54a025c6a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 12 00:43:05.888351 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:05.888304    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:43:06.186666 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:06.186563    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:43:06.360589 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:06.360547    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cvpvm"
Sep 12 00:43:12.594787 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.594748    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"]
Sep 12 00:43:12.595202 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.595050    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2386a3-9b18-48b8-854f-f258feb412b7" containerName="authorino"
Sep 12 00:43:12.595202 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.595061    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2386a3-9b18-48b8-854f-f258feb412b7" containerName="authorino"
Sep 12 00:43:12.595202 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.595128    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d2386a3-9b18-48b8-854f-f258feb412b7" containerName="authorino"
Sep 12 00:43:12.597111 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.597089    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.599672 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.599646    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-x99zn\""
Sep 12 00:43:12.599803 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.599696    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 12 00:43:12.600763 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.600741    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 12 00:43:12.600877 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.600821    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 12 00:43:12.607094 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.607061    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"]
Sep 12 00:43:12.645919 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.645876    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936ed526-5c84-4985-b69f-63bb71e7727b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.645919 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.645920    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936ed526-5c84-4985-b69f-63bb71e7727b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.646165 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.645943    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbb6q\" (UniqueName: \"kubernetes.io/projected/936ed526-5c84-4985-b69f-63bb71e7727b-kube-api-access-rbb6q\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.646165 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.646035    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936ed526-5c84-4985-b69f-63bb71e7727b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.646165 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.646072    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936ed526-5c84-4985-b69f-63bb71e7727b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.646165 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.646092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936ed526-5c84-4985-b69f-63bb71e7727b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.747024 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.746983    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936ed526-5c84-4985-b69f-63bb71e7727b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.747223 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.747039    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936ed526-5c84-4985-b69f-63bb71e7727b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.747223 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.747071    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936ed526-5c84-4985-b69f-63bb71e7727b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.747223 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.747124    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936ed526-5c84-4985-b69f-63bb71e7727b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.747383 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.747234    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936ed526-5c84-4985-b69f-63bb71e7727b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.747383 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.747283    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbb6q\" (UniqueName: \"kubernetes.io/projected/936ed526-5c84-4985-b69f-63bb71e7727b-kube-api-access-rbb6q\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.747502 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.747476    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/936ed526-5c84-4985-b69f-63bb71e7727b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.747590 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.747504    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/936ed526-5c84-4985-b69f-63bb71e7727b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.747673 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.747651    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936ed526-5c84-4985-b69f-63bb71e7727b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.749387 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.749360    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/936ed526-5c84-4985-b69f-63bb71e7727b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.749699 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.749677    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/936ed526-5c84-4985-b69f-63bb71e7727b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.754815 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.754787    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbb6q\" (UniqueName: \"kubernetes.io/projected/936ed526-5c84-4985-b69f-63bb71e7727b-kube-api-access-rbb6q\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn\" (UID: \"936ed526-5c84-4985-b69f-63bb71e7727b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:12.908682 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:12.908547    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:13.183988 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:13.183958    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"]
Sep 12 00:43:13.186096 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:43:13.186067    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936ed526_5c84_4985_b69f_63bb71e7727b.slice/crio-55b1d8a58d73830631e7beca5a7b3318db1f043d757d579d4a1979009e723ba4 WatchSource:0}: Error finding container 55b1d8a58d73830631e7beca5a7b3318db1f043d757d579d4a1979009e723ba4: Status 404 returned error can't find the container with id 55b1d8a58d73830631e7beca5a7b3318db1f043d757d579d4a1979009e723ba4
Sep 12 00:43:13.392973 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:13.392916    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn" event={"ID":"936ed526-5c84-4985-b69f-63bb71e7727b","Type":"ContainerStarted","Data":"55b1d8a58d73830631e7beca5a7b3318db1f043d757d579d4a1979009e723ba4"}
Sep 12 00:43:15.404647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:15.404605    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn" event={"ID":"936ed526-5c84-4985-b69f-63bb71e7727b","Type":"ContainerStarted","Data":"acb4ed5a618b8f664e067e39c5396213be237353d1074d6dde1cb85797724988"}
Sep 12 00:43:15.421418 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:15.421347    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn" podStartSLOduration=2.235011174 podStartE2EDuration="3.421325091s" totalImagesPullingTime="1.186313917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:15.419946003 +0000 UTC m=+827.623946839" watchObservedRunningTime="2026-09-12 00:43:15.421325091 +0000 UTC m=+827.625325915"
Sep 12 00:43:22.909720 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:22.909673    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:22.910252 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:22.909758    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:22.922152 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:22.922121    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:23.449941 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:23.449899    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn"
Sep 12 00:43:47.820909 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.820872    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"]
Sep 12 00:43:47.824435 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.824408    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.826924 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.826892    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 12 00:43:47.833635 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.833599    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"]
Sep 12 00:43:47.861945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.861911    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30615d1d-857d-4b84-9a65-a186f63f565a-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.862162 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.861963    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30615d1d-857d-4b84-9a65-a186f63f565a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.862162 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.862037    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30615d1d-857d-4b84-9a65-a186f63f565a-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.862162 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.862067    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30615d1d-857d-4b84-9a65-a186f63f565a-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.862162 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.862108    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30615d1d-857d-4b84-9a65-a186f63f565a-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.862162 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.862130    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd4dx\" (UniqueName: \"kubernetes.io/projected/30615d1d-857d-4b84-9a65-a186f63f565a-kube-api-access-nd4dx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.963446 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.963396    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30615d1d-857d-4b84-9a65-a186f63f565a-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.963446 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.963444    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd4dx\" (UniqueName: \"kubernetes.io/projected/30615d1d-857d-4b84-9a65-a186f63f565a-kube-api-access-nd4dx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.963749 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.963481    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30615d1d-857d-4b84-9a65-a186f63f565a-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.963749 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.963507    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30615d1d-857d-4b84-9a65-a186f63f565a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.963749 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.963542    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30615d1d-857d-4b84-9a65-a186f63f565a-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.963749 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.963561    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30615d1d-857d-4b84-9a65-a186f63f565a-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.964005 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.963980    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30615d1d-857d-4b84-9a65-a186f63f565a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.964056 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.964021    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30615d1d-857d-4b84-9a65-a186f63f565a-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.964092 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.964025    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30615d1d-857d-4b84-9a65-a186f63f565a-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.965825 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.965798    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30615d1d-857d-4b84-9a65-a186f63f565a-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.966135 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.966117    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30615d1d-857d-4b84-9a65-a186f63f565a-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:47.971642 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:47.971619    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd4dx\" (UniqueName: \"kubernetes.io/projected/30615d1d-857d-4b84-9a65-a186f63f565a-kube-api-access-nd4dx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7\" (UID: \"30615d1d-857d-4b84-9a65-a186f63f565a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:48.135767 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:48.135670    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:48.406479 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:48.406401    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"]
Sep 12 00:43:48.409497 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:43:48.409465    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30615d1d_857d_4b84_9a65_a186f63f565a.slice/crio-d58b539732b01ba1cfb4def858c129a8e00834447079584d12080c43aab30806 WatchSource:0}: Error finding container d58b539732b01ba1cfb4def858c129a8e00834447079584d12080c43aab30806: Status 404 returned error can't find the container with id d58b539732b01ba1cfb4def858c129a8e00834447079584d12080c43aab30806
Sep 12 00:43:48.544353 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:48.544310    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7" event={"ID":"30615d1d-857d-4b84-9a65-a186f63f565a","Type":"ContainerStarted","Data":"d58b539732b01ba1cfb4def858c129a8e00834447079584d12080c43aab30806"}
Sep 12 00:43:49.552677 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:49.552608    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7" event={"ID":"30615d1d-857d-4b84-9a65-a186f63f565a","Type":"ContainerStarted","Data":"7bbcc2788731c05601375543c2ea25ba4dced568842fc67b2deaeccc416a748f"}
Sep 12 00:43:49.571711 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:49.571639    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7" podStartSLOduration=2.393287576 podStartE2EDuration="2.571616955s" totalImagesPullingTime="178.329379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:49.5712881 +0000 UTC m=+861.775288924" watchObservedRunningTime="2026-09-12 00:43:49.571616955 +0000 UTC m=+861.775617779"
Sep 12 00:43:58.136745 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:58.136677    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:58.137193 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:58.136782    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:58.149384 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:58.149345    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:43:58.601666 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:43:58.601631    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7"
Sep 12 00:44:33.506377 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.506339    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"]
Sep 12 00:44:33.509827 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.509796    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.512345 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.512313    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 12 00:44:33.519842 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.519809    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"]
Sep 12 00:44:33.564378 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.564337    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hhxf\" (UniqueName: \"kubernetes.io/projected/22554bd1-0bcb-4a6d-8935-09890d1e0b47-kube-api-access-5hhxf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.564378 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.564381    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22554bd1-0bcb-4a6d-8935-09890d1e0b47-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.564660 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.564409    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22554bd1-0bcb-4a6d-8935-09890d1e0b47-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.564660 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.564502    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22554bd1-0bcb-4a6d-8935-09890d1e0b47-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.564660 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.564525    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22554bd1-0bcb-4a6d-8935-09890d1e0b47-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.564660 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.564605    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22554bd1-0bcb-4a6d-8935-09890d1e0b47-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.665441 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.665397    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22554bd1-0bcb-4a6d-8935-09890d1e0b47-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.665441 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.665442    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22554bd1-0bcb-4a6d-8935-09890d1e0b47-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.665707 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.665481    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22554bd1-0bcb-4a6d-8935-09890d1e0b47-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.665707 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.665512    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hhxf\" (UniqueName: \"kubernetes.io/projected/22554bd1-0bcb-4a6d-8935-09890d1e0b47-kube-api-access-5hhxf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.665707 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.665530    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22554bd1-0bcb-4a6d-8935-09890d1e0b47-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.665707 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.665547    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22554bd1-0bcb-4a6d-8935-09890d1e0b47-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.666023 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.665992    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22554bd1-0bcb-4a6d-8935-09890d1e0b47-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.666153 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.666022    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22554bd1-0bcb-4a6d-8935-09890d1e0b47-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.666153 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.666038    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22554bd1-0bcb-4a6d-8935-09890d1e0b47-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.668274 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.668243    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22554bd1-0bcb-4a6d-8935-09890d1e0b47-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.668647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.668625    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22554bd1-0bcb-4a6d-8935-09890d1e0b47-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.673639 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.673605    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hhxf\" (UniqueName: \"kubernetes.io/projected/22554bd1-0bcb-4a6d-8935-09890d1e0b47-kube-api-access-5hhxf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mjd2z\" (UID: \"22554bd1-0bcb-4a6d-8935-09890d1e0b47\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:33.823258 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:33.823142    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:34.093126 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:34.093028    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"]
Sep 12 00:44:34.096940 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:44:34.096910    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22554bd1_0bcb_4a6d_8935_09890d1e0b47.slice/crio-631d94e8306042aebf711d713e99e45d90dc7809b4c8dc83b19db70b04a0b310 WatchSource:0}: Error finding container 631d94e8306042aebf711d713e99e45d90dc7809b4c8dc83b19db70b04a0b310: Status 404 returned error can't find the container with id 631d94e8306042aebf711d713e99e45d90dc7809b4c8dc83b19db70b04a0b310
Sep 12 00:44:34.746029 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:34.745988    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z" event={"ID":"22554bd1-0bcb-4a6d-8935-09890d1e0b47","Type":"ContainerStarted","Data":"40512cd5bd69dce6f004611a68ca185d14a7085f9c435e40132a42fb6e7cecdc"}
Sep 12 00:44:34.746431 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:34.746059    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z" event={"ID":"22554bd1-0bcb-4a6d-8935-09890d1e0b47","Type":"ContainerStarted","Data":"631d94e8306042aebf711d713e99e45d90dc7809b4c8dc83b19db70b04a0b310"}
Sep 12 00:44:34.762659 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:34.762542    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z" podStartSLOduration=1.574600002 podStartE2EDuration="1.76252105s" totalImagesPullingTime="187.921048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:34.76185918 +0000 UTC m=+906.965860001" watchObservedRunningTime="2026-09-12 00:44:34.76252105 +0000 UTC m=+906.966521875"
Sep 12 00:44:43.823721 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:43.823673    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:43.824208 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:43.823747    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:43.836424 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:43.836393    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:44.808505 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:44.808470    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mjd2z"
Sep 12 00:44:46.713172 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.713132    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"]
Sep 12 00:44:46.716803 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.716773    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.719395 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.719369    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 12 00:44:46.725809 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.724981    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"]
Sep 12 00:44:46.786904 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.786859    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcef50e-8eff-4c04-b398-ac7219f37290-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.786904 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.786911    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl86z\" (UniqueName: \"kubernetes.io/projected/bdcef50e-8eff-4c04-b398-ac7219f37290-kube-api-access-rl86z\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.787168 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.787013    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcef50e-8eff-4c04-b398-ac7219f37290-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.787168 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.787061    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcef50e-8eff-4c04-b398-ac7219f37290-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.787168 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.787084    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcef50e-8eff-4c04-b398-ac7219f37290-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.787168 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.787136    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcef50e-8eff-4c04-b398-ac7219f37290-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.888724 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.888676    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcef50e-8eff-4c04-b398-ac7219f37290-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.888932 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.888738    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcef50e-8eff-4c04-b398-ac7219f37290-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.888932 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.888770    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcef50e-8eff-4c04-b398-ac7219f37290-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.888932 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.888829    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcef50e-8eff-4c04-b398-ac7219f37290-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.889097 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.889070    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcef50e-8eff-4c04-b398-ac7219f37290-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.889172 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.889152    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl86z\" (UniqueName: \"kubernetes.io/projected/bdcef50e-8eff-4c04-b398-ac7219f37290-kube-api-access-rl86z\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.889303 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.889237    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdcef50e-8eff-4c04-b398-ac7219f37290-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.889303 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.889179    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdcef50e-8eff-4c04-b398-ac7219f37290-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.889493 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.889437    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdcef50e-8eff-4c04-b398-ac7219f37290-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.892333 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.892305    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdcef50e-8eff-4c04-b398-ac7219f37290-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.892472 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.892448    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdcef50e-8eff-4c04-b398-ac7219f37290-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:46.897700 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:46.897674    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl86z\" (UniqueName: \"kubernetes.io/projected/bdcef50e-8eff-4c04-b398-ac7219f37290-kube-api-access-rl86z\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns\" (UID: \"bdcef50e-8eff-4c04-b398-ac7219f37290\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:47.028502 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:47.028464    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:47.299483 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:47.299403    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"]
Sep 12 00:44:47.302886 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:44:47.302853    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdcef50e_8eff_4c04_b398_ac7219f37290.slice/crio-58bbf6e422c5d9ae81a3545be01046af955a71e7da5f3a6751745725e48a67da WatchSource:0}: Error finding container 58bbf6e422c5d9ae81a3545be01046af955a71e7da5f3a6751745725e48a67da: Status 404 returned error can't find the container with id 58bbf6e422c5d9ae81a3545be01046af955a71e7da5f3a6751745725e48a67da
Sep 12 00:44:47.816828 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:47.816792    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns" event={"ID":"bdcef50e-8eff-4c04-b398-ac7219f37290","Type":"ContainerStarted","Data":"13e4a63e49a3509811f4be23965fac7b69f8a56daa38fec40be249b86bb88191"}
Sep 12 00:44:47.817250 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:47.816835    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns" event={"ID":"bdcef50e-8eff-4c04-b398-ac7219f37290","Type":"ContainerStarted","Data":"58bbf6e422c5d9ae81a3545be01046af955a71e7da5f3a6751745725e48a67da"}
Sep 12 00:44:47.833313 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:47.833255    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns" podStartSLOduration=1.6412439989999998 podStartE2EDuration="1.833235684s" totalImagesPullingTime="191.991685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:47.832298428 +0000 UTC m=+920.036299242" watchObservedRunningTime="2026-09-12 00:44:47.833235684 +0000 UTC m=+920.037236506"
Sep 12 00:44:57.029523 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:57.029470    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:57.030086 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:57.029541    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:57.042490 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:57.042460    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:44:57.874357 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:44:57.874328    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns"
Sep 12 00:45:46.976337 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:46.976293    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-664f9b94b6-rxqjx"]
Sep 12 00:45:46.979822 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:46.979798    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-664f9b94b6-rxqjx"
Sep 12 00:45:46.987096 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:46.987068    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-664f9b94b6-rxqjx"]
Sep 12 00:45:47.128502 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:47.128459    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cae3b771-b860-4e3d-b2ab-7fb41fc39b4f-tls-cert\") pod \"authorino-664f9b94b6-rxqjx\" (UID: \"cae3b771-b860-4e3d-b2ab-7fb41fc39b4f\") " pod="kuadrant-system/authorino-664f9b94b6-rxqjx"
Sep 12 00:45:47.128502 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:47.128506    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9tq\" (UniqueName: \"kubernetes.io/projected/cae3b771-b860-4e3d-b2ab-7fb41fc39b4f-kube-api-access-6m9tq\") pod \"authorino-664f9b94b6-rxqjx\" (UID: \"cae3b771-b860-4e3d-b2ab-7fb41fc39b4f\") " pod="kuadrant-system/authorino-664f9b94b6-rxqjx"
Sep 12 00:45:47.229750 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:47.229651    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cae3b771-b860-4e3d-b2ab-7fb41fc39b4f-tls-cert\") pod \"authorino-664f9b94b6-rxqjx\" (UID: \"cae3b771-b860-4e3d-b2ab-7fb41fc39b4f\") " pod="kuadrant-system/authorino-664f9b94b6-rxqjx"
Sep 12 00:45:47.229750 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:47.229709    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9tq\" (UniqueName: \"kubernetes.io/projected/cae3b771-b860-4e3d-b2ab-7fb41fc39b4f-kube-api-access-6m9tq\") pod \"authorino-664f9b94b6-rxqjx\" (UID: \"cae3b771-b860-4e3d-b2ab-7fb41fc39b4f\") " pod="kuadrant-system/authorino-664f9b94b6-rxqjx"
Sep 12 00:45:47.232149 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:47.232116    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cae3b771-b860-4e3d-b2ab-7fb41fc39b4f-tls-cert\") pod \"authorino-664f9b94b6-rxqjx\" (UID: \"cae3b771-b860-4e3d-b2ab-7fb41fc39b4f\") " pod="kuadrant-system/authorino-664f9b94b6-rxqjx"
Sep 12 00:45:47.238273 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:47.238248    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9tq\" (UniqueName: \"kubernetes.io/projected/cae3b771-b860-4e3d-b2ab-7fb41fc39b4f-kube-api-access-6m9tq\") pod \"authorino-664f9b94b6-rxqjx\" (UID: \"cae3b771-b860-4e3d-b2ab-7fb41fc39b4f\") " pod="kuadrant-system/authorino-664f9b94b6-rxqjx"
Sep 12 00:45:47.289394 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:47.289351    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-664f9b94b6-rxqjx"
Sep 12 00:45:47.560706 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:47.560678    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-664f9b94b6-rxqjx"]
Sep 12 00:45:47.563306 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:45:47.563277    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcae3b771_b860_4e3d_b2ab_7fb41fc39b4f.slice/crio-b099c98896f5dab0d84b56d98aff2a0c372c4642d9379aedbea74a962903b4b6 WatchSource:0}: Error finding container b099c98896f5dab0d84b56d98aff2a0c372c4642d9379aedbea74a962903b4b6: Status 404 returned error can't find the container with id b099c98896f5dab0d84b56d98aff2a0c372c4642d9379aedbea74a962903b4b6
Sep 12 00:45:48.093320 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:48.093283    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-664f9b94b6-rxqjx" event={"ID":"cae3b771-b860-4e3d-b2ab-7fb41fc39b4f","Type":"ContainerStarted","Data":"b099c98896f5dab0d84b56d98aff2a0c372c4642d9379aedbea74a962903b4b6"}
Sep 12 00:45:49.101667 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:49.101609    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-664f9b94b6-rxqjx" event={"ID":"cae3b771-b860-4e3d-b2ab-7fb41fc39b4f","Type":"ContainerStarted","Data":"5e2b795bfb50e274be749bcfa1e932559257c34777c60e97c636a1e37536dd6a"}
Sep 12 00:45:49.124133 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:49.124062    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-664f9b94b6-rxqjx" podStartSLOduration=2.292944299 podStartE2EDuration="3.124037265s" totalImagesPullingTime="831.092966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:49.121453944 +0000 UTC m=+981.325454767" watchObservedRunningTime="2026-09-12 00:45:49.124037265 +0000 UTC m=+981.328038089"
Sep 12 00:45:49.149513 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:49.149456    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b67b89f7f-qj4t8"]
Sep 12 00:45:49.149773 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:49.149739    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7b67b89f7f-qj4t8" podUID="856b7345-50af-4e17-b908-0220753f59fa" containerName="authorino" containerID="cri-o://63e0062827de5268e1cae2b708fd0fe31c33cf3b86ef84cf36ef2e26889b1ba9" gracePeriod=30
Sep 12 00:45:49.415832 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:49.415794    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b67b89f7f-qj4t8"
Sep 12 00:45:49.549483 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:49.549444    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/856b7345-50af-4e17-b908-0220753f59fa-tls-cert\") pod \"856b7345-50af-4e17-b908-0220753f59fa\" (UID: \"856b7345-50af-4e17-b908-0220753f59fa\") "
Sep 12 00:45:49.549656 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:49.549500    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-857p5\" (UniqueName: \"kubernetes.io/projected/856b7345-50af-4e17-b908-0220753f59fa-kube-api-access-857p5\") pod \"856b7345-50af-4e17-b908-0220753f59fa\" (UID: \"856b7345-50af-4e17-b908-0220753f59fa\") "
Sep 12 00:45:49.551597 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:49.551551    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856b7345-50af-4e17-b908-0220753f59fa-kube-api-access-857p5" (OuterVolumeSpecName: "kube-api-access-857p5") pod "856b7345-50af-4e17-b908-0220753f59fa" (UID: "856b7345-50af-4e17-b908-0220753f59fa"). InnerVolumeSpecName "kube-api-access-857p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:45:49.561034 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:49.560999    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856b7345-50af-4e17-b908-0220753f59fa-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "856b7345-50af-4e17-b908-0220753f59fa" (UID: "856b7345-50af-4e17-b908-0220753f59fa"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:45:49.650522 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:49.650439    2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/856b7345-50af-4e17-b908-0220753f59fa-tls-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:45:49.650522 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:49.650472    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-857p5\" (UniqueName: \"kubernetes.io/projected/856b7345-50af-4e17-b908-0220753f59fa-kube-api-access-857p5\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:45:50.109109 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:50.109066    2567 generic.go:356] "Generic (PLEG): container finished" podID="856b7345-50af-4e17-b908-0220753f59fa" containerID="63e0062827de5268e1cae2b708fd0fe31c33cf3b86ef84cf36ef2e26889b1ba9" exitCode=0
Sep 12 00:45:50.109618 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:50.109132    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b67b89f7f-qj4t8"
Sep 12 00:45:50.109618 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:50.109153    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b67b89f7f-qj4t8" event={"ID":"856b7345-50af-4e17-b908-0220753f59fa","Type":"ContainerDied","Data":"63e0062827de5268e1cae2b708fd0fe31c33cf3b86ef84cf36ef2e26889b1ba9"}
Sep 12 00:45:50.109618 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:50.109199    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b67b89f7f-qj4t8" event={"ID":"856b7345-50af-4e17-b908-0220753f59fa","Type":"ContainerDied","Data":"c4b545646eaf115c66d68a1f85fa87110e7eecca16116957386d70ee00fed1a3"}
Sep 12 00:45:50.109618 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:50.109219    2567 scope.go:117] "RemoveContainer" containerID="63e0062827de5268e1cae2b708fd0fe31c33cf3b86ef84cf36ef2e26889b1ba9"
Sep 12 00:45:50.120601 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:50.120546    2567 scope.go:117] "RemoveContainer" containerID="63e0062827de5268e1cae2b708fd0fe31c33cf3b86ef84cf36ef2e26889b1ba9"
Sep 12 00:45:50.121256 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:45:50.121225    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e0062827de5268e1cae2b708fd0fe31c33cf3b86ef84cf36ef2e26889b1ba9\": container with ID starting with 63e0062827de5268e1cae2b708fd0fe31c33cf3b86ef84cf36ef2e26889b1ba9 not found: ID does not exist" containerID="63e0062827de5268e1cae2b708fd0fe31c33cf3b86ef84cf36ef2e26889b1ba9"
Sep 12 00:45:50.121350 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:50.121264    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e0062827de5268e1cae2b708fd0fe31c33cf3b86ef84cf36ef2e26889b1ba9"} err="failed to get container status \"63e0062827de5268e1cae2b708fd0fe31c33cf3b86ef84cf36ef2e26889b1ba9\": rpc error: code = NotFound desc = could not find container \"63e0062827de5268e1cae2b708fd0fe31c33cf3b86ef84cf36ef2e26889b1ba9\": container with ID starting with 63e0062827de5268e1cae2b708fd0fe31c33cf3b86ef84cf36ef2e26889b1ba9 not found: ID does not exist"
Sep 12 00:45:50.139176 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:50.139133    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b67b89f7f-qj4t8"]
Sep 12 00:45:50.147305 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:50.147273    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7b67b89f7f-qj4t8"]
Sep 12 00:45:50.274828 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:45:50.274744    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="856b7345-50af-4e17-b908-0220753f59fa" path="/var/lib/kubelet/pods/856b7345-50af-4e17-b908-0220753f59fa/volumes"
Sep 12 00:46:07.468479 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.468439    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"]
Sep 12 00:46:07.468908 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.468833    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856b7345-50af-4e17-b908-0220753f59fa" containerName="authorino"
Sep 12 00:46:07.468908 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.468848    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="856b7345-50af-4e17-b908-0220753f59fa" containerName="authorino"
Sep 12 00:46:07.468908 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.468903    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="856b7345-50af-4e17-b908-0220753f59fa" containerName="authorino"
Sep 12 00:46:07.471816 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.471791    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.474603 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.474560    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-876808-openshift-default-dockercfg-l8sh6\""
Sep 12 00:46:07.485190 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.485151    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"]
Sep 12 00:46:07.507759 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.507724    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-workload-certs\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.507962 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.507767    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-credential-socket\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.507962 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.507792    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cm84\" (UniqueName: \"kubernetes.io/projected/3ed5365d-b02a-4167-b878-2e78a963ad37-kube-api-access-8cm84\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.507962 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.507846    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-envoy\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.507962 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.507900    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-podinfo\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.507962 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.507938    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-token\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.508219 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.508020    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-workload-socket\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.508219 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.508068    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-data\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.508219 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.508102    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed5365d-b02a-4167-b878-2e78a963ad37-istiod-ca-cert\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.608971 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.608931    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-workload-certs\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.609171 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.608985    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-credential-socket\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.609171 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.609019    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cm84\" (UniqueName: \"kubernetes.io/projected/3ed5365d-b02a-4167-b878-2e78a963ad37-kube-api-access-8cm84\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.609171 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.609044    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-envoy\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.609171 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.609077    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-podinfo\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.609171 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.609113    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-token\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.609171 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.609173    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-workload-socket\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.609511 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.609205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-data\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.609511 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.609266    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed5365d-b02a-4167-b878-2e78a963ad37-istiod-ca-cert\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.609511 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.609398    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-workload-certs\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.609816 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.609770    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-workload-socket\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.609878 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.609793    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-credential-socket\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.609935 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.609897    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-data\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.610331 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.610309    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed5365d-b02a-4167-b878-2e78a963ad37-istiod-ca-cert\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.612051 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.612025    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-podinfo\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.612155 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.612078    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-envoy\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.624257 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.624190    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-token\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.624677 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.624646    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cm84\" (UniqueName: \"kubernetes.io/projected/3ed5365d-b02a-4167-b878-2e78a963ad37-kube-api-access-8cm84\") pod \"e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") " pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:07.782553 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:07.782505    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:08.061057 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:08.060934    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"]
Sep 12 00:46:08.064080 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:46:08.064052    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed5365d_b02a_4167_b878_2e78a963ad37.slice/crio-918b4d00648108986f7ff9b8669bcad5515251839d350aa0fcb9770e5bfc684e WatchSource:0}: Error finding container 918b4d00648108986f7ff9b8669bcad5515251839d350aa0fcb9770e5bfc684e: Status 404 returned error can't find the container with id 918b4d00648108986f7ff9b8669bcad5515251839d350aa0fcb9770e5bfc684e
Sep 12 00:46:08.066367 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:08.066326    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:08.066459 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:08.066429    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:08.066688 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:08.066469    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:08.194049 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:08.194004    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" event={"ID":"3ed5365d-b02a-4167-b878-2e78a963ad37","Type":"ContainerStarted","Data":"5d3bcb637c77ff77e3901528a54f47d624c368f63e231a11947b2daeacebc672"}
Sep 12 00:46:08.194252 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:08.194071    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" event={"ID":"3ed5365d-b02a-4167-b878-2e78a963ad37","Type":"ContainerStarted","Data":"918b4d00648108986f7ff9b8669bcad5515251839d350aa0fcb9770e5bfc684e"}
Sep 12 00:46:08.216241 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:08.216188    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" podStartSLOduration=1.216171462 podStartE2EDuration="1.216171462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:46:08.214377769 +0000 UTC m=+1000.418378591" watchObservedRunningTime="2026-09-12 00:46:08.216171462 +0000 UTC m=+1000.420172283"
Sep 12 00:46:08.782946 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:08.782912    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:09.787991 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:09.787957    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:10.203412 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:10.203332    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:10.204273 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:10.204253    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:13.599241 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:13.599200    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7688-e8729c"]
Sep 12 00:46:13.602617 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:13.602593    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-e8729c"
Sep 12 00:46:13.605252 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:13.605222    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tqwft\""
Sep 12 00:46:13.609079 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:13.609053    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7688-e8729c"]
Sep 12 00:46:13.667682 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:13.667644    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v7wv\" (UniqueName: \"kubernetes.io/projected/8a29bcaf-2482-44ec-b0d6-0358d388625a-kube-api-access-4v7wv\") pod \"test-curl-7688-e8729c\" (UID: \"8a29bcaf-2482-44ec-b0d6-0358d388625a\") " pod="opendatahub/test-curl-7688-e8729c"
Sep 12 00:46:13.768353 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:13.768298    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v7wv\" (UniqueName: \"kubernetes.io/projected/8a29bcaf-2482-44ec-b0d6-0358d388625a-kube-api-access-4v7wv\") pod \"test-curl-7688-e8729c\" (UID: \"8a29bcaf-2482-44ec-b0d6-0358d388625a\") " pod="opendatahub/test-curl-7688-e8729c"
Sep 12 00:46:13.776699 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:13.776662    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v7wv\" (UniqueName: \"kubernetes.io/projected/8a29bcaf-2482-44ec-b0d6-0358d388625a-kube-api-access-4v7wv\") pod \"test-curl-7688-e8729c\" (UID: \"8a29bcaf-2482-44ec-b0d6-0358d388625a\") " pod="opendatahub/test-curl-7688-e8729c"
Sep 12 00:46:13.913415 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:13.913326    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-e8729c"
Sep 12 00:46:14.188654 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:14.188549    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7688-e8729c"]
Sep 12 00:46:14.191666 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:46:14.191631    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a29bcaf_2482_44ec_b0d6_0358d388625a.slice/crio-dea89f59f78381aafee9d75a826c1c65a7d3fa901cb8ed1e80c871c548baced7 WatchSource:0}: Error finding container dea89f59f78381aafee9d75a826c1c65a7d3fa901cb8ed1e80c871c548baced7: Status 404 returned error can't find the container with id dea89f59f78381aafee9d75a826c1c65a7d3fa901cb8ed1e80c871c548baced7
Sep 12 00:46:14.223138 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:14.223081    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7688-e8729c" event={"ID":"8a29bcaf-2482-44ec-b0d6-0358d388625a","Type":"ContainerStarted","Data":"dea89f59f78381aafee9d75a826c1c65a7d3fa901cb8ed1e80c871c548baced7"}
Sep 12 00:46:15.228792 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:15.228755    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7688-e8729c" event={"ID":"8a29bcaf-2482-44ec-b0d6-0358d388625a","Type":"ContainerStarted","Data":"84dfc002f29539048b2d8eadab0a6709ac56a3f4c0c20b1902fe19afc90bdd37"}
Sep 12 00:46:15.242921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:15.242866    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7688-e8729c" podStartSLOduration=1.52351119 podStartE2EDuration="2.24284938s" totalImagesPullingTime="719.33819ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:46:15.241771199 +0000 UTC m=+1007.445772022" watchObservedRunningTime="2026-09-12 00:46:15.24284938 +0000 UTC m=+1007.446850202"
Sep 12 00:46:15.827593 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:15.827535    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7688-e8729c"]
Sep 12 00:46:15.831948 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:15.831920    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7688-e8729c"]
Sep 12 00:46:16.234650 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:46:16.234617    2567 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7688-e8729c" container="test-curl-7688-e8729c"
Sep 12 00:46:16.628223 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.628135    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"]
Sep 12 00:46:16.628712 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.628692    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a29bcaf-2482-44ec-b0d6-0358d388625a" containerName="test-curl-7688-e8729c"
Sep 12 00:46:16.628768 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.628717    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a29bcaf-2482-44ec-b0d6-0358d388625a" containerName="test-curl-7688-e8729c"
Sep 12 00:46:16.628815 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.628803    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a29bcaf-2482-44ec-b0d6-0358d388625a" containerName="test-curl-7688-e8729c"
Sep 12 00:46:16.632420 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.632383    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.634959 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.634931    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-4b6ed7-openshift-default-dockercfg-mv4nl\""
Sep 12 00:46:16.643329 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.643294    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"]
Sep 12 00:46:16.700439 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.700397    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-istio-envoy\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.700685 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.700448    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-workload-socket\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.700685 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.700507    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872c43fb-40e8-4695-8de4-1d85948b786c-istiod-ca-cert\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.700685 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.700595    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-workload-certs\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.700685 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.700632    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-istio-data\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.700903 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.700701    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872c43fb-40e8-4695-8de4-1d85948b786c-istio-podinfo\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.700903 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.700718    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57m6j\" (UniqueName: \"kubernetes.io/projected/872c43fb-40e8-4695-8de4-1d85948b786c-kube-api-access-57m6j\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.700903 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.700796    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-credential-socket\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.700903 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.700830    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872c43fb-40e8-4695-8de4-1d85948b786c-istio-token\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.802019 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.801975    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-istio-data\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.802218 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.802060    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872c43fb-40e8-4695-8de4-1d85948b786c-istio-podinfo\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.802218 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.802090    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57m6j\" (UniqueName: \"kubernetes.io/projected/872c43fb-40e8-4695-8de4-1d85948b786c-kube-api-access-57m6j\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.802218 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.802133    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-credential-socket\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.802218 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.802167    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872c43fb-40e8-4695-8de4-1d85948b786c-istio-token\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.802423 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.802223    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-istio-envoy\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.802423 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.802259    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-workload-socket\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.802423 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.802297    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872c43fb-40e8-4695-8de4-1d85948b786c-istiod-ca-cert\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.802423 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.802339    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-workload-certs\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.802640 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.802491    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-istio-data\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.802746 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.802722    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-workload-certs\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.802988 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.802949    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-workload-socket\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.803467 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.803424    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872c43fb-40e8-4695-8de4-1d85948b786c-istiod-ca-cert\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.803607 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.803552    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-credential-socket\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.805482 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.805455    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872c43fb-40e8-4695-8de4-1d85948b786c-istio-podinfo\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.805765 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.805744    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-istio-envoy\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.812207 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.812172    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872c43fb-40e8-4695-8de4-1d85948b786c-istio-token\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.812744 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.812707    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57m6j\" (UniqueName: \"kubernetes.io/projected/872c43fb-40e8-4695-8de4-1d85948b786c-kube-api-access-57m6j\") pod \"e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") " pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:16.946886 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:16.946764    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:17.238907 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:17.238834    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7688-e8729c" podUID="8a29bcaf-2482-44ec-b0d6-0358d388625a" containerName="test-curl-7688-e8729c" containerID="cri-o://84dfc002f29539048b2d8eadab0a6709ac56a3f4c0c20b1902fe19afc90bdd37" gracePeriod=2
Sep 12 00:46:17.267407 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:17.267365    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"]
Sep 12 00:46:17.271874 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:46:17.271833    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872c43fb_40e8_4695_8de4_1d85948b786c.slice/crio-98d6a400b33d6a97fd1cb11a0e4a8a89430fdb9633f959a689f0aa110174e6b2 WatchSource:0}: Error finding container 98d6a400b33d6a97fd1cb11a0e4a8a89430fdb9633f959a689f0aa110174e6b2: Status 404 returned error can't find the container with id 98d6a400b33d6a97fd1cb11a0e4a8a89430fdb9633f959a689f0aa110174e6b2
Sep 12 00:46:17.274479 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:17.274439    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:17.274631 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:17.274517    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:17.274631 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:17.274551    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:17.968411 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:17.968364    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"]
Sep 12 00:46:17.972596 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:17.972460    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:17.977594 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:17.976964    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-220bac51-openshift-default-dockercfg-t6cdb\""
Sep 12 00:46:18.012509 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.012462    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"]
Sep 12 00:46:18.015825 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.015773    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-workload-socket\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.015825 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.015825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-credential-socket\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.016082 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.015869    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86tn\" (UniqueName: \"kubernetes.io/projected/8e53fede-e45a-4fe5-ba60-9ef016babd27-kube-api-access-d86tn\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.016082 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.015907    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e53fede-e45a-4fe5-ba60-9ef016babd27-istiod-ca-cert\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.016082 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.015961    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-podinfo\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.016082 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.016001    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-token\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.016082 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.016054    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-envoy\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.016373 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.016093    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-data\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.016373 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.016144    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-workload-certs\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.117048 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.117005    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-envoy\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.117339 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.117056    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-data\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.117339 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.117101    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-workload-certs\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.117339 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.117141    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-workload-socket\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.117339 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.117190    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-credential-socket\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.117339 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.117255    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86tn\" (UniqueName: \"kubernetes.io/projected/8e53fede-e45a-4fe5-ba60-9ef016babd27-kube-api-access-d86tn\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.117339 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.117296    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e53fede-e45a-4fe5-ba60-9ef016babd27-istiod-ca-cert\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.117685 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.117349    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-podinfo\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.117685 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.117393    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-token\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.117685 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.117528    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-workload-socket\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.117685 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.117560    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-workload-certs\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.117685 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.117656    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-credential-socket\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.118171 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.118142    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e53fede-e45a-4fe5-ba60-9ef016babd27-istiod-ca-cert\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.118289 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.118214    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-data\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.119603 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.119557    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-envoy\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.119890 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.119870    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-podinfo\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.126239 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.126202    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86tn\" (UniqueName: \"kubernetes.io/projected/8e53fede-e45a-4fe5-ba60-9ef016babd27-kube-api-access-d86tn\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.126432 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.126411    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-token\") pod \"e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") " pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.253474 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.253419    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" event={"ID":"872c43fb-40e8-4695-8de4-1d85948b786c","Type":"ContainerStarted","Data":"e0aed1d07042242026bf4fcecc3a11e4c8c96bc01928d1038203c2bc56cdd055"}
Sep 12 00:46:18.254016 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.253503    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" event={"ID":"872c43fb-40e8-4695-8de4-1d85948b786c","Type":"ContainerStarted","Data":"98d6a400b33d6a97fd1cb11a0e4a8a89430fdb9633f959a689f0aa110174e6b2"}
Sep 12 00:46:18.281185 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.281116    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" podStartSLOduration=2.2810904020000002 podStartE2EDuration="2.281090402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:46:18.279669893 +0000 UTC m=+1010.483670716" watchObservedRunningTime="2026-09-12 00:46:18.281090402 +0000 UTC m=+1010.485091225"
Sep 12 00:46:18.295958 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.295921    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:18.600672 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.600589    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"]
Sep 12 00:46:18.604187 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:46:18.604159    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e53fede_e45a_4fe5_ba60_9ef016babd27.slice/crio-d6b8667e33b1a815faeaf164e3f5205abcd59fffb281d9a1bbb82fa0c43e0733 WatchSource:0}: Error finding container d6b8667e33b1a815faeaf164e3f5205abcd59fffb281d9a1bbb82fa0c43e0733: Status 404 returned error can't find the container with id d6b8667e33b1a815faeaf164e3f5205abcd59fffb281d9a1bbb82fa0c43e0733
Sep 12 00:46:18.606422 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.606388    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:18.606491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.606459    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:18.606539 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.606489    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:18.948072 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.947961    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:18.958638 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:18.958599    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:19.270709 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:19.270404    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72" event={"ID":"8e53fede-e45a-4fe5-ba60-9ef016babd27","Type":"ContainerStarted","Data":"64408dd8f5b50953010d14b21bfc2942f447f20e3e4885fbc04c829fc475955c"}
Sep 12 00:46:19.270709 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:19.270471    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:19.270709 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:19.270499    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72" event={"ID":"8e53fede-e45a-4fe5-ba60-9ef016babd27","Type":"ContainerStarted","Data":"d6b8667e33b1a815faeaf164e3f5205abcd59fffb281d9a1bbb82fa0c43e0733"}
Sep 12 00:46:19.274227 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:19.274183    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:19.292550 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:19.292490    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72" podStartSLOduration=2.29247276 podStartE2EDuration="2.29247276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:46:19.292266852 +0000 UTC m=+1011.496267674" watchObservedRunningTime="2026-09-12 00:46:19.29247276 +0000 UTC m=+1011.496473581"
Sep 12 00:46:19.296107 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:19.296080    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:19.301586 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:19.301551    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:19.452158 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:19.451798    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-e8729c"
Sep 12 00:46:19.457907 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:46:19.457836    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7688-e8729c\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="8a29bcaf-2482-44ec-b0d6-0358d388625a" pod="opendatahub/test-curl-7688-e8729c"
Sep 12 00:46:19.533898 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:19.533794    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v7wv\" (UniqueName: \"kubernetes.io/projected/8a29bcaf-2482-44ec-b0d6-0358d388625a-kube-api-access-4v7wv\") pod \"8a29bcaf-2482-44ec-b0d6-0358d388625a\" (UID: \"8a29bcaf-2482-44ec-b0d6-0358d388625a\") "
Sep 12 00:46:19.536081 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:19.536051    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a29bcaf-2482-44ec-b0d6-0358d388625a-kube-api-access-4v7wv" (OuterVolumeSpecName: "kube-api-access-4v7wv") pod "8a29bcaf-2482-44ec-b0d6-0358d388625a" (UID: "8a29bcaf-2482-44ec-b0d6-0358d388625a"). InnerVolumeSpecName "kube-api-access-4v7wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:46:19.634922 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:19.634886    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v7wv\" (UniqueName: \"kubernetes.io/projected/8a29bcaf-2482-44ec-b0d6-0358d388625a-kube-api-access-4v7wv\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:46:20.274507 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:20.274474    2567 generic.go:356] "Generic (PLEG): container finished" podID="8a29bcaf-2482-44ec-b0d6-0358d388625a" containerID="84dfc002f29539048b2d8eadab0a6709ac56a3f4c0c20b1902fe19afc90bdd37" exitCode=137
Sep 12 00:46:20.274967 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:20.274541    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-e8729c"
Sep 12 00:46:20.274967 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:20.274481    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a29bcaf-2482-44ec-b0d6-0358d388625a" path="/var/lib/kubelet/pods/8a29bcaf-2482-44ec-b0d6-0358d388625a/volumes"
Sep 12 00:46:20.275161 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:20.275141    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:20.275224 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:20.275190    2567 scope.go:117] "RemoveContainer" containerID="84dfc002f29539048b2d8eadab0a6709ac56a3f4c0c20b1902fe19afc90bdd37"
Sep 12 00:46:20.275881 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:20.275861    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:20.277777 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:46:20.277754    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7688-e8729c\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="8a29bcaf-2482-44ec-b0d6-0358d388625a" pod="opendatahub/test-curl-7688-e8729c"
Sep 12 00:46:20.284134 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:20.284116    2567 scope.go:117] "RemoveContainer" containerID="84dfc002f29539048b2d8eadab0a6709ac56a3f4c0c20b1902fe19afc90bdd37"
Sep 12 00:46:20.284450 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:46:20.284430    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84dfc002f29539048b2d8eadab0a6709ac56a3f4c0c20b1902fe19afc90bdd37\": container with ID starting with 84dfc002f29539048b2d8eadab0a6709ac56a3f4c0c20b1902fe19afc90bdd37 not found: ID does not exist" containerID="84dfc002f29539048b2d8eadab0a6709ac56a3f4c0c20b1902fe19afc90bdd37"
Sep 12 00:46:20.284511 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:20.284457    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84dfc002f29539048b2d8eadab0a6709ac56a3f4c0c20b1902fe19afc90bdd37"} err="failed to get container status \"84dfc002f29539048b2d8eadab0a6709ac56a3f4c0c20b1902fe19afc90bdd37\": rpc error: code = NotFound desc = could not find container \"84dfc002f29539048b2d8eadab0a6709ac56a3f4c0c20b1902fe19afc90bdd37\": container with ID starting with 84dfc002f29539048b2d8eadab0a6709ac56a3f4c0c20b1902fe19afc90bdd37 not found: ID does not exist"
Sep 12 00:46:20.301295 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:46:20.301261    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7688-e8729c\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="8a29bcaf-2482-44ec-b0d6-0358d388625a" pod="opendatahub/test-curl-7688-e8729c"
Sep 12 00:46:20.303258 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:46:20.303226    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7688-e8729c\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="8a29bcaf-2482-44ec-b0d6-0358d388625a" pod="opendatahub/test-curl-7688-e8729c"
Sep 12 00:46:25.629837 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:25.629798    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"]
Sep 12 00:46:25.633315 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:25.633287    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"
Sep 12 00:46:25.636567 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:25.636530    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-4b6ed7-dockercfg-mx46k\""
Sep 12 00:46:25.636697 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:25.636566    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-4b6ed7\""
Sep 12 00:46:25.643368 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:25.643285    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"]
Sep 12 00:46:25.710401 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:25.710358    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7nt\" (UniqueName: \"kubernetes.io/projected/2d3b89ec-2c36-4409-840a-ebd1e85ebee8-kube-api-access-5h7nt\") pod \"payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds\" (UID: \"2d3b89ec-2c36-4409-840a-ebd1e85ebee8\") " pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"
Sep 12 00:46:25.710617 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:25.710411    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b89ec-2c36-4409-840a-ebd1e85ebee8-plugins-config-volume\") pod \"payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds\" (UID: \"2d3b89ec-2c36-4409-840a-ebd1e85ebee8\") " pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"
Sep 12 00:46:25.811787 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:25.811746    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b89ec-2c36-4409-840a-ebd1e85ebee8-plugins-config-volume\") pod \"payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds\" (UID: \"2d3b89ec-2c36-4409-840a-ebd1e85ebee8\") " pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"
Sep 12 00:46:25.811991 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:25.811871    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7nt\" (UniqueName: \"kubernetes.io/projected/2d3b89ec-2c36-4409-840a-ebd1e85ebee8-kube-api-access-5h7nt\") pod \"payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds\" (UID: \"2d3b89ec-2c36-4409-840a-ebd1e85ebee8\") " pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"
Sep 12 00:46:25.812547 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:25.812519    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b89ec-2c36-4409-840a-ebd1e85ebee8-plugins-config-volume\") pod \"payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds\" (UID: \"2d3b89ec-2c36-4409-840a-ebd1e85ebee8\") " pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"
Sep 12 00:46:25.823782 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:25.823753    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7nt\" (UniqueName: \"kubernetes.io/projected/2d3b89ec-2c36-4409-840a-ebd1e85ebee8-kube-api-access-5h7nt\") pod \"payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds\" (UID: \"2d3b89ec-2c36-4409-840a-ebd1e85ebee8\") " pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"
Sep 12 00:46:25.950902 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:25.950812    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"
Sep 12 00:46:26.233053 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:26.233000    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"]
Sep 12 00:46:26.236003 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:46:26.235971    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3b89ec_2c36_4409_840a_ebd1e85ebee8.slice/crio-f821f0d92096191a66eb5bfbfbae0d4417e67572c044aa84881c8cc0d9c720a5 WatchSource:0}: Error finding container f821f0d92096191a66eb5bfbfbae0d4417e67572c044aa84881c8cc0d9c720a5: Status 404 returned error can't find the container with id f821f0d92096191a66eb5bfbfbae0d4417e67572c044aa84881c8cc0d9c720a5
Sep 12 00:46:26.245711 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:26.245683    2567 patch_prober.go:28] interesting pod/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 12 00:46:26.245812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:26.245735    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 12 00:46:26.322505 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:26.322449    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds" event={"ID":"2d3b89ec-2c36-4409-840a-ebd1e85ebee8","Type":"ContainerStarted","Data":"f821f0d92096191a66eb5bfbfbae0d4417e67572c044aa84881c8cc0d9c720a5"}
Sep 12 00:46:27.329856 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:27.329809    2567 generic.go:356] "Generic (PLEG): container finished" podID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerID="5d3bcb637c77ff77e3901528a54f47d624c368f63e231a11947b2daeacebc672" exitCode=0
Sep 12 00:46:27.330392 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:27.329890    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" event={"ID":"3ed5365d-b02a-4167-b878-2e78a963ad37","Type":"ContainerDied","Data":"5d3bcb637c77ff77e3901528a54f47d624c368f63e231a11947b2daeacebc672"}
Sep 12 00:46:27.330392 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:27.330357    2567 scope.go:117] "RemoveContainer" containerID="5d3bcb637c77ff77e3901528a54f47d624c368f63e231a11947b2daeacebc672"
Sep 12 00:46:27.331675 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:27.331640    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:27.331781 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:27.331718    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:27.331781 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:27.331762    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:28.361598 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:28.358998    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" event={"ID":"3ed5365d-b02a-4167-b878-2e78a963ad37","Type":"ContainerStarted","Data":"6560a0a0d0c6c0d1a90b823fe5daee035765c2d94feda30340ca16dbdee0da03"}
Sep 12 00:46:28.782858 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:28.782824    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:28.783226 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:28.783202    2567 patch_prober.go:28] interesting pod/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 12 00:46:28.783289 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:28.783253    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 12 00:46:29.380606 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:29.380531    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds" event={"ID":"2d3b89ec-2c36-4409-840a-ebd1e85ebee8","Type":"ContainerStarted","Data":"b8f69719c251bc7df85d36d9b6a32f0e77b1c07af9611c7c6e8c0f059df227b6"}
Sep 12 00:46:29.399137 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:29.399080    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds" podStartSLOduration=1.9407086709999999 podStartE2EDuration="4.399061548s" totalImagesPullingTime="2.458352877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:46:29.396387815 +0000 UTC m=+1021.600388640" watchObservedRunningTime="2026-09-12 00:46:29.399061548 +0000 UTC m=+1021.603062364"
Sep 12 00:46:29.783737 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:29.783688    2567 patch_prober.go:28] interesting pod/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 12 00:46:29.783955 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:29.783757    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 12 00:46:30.385354 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:30.385318    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"
Sep 12 00:46:30.782935 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:30.782894    2567 patch_prober.go:28] interesting pod/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 12 00:46:30.783117 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:30.783046    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 12 00:46:31.783449 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:31.783406    2567 patch_prober.go:28] interesting pod/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 12 00:46:31.783964 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:31.783477    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 12 00:46:32.783625 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:32.783544    2567 patch_prober.go:28] interesting pod/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 12 00:46:32.784164 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:32.783637    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 12 00:46:33.979945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:33.979897    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"]
Sep 12 00:46:33.989605 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:33.988740    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:33.996791 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:33.995355    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bb6e74-openshift-default-dockercfg-9wfdt\""
Sep 12 00:46:34.000957 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.000921    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"]
Sep 12 00:46:34.017123 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.017094    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-podinfo\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.017270 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.017137    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-credential-socket\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.017270 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.017183    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-token\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.017270 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.017205    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-workload-certs\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.017270 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.017233    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-workload-socket\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.017270 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.017250    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-data\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.017270 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.017267    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-envoy\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.017469 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.017288    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istiod-ca-cert\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.017469 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.017307    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmc5q\" (UniqueName: \"kubernetes.io/projected/d5c5e730-811c-4c4c-9f65-c907f374fd2e-kube-api-access-wmc5q\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.061076 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.061032    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:34.062870 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.062843    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:34.072916 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.072881    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 00:46:34.117980 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.117945    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmc5q\" (UniqueName: \"kubernetes.io/projected/d5c5e730-811c-4c4c-9f65-c907f374fd2e-kube-api-access-wmc5q\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.118183 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.118103    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-podinfo\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.118183 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.118157    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-credential-socket\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.118315 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.118199    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-token\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.118315 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.118235    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-workload-certs\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.118423 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.118315    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-workload-socket\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.118423 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.118341    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-data\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.118423 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.118381    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-envoy\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.118423 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.118419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istiod-ca-cert\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.118862 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.118754    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-credential-socket\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.119417 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.119367    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-data\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.119530 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.119413    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-workload-socket\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.119822 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.119779    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istiod-ca-cert\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.119920 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.119858    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-workload-certs\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.121449 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.121406    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-envoy\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.122741 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.122669    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-podinfo\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.135590 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.135505    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmc5q\" (UniqueName: \"kubernetes.io/projected/d5c5e730-811c-4c4c-9f65-c907f374fd2e-kube-api-access-wmc5q\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.136414 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.136358    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-token\") pod \"e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") " pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.312325 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.312188    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:34.645847 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.645752    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"]
Sep 12 00:46:34.651598 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:46:34.651550    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c5e730_811c_4c4c_9f65_c907f374fd2e.slice/crio-89b6293dcd63f2e73dec42c98100935a464afd0d04c08faac319c245ebcf765d WatchSource:0}: Error finding container 89b6293dcd63f2e73dec42c98100935a464afd0d04c08faac319c245ebcf765d: Status 404 returned error can't find the container with id 89b6293dcd63f2e73dec42c98100935a464afd0d04c08faac319c245ebcf765d
Sep 12 00:46:34.654552 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.654495    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:34.654682 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.654615    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:34.654682 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:34.654666    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:35.416807 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:35.416714    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br" event={"ID":"d5c5e730-811c-4c4c-9f65-c907f374fd2e","Type":"ContainerStarted","Data":"330fd9f149bbe6c1d2d5218c5050db88e183d69b45cbc6e9d485163bfe26f533"}
Sep 12 00:46:35.417208 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:35.416837    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br" event={"ID":"d5c5e730-811c-4c4c-9f65-c907f374fd2e","Type":"ContainerStarted","Data":"89b6293dcd63f2e73dec42c98100935a464afd0d04c08faac319c245ebcf765d"}
Sep 12 00:46:35.438129 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:35.438047    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br" podStartSLOduration=2.438021428 podStartE2EDuration="2.438021428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:46:35.436355048 +0000 UTC m=+1027.640355870" watchObservedRunningTime="2026-09-12 00:46:35.438021428 +0000 UTC m=+1027.642022250"
Sep 12 00:46:35.971185 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:35.971153    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd"]
Sep 12 00:46:35.975963 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:35.975933    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd"
Sep 12 00:46:35.979185 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:35.978714    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:46:35.979185 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:35.979000    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:46:35.983182 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:35.983082    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd"]
Sep 12 00:46:36.045239 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:36.045189    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323307c8-999e-4c72-92a1-202465079fda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-220bac51-chjrd\" (UID: \"323307c8-999e-4c72-92a1-202465079fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd"
Sep 12 00:46:36.146538 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:36.146480    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323307c8-999e-4c72-92a1-202465079fda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-220bac51-chjrd\" (UID: \"323307c8-999e-4c72-92a1-202465079fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd"
Sep 12 00:46:36.147409 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:36.147373    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323307c8-999e-4c72-92a1-202465079fda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-220bac51-chjrd\" (UID: \"323307c8-999e-4c72-92a1-202465079fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd"
Sep 12 00:46:36.305793 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:36.305692    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd"
Sep 12 00:46:36.313052 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:36.313024    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:36.320314 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:36.320288    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:36.422552 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:36.422521    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:36.423647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:36.423624    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:46:36.625403 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:36.625176    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd"]
Sep 12 00:46:36.630466 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:46:36.630421    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323307c8_999e_4c72_92a1_202465079fda.slice/crio-267f9432270f384b5242b5fedfacb1c629ab9adadf3afe87649ec47f46943329 WatchSource:0}: Error finding container 267f9432270f384b5242b5fedfacb1c629ab9adadf3afe87649ec47f46943329: Status 404 returned error can't find the container with id 267f9432270f384b5242b5fedfacb1c629ab9adadf3afe87649ec47f46943329
Sep 12 00:46:37.428485 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:37.428415    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd" event={"ID":"323307c8-999e-4c72-92a1-202465079fda","Type":"ContainerStarted","Data":"267f9432270f384b5242b5fedfacb1c629ab9adadf3afe87649ec47f46943329"}
Sep 12 00:46:40.448437 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:40.448392    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd" event={"ID":"323307c8-999e-4c72-92a1-202465079fda","Type":"ContainerStarted","Data":"da6558c2eceddff0807d5d685dd2a3f31fd2e7f81d0183174bcd886ac7ea1bca"}
Sep 12 00:46:40.488917 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:40.488859    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd" podStartSLOduration=2.48022681 podStartE2EDuration="5.488840363s" totalImagesPullingTime="3.008613553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:46:40.488674493 +0000 UTC m=+1032.692675329" watchObservedRunningTime="2026-09-12 00:46:40.488840363 +0000 UTC m=+1032.692841184"
Sep 12 00:46:41.392824 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:41.392788    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"
Sep 12 00:46:41.457448 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:41.457406    2567 generic.go:356] "Generic (PLEG): container finished" podID="323307c8-999e-4c72-92a1-202465079fda" containerID="da6558c2eceddff0807d5d685dd2a3f31fd2e7f81d0183174bcd886ac7ea1bca" exitCode=0
Sep 12 00:46:41.457978 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:41.457463    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd" event={"ID":"323307c8-999e-4c72-92a1-202465079fda","Type":"ContainerDied","Data":"da6558c2eceddff0807d5d685dd2a3f31fd2e7f81d0183174bcd886ac7ea1bca"}
Sep 12 00:46:42.464258 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.464225    2567 generic.go:356] "Generic (PLEG): container finished" podID="872c43fb-40e8-4695-8de4-1d85948b786c" containerID="e0aed1d07042242026bf4fcecc3a11e4c8c96bc01928d1038203c2bc56cdd055" exitCode=0
Sep 12 00:46:42.464762 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.464307    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" event={"ID":"872c43fb-40e8-4695-8de4-1d85948b786c","Type":"ContainerDied","Data":"e0aed1d07042242026bf4fcecc3a11e4c8c96bc01928d1038203c2bc56cdd055"}
Sep 12 00:46:42.464812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.464793    2567 scope.go:117] "RemoveContainer" containerID="e0aed1d07042242026bf4fcecc3a11e4c8c96bc01928d1038203c2bc56cdd055"
Sep 12 00:46:42.465980 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.465949    2567 generic.go:356] "Generic (PLEG): container finished" podID="8e53fede-e45a-4fe5-ba60-9ef016babd27" containerID="64408dd8f5b50953010d14b21bfc2942f447f20e3e4885fbc04c829fc475955c" exitCode=0
Sep 12 00:46:42.466098 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.466016    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72" event={"ID":"8e53fede-e45a-4fe5-ba60-9ef016babd27","Type":"ContainerDied","Data":"64408dd8f5b50953010d14b21bfc2942f447f20e3e4885fbc04c829fc475955c"}
Sep 12 00:46:42.466098 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.466074    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:42.466217 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.466167    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:42.466273 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.466243    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:42.466463 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.466448    2567 scope.go:117] "RemoveContainer" containerID="64408dd8f5b50953010d14b21bfc2942f447f20e3e4885fbc04c829fc475955c"
Sep 12 00:46:42.467813 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.467779    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:42.467889 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.467870    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:42.467962 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.467945    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:46:42.638673 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.638620    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd"
Sep 12 00:46:42.729361 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.729320    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323307c8-999e-4c72-92a1-202465079fda-maas-api-service-ca\") pod \"323307c8-999e-4c72-92a1-202465079fda\" (UID: \"323307c8-999e-4c72-92a1-202465079fda\") "
Sep 12 00:46:42.729789 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.729759    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323307c8-999e-4c72-92a1-202465079fda-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "323307c8-999e-4c72-92a1-202465079fda" (UID: "323307c8-999e-4c72-92a1-202465079fda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:46:42.830522 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:42.830427    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323307c8-999e-4c72-92a1-202465079fda-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:46:43.480345 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:43.480295    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" event={"ID":"872c43fb-40e8-4695-8de4-1d85948b786c","Type":"ContainerStarted","Data":"c0b0fd8c7fc7c83a3cadd94ec239000b35aedd018828d249333f9671dabfc87e"}
Sep 12 00:46:43.484693 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:43.484649    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72" event={"ID":"8e53fede-e45a-4fe5-ba60-9ef016babd27","Type":"ContainerStarted","Data":"739212f279c86aac78bccdac67ea415014e9565577c54522dc49031ec00dc7cb"}
Sep 12 00:46:43.488144 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:43.488109    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd" event={"ID":"323307c8-999e-4c72-92a1-202465079fda","Type":"ContainerDied","Data":"267f9432270f384b5242b5fedfacb1c629ab9adadf3afe87649ec47f46943329"}
Sep 12 00:46:43.488382 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:43.488365    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="267f9432270f384b5242b5fedfacb1c629ab9adadf3afe87649ec47f46943329"
Sep 12 00:46:43.488538 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:43.488525    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd"
Sep 12 00:46:43.948336 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:43.948235    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:43.948776 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:43.948750    2567 patch_prober.go:28] interesting pod/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 12 00:46:43.948969 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:43.948943    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 12 00:46:44.296985 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:44.296944    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:44.297235 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:44.296988    2567 patch_prober.go:28] interesting pod/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 12 00:46:44.297337 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:44.297289    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72" podUID="8e53fede-e45a-4fe5-ba60-9ef016babd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 12 00:46:44.948160 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:44.948109    2567 patch_prober.go:28] interesting pod/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 12 00:46:44.948708 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:44.948186    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 12 00:46:45.296723 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:45.296681    2567 patch_prober.go:28] interesting pod/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 12 00:46:45.296917 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:45.296741    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72" podUID="8e53fede-e45a-4fe5-ba60-9ef016babd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 12 00:46:45.948296 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:45.948243    2567 patch_prober.go:28] interesting pod/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 12 00:46:45.948872 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:45.948323    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 12 00:46:46.297257 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:46.297220    2567 patch_prober.go:28] interesting pod/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 12 00:46:46.297441 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:46.297286    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72" podUID="8e53fede-e45a-4fe5-ba60-9ef016babd27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 12 00:46:46.729887 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:46.729827    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd"]
Sep 12 00:46:46.731785 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:46.731759    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-220bac51-chjrd"]
Sep 12 00:46:46.864600 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:46.858231    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"]
Sep 12 00:46:46.864600 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:46.858486    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72" podUID="8e53fede-e45a-4fe5-ba60-9ef016babd27" containerName="istio-proxy" containerID="cri-o://739212f279c86aac78bccdac67ea415014e9565577c54522dc49031ec00dc7cb" gracePeriod=30
Sep 12 00:46:46.951605 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:46.949134    2567 patch_prober.go:28] interesting pod/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 12 00:46:46.951605 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:46.949220    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 12 00:46:47.950068 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:47.950018    2567 patch_prober.go:28] interesting pod/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 12 00:46:47.950289 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:47.950092    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 12 00:46:48.277848 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:48.277799    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="323307c8-999e-4c72-92a1-202465079fda" path="/var/lib/kubelet/pods/323307c8-999e-4c72-92a1-202465079fda/volumes"
Sep 12 00:46:48.961313 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:48.961273    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:48.962187 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:48.962155    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:48.965172 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:48.965135    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:46:50.276205 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:50.276172    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:54.436163 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.436133    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:54.564825 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.564725    2567 generic.go:356] "Generic (PLEG): container finished" podID="8e53fede-e45a-4fe5-ba60-9ef016babd27" containerID="739212f279c86aac78bccdac67ea415014e9565577c54522dc49031ec00dc7cb" exitCode=0
Sep 12 00:46:54.564825 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.564779    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"
Sep 12 00:46:54.564825 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.564805    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72" event={"ID":"8e53fede-e45a-4fe5-ba60-9ef016babd27","Type":"ContainerDied","Data":"739212f279c86aac78bccdac67ea415014e9565577c54522dc49031ec00dc7cb"}
Sep 12 00:46:54.565045 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.564870    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72" event={"ID":"8e53fede-e45a-4fe5-ba60-9ef016babd27","Type":"ContainerDied","Data":"d6b8667e33b1a815faeaf164e3f5205abcd59fffb281d9a1bbb82fa0c43e0733"}
Sep 12 00:46:54.565045 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.564895    2567 scope.go:117] "RemoveContainer" containerID="739212f279c86aac78bccdac67ea415014e9565577c54522dc49031ec00dc7cb"
Sep 12 00:46:54.573654 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.573632    2567 scope.go:117] "RemoveContainer" containerID="64408dd8f5b50953010d14b21bfc2942f447f20e3e4885fbc04c829fc475955c"
Sep 12 00:46:54.581889 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.581863    2567 scope.go:117] "RemoveContainer" containerID="739212f279c86aac78bccdac67ea415014e9565577c54522dc49031ec00dc7cb"
Sep 12 00:46:54.582196 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:46:54.582176    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739212f279c86aac78bccdac67ea415014e9565577c54522dc49031ec00dc7cb\": container with ID starting with 739212f279c86aac78bccdac67ea415014e9565577c54522dc49031ec00dc7cb not found: ID does not exist" containerID="739212f279c86aac78bccdac67ea415014e9565577c54522dc49031ec00dc7cb"
Sep 12 00:46:54.582252 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.582205    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739212f279c86aac78bccdac67ea415014e9565577c54522dc49031ec00dc7cb"} err="failed to get container status \"739212f279c86aac78bccdac67ea415014e9565577c54522dc49031ec00dc7cb\": rpc error: code = NotFound desc = could not find container \"739212f279c86aac78bccdac67ea415014e9565577c54522dc49031ec00dc7cb\": container with ID starting with 739212f279c86aac78bccdac67ea415014e9565577c54522dc49031ec00dc7cb not found: ID does not exist"
Sep 12 00:46:54.582252 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.582223    2567 scope.go:117] "RemoveContainer" containerID="64408dd8f5b50953010d14b21bfc2942f447f20e3e4885fbc04c829fc475955c"
Sep 12 00:46:54.582510 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:46:54.582487    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64408dd8f5b50953010d14b21bfc2942f447f20e3e4885fbc04c829fc475955c\": container with ID starting with 64408dd8f5b50953010d14b21bfc2942f447f20e3e4885fbc04c829fc475955c not found: ID does not exist" containerID="64408dd8f5b50953010d14b21bfc2942f447f20e3e4885fbc04c829fc475955c"
Sep 12 00:46:54.582580 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.582523    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64408dd8f5b50953010d14b21bfc2942f447f20e3e4885fbc04c829fc475955c"} err="failed to get container status \"64408dd8f5b50953010d14b21bfc2942f447f20e3e4885fbc04c829fc475955c\": rpc error: code = NotFound desc = could not find container \"64408dd8f5b50953010d14b21bfc2942f447f20e3e4885fbc04c829fc475955c\": container with ID starting with 64408dd8f5b50953010d14b21bfc2942f447f20e3e4885fbc04c829fc475955c not found: ID does not exist"
Sep 12 00:46:54.591848 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.591816    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-credential-socket\") pod \"8e53fede-e45a-4fe5-ba60-9ef016babd27\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") "
Sep 12 00:46:54.591993 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.591865    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-workload-socket\") pod \"8e53fede-e45a-4fe5-ba60-9ef016babd27\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") "
Sep 12 00:46:54.591993 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.591887    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d86tn\" (UniqueName: \"kubernetes.io/projected/8e53fede-e45a-4fe5-ba60-9ef016babd27-kube-api-access-d86tn\") pod \"8e53fede-e45a-4fe5-ba60-9ef016babd27\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") "
Sep 12 00:46:54.591993 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.591923    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-data\") pod \"8e53fede-e45a-4fe5-ba60-9ef016babd27\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") "
Sep 12 00:46:54.591993 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.591955    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-token\") pod \"8e53fede-e45a-4fe5-ba60-9ef016babd27\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") "
Sep 12 00:46:54.591993 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.591990    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-podinfo\") pod \"8e53fede-e45a-4fe5-ba60-9ef016babd27\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") "
Sep 12 00:46:54.592259 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.592124    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e53fede-e45a-4fe5-ba60-9ef016babd27" (UID: "8e53fede-e45a-4fe5-ba60-9ef016babd27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:54.592259 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.592180    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e53fede-e45a-4fe5-ba60-9ef016babd27" (UID: "8e53fede-e45a-4fe5-ba60-9ef016babd27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:54.592368 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.592322    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e53fede-e45a-4fe5-ba60-9ef016babd27" (UID: "8e53fede-e45a-4fe5-ba60-9ef016babd27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:54.592368 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.592361    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e53fede-e45a-4fe5-ba60-9ef016babd27-istiod-ca-cert\") pod \"8e53fede-e45a-4fe5-ba60-9ef016babd27\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") "
Sep 12 00:46:54.592488 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.592410    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-envoy\") pod \"8e53fede-e45a-4fe5-ba60-9ef016babd27\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") "
Sep 12 00:46:54.592488 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.592436    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e53fede-e45a-4fe5-ba60-9ef016babd27-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e53fede-e45a-4fe5-ba60-9ef016babd27" (UID: "8e53fede-e45a-4fe5-ba60-9ef016babd27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:46:54.592488 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.592442    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-workload-certs\") pod \"8e53fede-e45a-4fe5-ba60-9ef016babd27\" (UID: \"8e53fede-e45a-4fe5-ba60-9ef016babd27\") "
Sep 12 00:46:54.592705 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.592683    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e53fede-e45a-4fe5-ba60-9ef016babd27" (UID: "8e53fede-e45a-4fe5-ba60-9ef016babd27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:54.592866 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.592850    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e53fede-e45a-4fe5-ba60-9ef016babd27-istiod-ca-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:46:54.592927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.592867    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-workload-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:46:54.592927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.592876    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-credential-socket\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:46:54.592927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.592886    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-workload-socket\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:46:54.592927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.592898    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-data\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:46:54.594505 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.594470    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e53fede-e45a-4fe5-ba60-9ef016babd27" (UID: "8e53fede-e45a-4fe5-ba60-9ef016babd27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 00:46:54.594645 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.594508    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e53fede-e45a-4fe5-ba60-9ef016babd27" (UID: "8e53fede-e45a-4fe5-ba60-9ef016babd27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:46:54.594698 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.594638    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e53fede-e45a-4fe5-ba60-9ef016babd27" (UID: "8e53fede-e45a-4fe5-ba60-9ef016babd27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:46:54.594698 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.594669    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e53fede-e45a-4fe5-ba60-9ef016babd27-kube-api-access-d86tn" (OuterVolumeSpecName: "kube-api-access-d86tn") pod "8e53fede-e45a-4fe5-ba60-9ef016babd27" (UID: "8e53fede-e45a-4fe5-ba60-9ef016babd27"). InnerVolumeSpecName "kube-api-access-d86tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:46:54.694127 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.694086    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-token\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:46:54.694127 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.694120    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-podinfo\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:46:54.694127 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.694129    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e53fede-e45a-4fe5-ba60-9ef016babd27-istio-envoy\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:46:54.694127 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.694138    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d86tn\" (UniqueName: \"kubernetes.io/projected/8e53fede-e45a-4fe5-ba60-9ef016babd27-kube-api-access-d86tn\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:46:54.887821 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.887783    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"]
Sep 12 00:46:54.892131 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:54.892104    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-220bac51-openshift-default-5b57cb59c6-7sj72"]
Sep 12 00:46:55.597910 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.597879    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"]
Sep 12 00:46:55.598302 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.598216    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e53fede-e45a-4fe5-ba60-9ef016babd27" containerName="istio-proxy"
Sep 12 00:46:55.598302 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.598227    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e53fede-e45a-4fe5-ba60-9ef016babd27" containerName="istio-proxy"
Sep 12 00:46:55.598302 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.598245    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="323307c8-999e-4c72-92a1-202465079fda" containerName="revoke-keys"
Sep 12 00:46:55.598302 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.598250    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="323307c8-999e-4c72-92a1-202465079fda" containerName="revoke-keys"
Sep 12 00:46:55.598435 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.598305    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e53fede-e45a-4fe5-ba60-9ef016babd27" containerName="istio-proxy"
Sep 12 00:46:55.598435 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.598314    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="323307c8-999e-4c72-92a1-202465079fda" containerName="revoke-keys"
Sep 12 00:46:55.600157 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.600139    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.602492 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.602471    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-876808\"/\"kube-root-ca.crt\""
Sep 12 00:46:55.603724 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.603706    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-876808\"/\"openshift-service-ca.crt\""
Sep 12 00:46:55.603821 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.603739    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-876808\"/\"default-dockercfg-2c5sq\""
Sep 12 00:46:55.603821 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.603783    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-876808\"/\"simulator-w4-876808-kserve-self-signed-certs\""
Sep 12 00:46:55.609775 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.609741    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"]
Sep 12 00:46:55.705532 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.705490    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-home\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.705532 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.705532    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdmlm\" (UniqueName: \"kubernetes.io/projected/5f441720-a3c0-4875-9313-26f87334f860-kube-api-access-qdmlm\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.705789 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.705560    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-tmp-dir\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.705789 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.705655    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-model-cache\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.705789 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.705707    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f441720-a3c0-4875-9313-26f87334f860-tls-certs\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.705922 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.705798    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-dshm\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.806970 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.806908    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-dshm\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.806970 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.806977    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-home\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.807258 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.807011    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdmlm\" (UniqueName: \"kubernetes.io/projected/5f441720-a3c0-4875-9313-26f87334f860-kube-api-access-qdmlm\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.807258 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.807042    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-tmp-dir\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.807258 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.807142    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-model-cache\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.807258 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.807206    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f441720-a3c0-4875-9313-26f87334f860-tls-certs\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.807911 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.807854    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-tmp-dir\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.808081 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.808052    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-model-cache\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.808168 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.808155    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-home\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.809765 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.809740    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-dshm\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.809955 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.809929    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f441720-a3c0-4875-9313-26f87334f860-tls-certs\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.818884 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.818858    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdmlm\" (UniqueName: \"kubernetes.io/projected/5f441720-a3c0-4875-9313-26f87334f860-kube-api-access-qdmlm\") pod \"simulator-w4-876808-kserve-57cdb897cb-d29mg\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") " pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:55.909650 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:55.909544    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:46:56.205689 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:56.205654    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"]
Sep 12 00:46:56.208349 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:46:56.208314    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f441720_a3c0_4875_9313_26f87334f860.slice/crio-1c0a965d95c8148fcaef17b09df2ea8aa7007ef5a13c84c7ad78f4246bdfabca WatchSource:0}: Error finding container 1c0a965d95c8148fcaef17b09df2ea8aa7007ef5a13c84c7ad78f4246bdfabca: Status 404 returned error can't find the container with id 1c0a965d95c8148fcaef17b09df2ea8aa7007ef5a13c84c7ad78f4246bdfabca
Sep 12 00:46:56.274018 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:56.273984    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e53fede-e45a-4fe5-ba60-9ef016babd27" path="/var/lib/kubelet/pods/8e53fede-e45a-4fe5-ba60-9ef016babd27/volumes"
Sep 12 00:46:56.580029 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:56.579947    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg" event={"ID":"5f441720-a3c0-4875-9313-26f87334f860","Type":"ContainerStarted","Data":"152319cf398337aafb839a7d6f3f7b6e66279a475096b80c460ce18e96e781e5"}
Sep 12 00:46:56.580029 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:56.579995    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg" event={"ID":"5f441720-a3c0-4875-9313-26f87334f860","Type":"ContainerStarted","Data":"1c0a965d95c8148fcaef17b09df2ea8aa7007ef5a13c84c7ad78f4246bdfabca"}
Sep 12 00:46:56.599141 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:46:56.599053    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg" podStartSLOduration=1.59903317 podStartE2EDuration="1.59903317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:46:56.59673134 +0000 UTC m=+1048.800732163" watchObservedRunningTime="2026-09-12 00:46:56.59903317 +0000 UTC m=+1048.803033982"
Sep 12 00:47:00.601937 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:00.601899    2567 generic.go:356] "Generic (PLEG): container finished" podID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerID="330fd9f149bbe6c1d2d5218c5050db88e183d69b45cbc6e9d485163bfe26f533" exitCode=0
Sep 12 00:47:00.602379 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:00.601975    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br" event={"ID":"d5c5e730-811c-4c4c-9f65-c907f374fd2e","Type":"ContainerDied","Data":"330fd9f149bbe6c1d2d5218c5050db88e183d69b45cbc6e9d485163bfe26f533"}
Sep 12 00:47:00.602424 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:00.602389    2567 scope.go:117] "RemoveContainer" containerID="330fd9f149bbe6c1d2d5218c5050db88e183d69b45cbc6e9d485163bfe26f533"
Sep 12 00:47:00.603677 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:00.603646    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:47:00.603801 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:00.603729    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:47:00.603801 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:00.603776    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 00:47:01.608366 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:01.608328    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br" event={"ID":"d5c5e730-811c-4c4c-9f65-c907f374fd2e","Type":"ContainerStarted","Data":"ea6741d5cd678b1c8387db25439ad85c9661f6731ab7a4c28394ce1f563f298c"}
Sep 12 00:47:02.312429 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:02.312383    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:47:02.313558 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:02.313534    2567 patch_prober.go:28] interesting pod/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 12 00:47:02.313637 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:02.313600    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br" podUID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 12 00:47:03.313601 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:03.313540    2567 patch_prober.go:28] interesting pod/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 12 00:47:03.314110 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:03.313633    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br" podUID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 12 00:47:04.313158 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:04.313117    2567 patch_prober.go:28] interesting pod/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 12 00:47:04.313346 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:04.313187    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br" podUID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 12 00:47:05.312847 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:05.312809    2567 patch_prober.go:28] interesting pod/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 12 00:47:05.313256 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:05.312881    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br" podUID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 12 00:47:05.910759 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:05.910702    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:47:05.910975 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:05.910772    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:47:05.923515 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:05.923479    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:47:06.312995 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:06.312953    2567 patch_prober.go:28] interesting pod/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 12 00:47:06.313420 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:06.313019    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br" podUID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 12 00:47:06.422957 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:06.422921    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:47:06.668249 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:06.668151    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 00:47:07.318115 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:07.318078    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:47:07.319382 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:07.319358    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:47:15.062271 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.062230    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp"]
Sep 12 00:47:15.062727 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.062613    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e53fede-e45a-4fe5-ba60-9ef016babd27" containerName="istio-proxy"
Sep 12 00:47:15.062727 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.062628    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e53fede-e45a-4fe5-ba60-9ef016babd27" containerName="istio-proxy"
Sep 12 00:47:15.062727 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.062708    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e53fede-e45a-4fe5-ba60-9ef016babd27" containerName="istio-proxy"
Sep 12 00:47:15.067949 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.067905    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp"
Sep 12 00:47:15.070963 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.070780    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:47:15.070963 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.070813    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:47:15.070963 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.070887    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp"]
Sep 12 00:47:15.101427 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.101382    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d761409-477e-4148-bec3-aa01664cbb2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp\" (UID: \"9d761409-477e-4148-bec3-aa01664cbb2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp"
Sep 12 00:47:15.202457 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.202414    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d761409-477e-4148-bec3-aa01664cbb2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp\" (UID: \"9d761409-477e-4148-bec3-aa01664cbb2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp"
Sep 12 00:47:15.203120 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.203097    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d761409-477e-4148-bec3-aa01664cbb2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp\" (UID: \"9d761409-477e-4148-bec3-aa01664cbb2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp"
Sep 12 00:47:15.380776 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.380675    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp"
Sep 12 00:47:15.649207 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.649180    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp"]
Sep 12 00:47:15.651685 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:47:15.651653    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d761409_477e_4148_bec3_aa01664cbb2c.slice/crio-2c21c75e230dc93b7ae889f7d74f582d1d11157d02427f2af832970fe15efbc6 WatchSource:0}: Error finding container 2c21c75e230dc93b7ae889f7d74f582d1d11157d02427f2af832970fe15efbc6: Status 404 returned error can't find the container with id 2c21c75e230dc93b7ae889f7d74f582d1d11157d02427f2af832970fe15efbc6
Sep 12 00:47:15.678317 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:15.678288    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp" event={"ID":"9d761409-477e-4148-bec3-aa01664cbb2c","Type":"ContainerStarted","Data":"2c21c75e230dc93b7ae889f7d74f582d1d11157d02427f2af832970fe15efbc6"}
Sep 12 00:47:16.684729 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:16.684638    2567 generic.go:356] "Generic (PLEG): container finished" podID="9d761409-477e-4148-bec3-aa01664cbb2c" containerID="c8545d6d9a0b1eba863721efb5b8b04c5734af0558ab39cde917f587f5fe3064" exitCode=0
Sep 12 00:47:16.685104 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:16.684729    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp" event={"ID":"9d761409-477e-4148-bec3-aa01664cbb2c","Type":"ContainerDied","Data":"c8545d6d9a0b1eba863721efb5b8b04c5734af0558ab39cde917f587f5fe3064"}
Sep 12 00:47:17.829829 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:17.829463    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp"
Sep 12 00:47:17.928272 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:17.928237    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d761409-477e-4148-bec3-aa01664cbb2c-maas-api-service-ca\") pod \"9d761409-477e-4148-bec3-aa01664cbb2c\" (UID: \"9d761409-477e-4148-bec3-aa01664cbb2c\") "
Sep 12 00:47:17.928685 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:17.928656    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d761409-477e-4148-bec3-aa01664cbb2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d761409-477e-4148-bec3-aa01664cbb2c" (UID: "9d761409-477e-4148-bec3-aa01664cbb2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:47:18.029690 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:18.029655    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d761409-477e-4148-bec3-aa01664cbb2c-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:47:18.695490 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:18.695407    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp"
Sep 12 00:47:18.695490 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:18.695418    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp" event={"ID":"9d761409-477e-4148-bec3-aa01664cbb2c","Type":"ContainerDied","Data":"2c21c75e230dc93b7ae889f7d74f582d1d11157d02427f2af832970fe15efbc6"}
Sep 12 00:47:18.695490 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:18.695464    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c21c75e230dc93b7ae889f7d74f582d1d11157d02427f2af832970fe15efbc6"
Sep 12 00:47:22.053508 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:22.053406    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp"]
Sep 12 00:47:22.055560 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:22.055535    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-723148b6-jg6vp"]
Sep 12 00:47:22.273945 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:22.273910    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d761409-477e-4148-bec3-aa01664cbb2c" path="/var/lib/kubelet/pods/9d761409-477e-4148-bec3-aa01664cbb2c/volumes"
Sep 12 00:47:27.596144 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.596106    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"]
Sep 12 00:47:27.596628 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.596474    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d761409-477e-4148-bec3-aa01664cbb2c" containerName="revoke-keys"
Sep 12 00:47:27.596628 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.596485    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d761409-477e-4148-bec3-aa01664cbb2c" containerName="revoke-keys"
Sep 12 00:47:27.596628 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.596554    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d761409-477e-4148-bec3-aa01664cbb2c" containerName="revoke-keys"
Sep 12 00:47:27.598920 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.598903    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"
Sep 12 00:47:27.601667 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.601641    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-51ad044d-dockercfg-xsvml\""
Sep 12 00:47:27.601836 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.601818    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-51ad044d\""
Sep 12 00:47:27.609158 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.609128    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"]
Sep 12 00:47:27.637326 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.637283    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"]
Sep 12 00:47:27.639861 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.639838    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"
Sep 12 00:47:27.649859 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.649833    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"]
Sep 12 00:47:27.722760 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.722703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrddk\" (UniqueName: \"kubernetes.io/projected/0a7e34e4-4320-45b0-a743-af876d1fd760-kube-api-access-jrddk\") pod \"payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w\" (UID: \"0a7e34e4-4320-45b0-a743-af876d1fd760\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"
Sep 12 00:47:27.722979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.722836    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9231c398-67f3-46a0-b8dd-3e6b3288dedf-plugins-config-volume\") pod \"payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8\" (UID: \"9231c398-67f3-46a0-b8dd-3e6b3288dedf\") " pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"
Sep 12 00:47:27.722979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.722900    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7e34e4-4320-45b0-a743-af876d1fd760-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w\" (UID: \"0a7e34e4-4320-45b0-a743-af876d1fd760\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"
Sep 12 00:47:27.723097 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.722975    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s25x\" (UniqueName: \"kubernetes.io/projected/9231c398-67f3-46a0-b8dd-3e6b3288dedf-kube-api-access-6s25x\") pod \"payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8\" (UID: \"9231c398-67f3-46a0-b8dd-3e6b3288dedf\") " pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"
Sep 12 00:47:27.824242 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.824193    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s25x\" (UniqueName: \"kubernetes.io/projected/9231c398-67f3-46a0-b8dd-3e6b3288dedf-kube-api-access-6s25x\") pod \"payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8\" (UID: \"9231c398-67f3-46a0-b8dd-3e6b3288dedf\") " pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"
Sep 12 00:47:27.824443 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.824295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrddk\" (UniqueName: \"kubernetes.io/projected/0a7e34e4-4320-45b0-a743-af876d1fd760-kube-api-access-jrddk\") pod \"payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w\" (UID: \"0a7e34e4-4320-45b0-a743-af876d1fd760\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"
Sep 12 00:47:27.824443 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.824367    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9231c398-67f3-46a0-b8dd-3e6b3288dedf-plugins-config-volume\") pod \"payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8\" (UID: \"9231c398-67f3-46a0-b8dd-3e6b3288dedf\") " pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"
Sep 12 00:47:27.824443 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.824416    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7e34e4-4320-45b0-a743-af876d1fd760-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w\" (UID: \"0a7e34e4-4320-45b0-a743-af876d1fd760\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"
Sep 12 00:47:27.825142 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.825118    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9231c398-67f3-46a0-b8dd-3e6b3288dedf-plugins-config-volume\") pod \"payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8\" (UID: \"9231c398-67f3-46a0-b8dd-3e6b3288dedf\") " pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"
Sep 12 00:47:27.825204 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.825121    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7e34e4-4320-45b0-a743-af876d1fd760-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w\" (UID: \"0a7e34e4-4320-45b0-a743-af876d1fd760\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"
Sep 12 00:47:27.832455 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.832425    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s25x\" (UniqueName: \"kubernetes.io/projected/9231c398-67f3-46a0-b8dd-3e6b3288dedf-kube-api-access-6s25x\") pod \"payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8\" (UID: \"9231c398-67f3-46a0-b8dd-3e6b3288dedf\") " pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"
Sep 12 00:47:27.832605 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.832507    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrddk\" (UniqueName: \"kubernetes.io/projected/0a7e34e4-4320-45b0-a743-af876d1fd760-kube-api-access-jrddk\") pod \"payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w\" (UID: \"0a7e34e4-4320-45b0-a743-af876d1fd760\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"
Sep 12 00:47:27.910312 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.910211    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"
Sep 12 00:47:27.951261 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:27.951164    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"
Sep 12 00:47:28.185818 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:28.185671    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"]
Sep 12 00:47:28.188502 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:47:28.188472    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7e34e4_4320_45b0_a743_af876d1fd760.slice/crio-95805182406fd42b6bfa9745ffb2c52cb1c9676c9c3c09f54386be4415756cda WatchSource:0}: Error finding container 95805182406fd42b6bfa9745ffb2c52cb1c9676c9c3c09f54386be4415756cda: Status 404 returned error can't find the container with id 95805182406fd42b6bfa9745ffb2c52cb1c9676c9c3c09f54386be4415756cda
Sep 12 00:47:28.233297 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:28.233269    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"]
Sep 12 00:47:28.235260 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:47:28.235227    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9231c398_67f3_46a0_b8dd_3e6b3288dedf.slice/crio-f7a35fd9d5cf72fcdbfed0ca5e4c73b9f48082f624256220ae8c8103861723fd WatchSource:0}: Error finding container f7a35fd9d5cf72fcdbfed0ca5e4c73b9f48082f624256220ae8c8103861723fd: Status 404 returned error can't find the container with id f7a35fd9d5cf72fcdbfed0ca5e4c73b9f48082f624256220ae8c8103861723fd
Sep 12 00:47:28.748262 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:28.748221    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8" event={"ID":"9231c398-67f3-46a0-b8dd-3e6b3288dedf","Type":"ContainerStarted","Data":"ac6b05cc8da23f113d023b4bee93011591b9943a82144d07fe6193011397fa2e"}
Sep 12 00:47:28.748787 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:28.748287    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8" event={"ID":"9231c398-67f3-46a0-b8dd-3e6b3288dedf","Type":"ContainerStarted","Data":"f7a35fd9d5cf72fcdbfed0ca5e4c73b9f48082f624256220ae8c8103861723fd"}
Sep 12 00:47:28.748787 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:28.748415    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"
Sep 12 00:47:28.749927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:28.749895    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w" event={"ID":"0a7e34e4-4320-45b0-a743-af876d1fd760","Type":"ContainerStarted","Data":"ca506526a3181d2fcdd52f745a159a35e7b3a950fffeab29db1e48972992aa62"}
Sep 12 00:47:28.750061 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:28.749948    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w" event={"ID":"0a7e34e4-4320-45b0-a743-af876d1fd760","Type":"ContainerStarted","Data":"95805182406fd42b6bfa9745ffb2c52cb1c9676c9c3c09f54386be4415756cda"}
Sep 12 00:47:28.750061 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:28.749987    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"
Sep 12 00:47:28.770252 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:28.770196    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8" podStartSLOduration=1.7701745070000001 podStartE2EDuration="1.770174507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:47:28.766074992 +0000 UTC m=+1080.970075814" watchObservedRunningTime="2026-09-12 00:47:28.770174507 +0000 UTC m=+1080.974175332"
Sep 12 00:47:28.788401 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:28.788336    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w" podStartSLOduration=1.78831799 podStartE2EDuration="1.78831799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:47:28.787360103 +0000 UTC m=+1080.991360926" watchObservedRunningTime="2026-09-12 00:47:28.78831799 +0000 UTC m=+1080.992318811"
Sep 12 00:47:33.196090 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.196052    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"]
Sep 12 00:47:33.199328 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.199305    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.201782 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.201756    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4d485e\"/\"openshift-service-ca.crt\""
Sep 12 00:47:33.201909 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.201756    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4d485e\"/\"kube-root-ca.crt\""
Sep 12 00:47:33.202814 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.202793    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4d485e\"/\"simulator-w0-4d485e-kserve-self-signed-certs\""
Sep 12 00:47:33.202922 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.202811    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4d485e\"/\"default-dockercfg-nvltd\""
Sep 12 00:47:33.206789 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.206762    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"]
Sep 12 00:47:33.278209 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.278168    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-tmp-dir\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.278396 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.278228    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2509fb91-4691-47f0-8985-7aa084a93cda-tls-certs\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.278396 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.278278    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-dshm\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.278396 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.278305    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkplv\" (UniqueName: \"kubernetes.io/projected/2509fb91-4691-47f0-8985-7aa084a93cda-kube-api-access-xkplv\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.278396 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.278347    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-model-cache\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.278528 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.278418    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-home\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.379692 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.379647    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2509fb91-4691-47f0-8985-7aa084a93cda-tls-certs\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.379879 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.379710    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-dshm\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.379879 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.379746    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkplv\" (UniqueName: \"kubernetes.io/projected/2509fb91-4691-47f0-8985-7aa084a93cda-kube-api-access-xkplv\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.379879 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.379802    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-model-cache\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.379879 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.379850    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-home\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.380090 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.379905    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-tmp-dir\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.380240 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.380219    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-model-cache\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.380341 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.380322    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-tmp-dir\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.380390 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.380336    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-home\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.382104 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.382071    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-dshm\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.382317 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.382298    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2509fb91-4691-47f0-8985-7aa084a93cda-tls-certs\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.387872 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.387849    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkplv\" (UniqueName: \"kubernetes.io/projected/2509fb91-4691-47f0-8985-7aa084a93cda-kube-api-access-xkplv\") pod \"simulator-w0-4d485e-kserve-69d8cdc45f-gq86t\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") " pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.509768 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.509731    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:33.780530 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:33.780451    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"]
Sep 12 00:47:33.783710 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:47:33.783680    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2509fb91_4691_47f0_8985_7aa084a93cda.slice/crio-ff849e88075cbd9fb674b98feebf983d2b05403cd1a12d6bb30c614ad90cfbaa WatchSource:0}: Error finding container ff849e88075cbd9fb674b98feebf983d2b05403cd1a12d6bb30c614ad90cfbaa: Status 404 returned error can't find the container with id ff849e88075cbd9fb674b98feebf983d2b05403cd1a12d6bb30c614ad90cfbaa
Sep 12 00:47:34.783953 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:34.783915    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t" event={"ID":"2509fb91-4691-47f0-8985-7aa084a93cda","Type":"ContainerStarted","Data":"62275a0c5676b8caee857c232533eac8731c2f1cb7b3546030c6ac14821d3a92"}
Sep 12 00:47:34.784405 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:34.783965    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t" event={"ID":"2509fb91-4691-47f0-8985-7aa084a93cda","Type":"ContainerStarted","Data":"ff849e88075cbd9fb674b98feebf983d2b05403cd1a12d6bb30c614ad90cfbaa"}
Sep 12 00:47:34.801529 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:34.801474    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t" podStartSLOduration=1.801457281 podStartE2EDuration="1.801457281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:47:34.801184046 +0000 UTC m=+1087.005184883" watchObservedRunningTime="2026-09-12 00:47:34.801457281 +0000 UTC m=+1087.005458103"
Sep 12 00:47:39.757379 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:39.757346    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"
Sep 12 00:47:39.757802 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:39.757422    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"
Sep 12 00:47:43.510145 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:43.510078    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:43.510598 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:43.510167    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:43.522788 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:43.522757    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:43.840989 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:43.840894    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:47:50.972498 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:50.972462    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8"]
Sep 12 00:47:50.975115 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:50.975097    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8"
Sep 12 00:47:50.977721 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:50.977698    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:47:50.977838 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:50.977719    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:47:50.982833 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:50.982800    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8"]
Sep 12 00:47:51.058457 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:51.058417    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358ff3a3-91b3-4e82-abc5-d838e57e4885-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8\" (UID: \"358ff3a3-91b3-4e82-abc5-d838e57e4885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8"
Sep 12 00:47:51.159999 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:51.159959    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358ff3a3-91b3-4e82-abc5-d838e57e4885-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8\" (UID: \"358ff3a3-91b3-4e82-abc5-d838e57e4885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8"
Sep 12 00:47:51.160624 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:51.160601    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358ff3a3-91b3-4e82-abc5-d838e57e4885-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8\" (UID: \"358ff3a3-91b3-4e82-abc5-d838e57e4885\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8"
Sep 12 00:47:51.288616 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:51.288510    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8"
Sep 12 00:47:51.559520 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:51.559487    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8"]
Sep 12 00:47:51.561688 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:47:51.561661    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358ff3a3_91b3_4e82_abc5_d838e57e4885.slice/crio-bda3afe86efca32695e3c9e7a16ca017417b18b189788ad7205beb1efc292bd0 WatchSource:0}: Error finding container bda3afe86efca32695e3c9e7a16ca017417b18b189788ad7205beb1efc292bd0: Status 404 returned error can't find the container with id bda3afe86efca32695e3c9e7a16ca017417b18b189788ad7205beb1efc292bd0
Sep 12 00:47:51.871775 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:51.871737    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8" event={"ID":"358ff3a3-91b3-4e82-abc5-d838e57e4885","Type":"ContainerStarted","Data":"925b6505e1c4e4e8f49c2e2bd9d421d3188abb8402b81dc0c3bc09fe915e3ad9"}
Sep 12 00:47:51.871950 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:51.871792    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8" event={"ID":"358ff3a3-91b3-4e82-abc5-d838e57e4885","Type":"ContainerStarted","Data":"bda3afe86efca32695e3c9e7a16ca017417b18b189788ad7205beb1efc292bd0"}
Sep 12 00:47:51.889688 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:51.889640    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8" podStartSLOduration=1.889623391 podStartE2EDuration="1.889623391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:47:51.888154332 +0000 UTC m=+1104.092155153" watchObservedRunningTime="2026-09-12 00:47:51.889623391 +0000 UTC m=+1104.093624212"
Sep 12 00:47:52.881222 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:52.881188    2567 generic.go:356] "Generic (PLEG): container finished" podID="358ff3a3-91b3-4e82-abc5-d838e57e4885" containerID="925b6505e1c4e4e8f49c2e2bd9d421d3188abb8402b81dc0c3bc09fe915e3ad9" exitCode=0
Sep 12 00:47:52.881639 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:52.881270    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8" event={"ID":"358ff3a3-91b3-4e82-abc5-d838e57e4885","Type":"ContainerDied","Data":"925b6505e1c4e4e8f49c2e2bd9d421d3188abb8402b81dc0c3bc09fe915e3ad9"}
Sep 12 00:47:54.021617 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:54.021590    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8"
Sep 12 00:47:54.091112 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:54.091075    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358ff3a3-91b3-4e82-abc5-d838e57e4885-maas-api-service-ca\") pod \"358ff3a3-91b3-4e82-abc5-d838e57e4885\" (UID: \"358ff3a3-91b3-4e82-abc5-d838e57e4885\") "
Sep 12 00:47:54.091479 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:54.091452    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358ff3a3-91b3-4e82-abc5-d838e57e4885-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "358ff3a3-91b3-4e82-abc5-d838e57e4885" (UID: "358ff3a3-91b3-4e82-abc5-d838e57e4885"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:47:54.192502 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:54.192413    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358ff3a3-91b3-4e82-abc5-d838e57e4885-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:47:54.894153 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:54.894118    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8"
Sep 12 00:47:54.894348 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:54.894157    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8" event={"ID":"358ff3a3-91b3-4e82-abc5-d838e57e4885","Type":"ContainerDied","Data":"bda3afe86efca32695e3c9e7a16ca017417b18b189788ad7205beb1efc292bd0"}
Sep 12 00:47:54.894348 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:54.894215    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bda3afe86efca32695e3c9e7a16ca017417b18b189788ad7205beb1efc292bd0"
Sep 12 00:47:57.161887 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.161852    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"]
Sep 12 00:47:57.162332 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.162094    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8" podUID="9231c398-67f3-46a0-b8dd-3e6b3288dedf" containerName="payload-processing" containerID="cri-o://ac6b05cc8da23f113d023b4bee93011591b9943a82144d07fe6193011397fa2e" gracePeriod=30
Sep 12 00:47:57.171262 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.170999    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"]
Sep 12 00:47:57.171471 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.171440    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w" podUID="0a7e34e4-4320-45b0-a743-af876d1fd760" containerName="payload-pre-processing" containerID="cri-o://ca506526a3181d2fcdd52f745a159a35e7b3a950fffeab29db1e48972992aa62" gracePeriod=30
Sep 12 00:47:57.309052 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.309005    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8"]
Sep 12 00:47:57.311086 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.311051    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-51ad044d-sqgr8"]
Sep 12 00:47:57.440237 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.440200    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"
Sep 12 00:47:57.458365 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.458335    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"
Sep 12 00:47:57.527810 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.527776    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s25x\" (UniqueName: \"kubernetes.io/projected/9231c398-67f3-46a0-b8dd-3e6b3288dedf-kube-api-access-6s25x\") pod \"9231c398-67f3-46a0-b8dd-3e6b3288dedf\" (UID: \"9231c398-67f3-46a0-b8dd-3e6b3288dedf\") "
Sep 12 00:47:57.527994 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.527836    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7e34e4-4320-45b0-a743-af876d1fd760-plugins-config-volume\") pod \"0a7e34e4-4320-45b0-a743-af876d1fd760\" (UID: \"0a7e34e4-4320-45b0-a743-af876d1fd760\") "
Sep 12 00:47:57.527994 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.527970    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrddk\" (UniqueName: \"kubernetes.io/projected/0a7e34e4-4320-45b0-a743-af876d1fd760-kube-api-access-jrddk\") pod \"0a7e34e4-4320-45b0-a743-af876d1fd760\" (UID: \"0a7e34e4-4320-45b0-a743-af876d1fd760\") "
Sep 12 00:47:57.528092 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.528053    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9231c398-67f3-46a0-b8dd-3e6b3288dedf-plugins-config-volume\") pod \"9231c398-67f3-46a0-b8dd-3e6b3288dedf\" (UID: \"9231c398-67f3-46a0-b8dd-3e6b3288dedf\") "
Sep 12 00:47:57.528185 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.528167    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7e34e4-4320-45b0-a743-af876d1fd760-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a7e34e4-4320-45b0-a743-af876d1fd760" (UID: "0a7e34e4-4320-45b0-a743-af876d1fd760"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:47:57.528424 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.528391    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9231c398-67f3-46a0-b8dd-3e6b3288dedf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9231c398-67f3-46a0-b8dd-3e6b3288dedf" (UID: "9231c398-67f3-46a0-b8dd-3e6b3288dedf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:47:57.528535 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.528482    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9231c398-67f3-46a0-b8dd-3e6b3288dedf-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:47:57.528535 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.528500    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7e34e4-4320-45b0-a743-af876d1fd760-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:47:57.530043 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.530022    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7e34e4-4320-45b0-a743-af876d1fd760-kube-api-access-jrddk" (OuterVolumeSpecName: "kube-api-access-jrddk") pod "0a7e34e4-4320-45b0-a743-af876d1fd760" (UID: "0a7e34e4-4320-45b0-a743-af876d1fd760"). InnerVolumeSpecName "kube-api-access-jrddk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:47:57.530099 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.530061    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9231c398-67f3-46a0-b8dd-3e6b3288dedf-kube-api-access-6s25x" (OuterVolumeSpecName: "kube-api-access-6s25x") pod "9231c398-67f3-46a0-b8dd-3e6b3288dedf" (UID: "9231c398-67f3-46a0-b8dd-3e6b3288dedf"). InnerVolumeSpecName "kube-api-access-6s25x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:47:57.629758 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.629717    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s25x\" (UniqueName: \"kubernetes.io/projected/9231c398-67f3-46a0-b8dd-3e6b3288dedf-kube-api-access-6s25x\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:47:57.629758 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.629751    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrddk\" (UniqueName: \"kubernetes.io/projected/0a7e34e4-4320-45b0-a743-af876d1fd760-kube-api-access-jrddk\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:47:57.916099 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.916063    2567 generic.go:356] "Generic (PLEG): container finished" podID="9231c398-67f3-46a0-b8dd-3e6b3288dedf" containerID="ac6b05cc8da23f113d023b4bee93011591b9943a82144d07fe6193011397fa2e" exitCode=0
Sep 12 00:47:57.916278 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.916108    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"
Sep 12 00:47:57.916278 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.916154    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8" event={"ID":"9231c398-67f3-46a0-b8dd-3e6b3288dedf","Type":"ContainerDied","Data":"ac6b05cc8da23f113d023b4bee93011591b9943a82144d07fe6193011397fa2e"}
Sep 12 00:47:57.916278 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.916212    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8" event={"ID":"9231c398-67f3-46a0-b8dd-3e6b3288dedf","Type":"ContainerDied","Data":"f7a35fd9d5cf72fcdbfed0ca5e4c73b9f48082f624256220ae8c8103861723fd"}
Sep 12 00:47:57.916278 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.916243    2567 scope.go:117] "RemoveContainer" containerID="ac6b05cc8da23f113d023b4bee93011591b9943a82144d07fe6193011397fa2e"
Sep 12 00:47:57.917668 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.917642    2567 generic.go:356] "Generic (PLEG): container finished" podID="0a7e34e4-4320-45b0-a743-af876d1fd760" containerID="ca506526a3181d2fcdd52f745a159a35e7b3a950fffeab29db1e48972992aa62" exitCode=0
Sep 12 00:47:57.917760 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.917682    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w" event={"ID":"0a7e34e4-4320-45b0-a743-af876d1fd760","Type":"ContainerDied","Data":"ca506526a3181d2fcdd52f745a159a35e7b3a950fffeab29db1e48972992aa62"}
Sep 12 00:47:57.917760 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.917746    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w" event={"ID":"0a7e34e4-4320-45b0-a743-af876d1fd760","Type":"ContainerDied","Data":"95805182406fd42b6bfa9745ffb2c52cb1c9676c9c3c09f54386be4415756cda"}
Sep 12 00:47:57.917860 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.917689    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"
Sep 12 00:47:57.925920 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.925903    2567 scope.go:117] "RemoveContainer" containerID="ac6b05cc8da23f113d023b4bee93011591b9943a82144d07fe6193011397fa2e"
Sep 12 00:47:57.926223 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:47:57.926203    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6b05cc8da23f113d023b4bee93011591b9943a82144d07fe6193011397fa2e\": container with ID starting with ac6b05cc8da23f113d023b4bee93011591b9943a82144d07fe6193011397fa2e not found: ID does not exist" containerID="ac6b05cc8da23f113d023b4bee93011591b9943a82144d07fe6193011397fa2e"
Sep 12 00:47:57.926268 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.926234    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6b05cc8da23f113d023b4bee93011591b9943a82144d07fe6193011397fa2e"} err="failed to get container status \"ac6b05cc8da23f113d023b4bee93011591b9943a82144d07fe6193011397fa2e\": rpc error: code = NotFound desc = could not find container \"ac6b05cc8da23f113d023b4bee93011591b9943a82144d07fe6193011397fa2e\": container with ID starting with ac6b05cc8da23f113d023b4bee93011591b9943a82144d07fe6193011397fa2e not found: ID does not exist"
Sep 12 00:47:57.926268 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.926253    2567 scope.go:117] "RemoveContainer" containerID="ca506526a3181d2fcdd52f745a159a35e7b3a950fffeab29db1e48972992aa62"
Sep 12 00:47:57.934447 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.934431    2567 scope.go:117] "RemoveContainer" containerID="ca506526a3181d2fcdd52f745a159a35e7b3a950fffeab29db1e48972992aa62"
Sep 12 00:47:57.934713 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:47:57.934696    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca506526a3181d2fcdd52f745a159a35e7b3a950fffeab29db1e48972992aa62\": container with ID starting with ca506526a3181d2fcdd52f745a159a35e7b3a950fffeab29db1e48972992aa62 not found: ID does not exist" containerID="ca506526a3181d2fcdd52f745a159a35e7b3a950fffeab29db1e48972992aa62"
Sep 12 00:47:57.934760 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.934719    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca506526a3181d2fcdd52f745a159a35e7b3a950fffeab29db1e48972992aa62"} err="failed to get container status \"ca506526a3181d2fcdd52f745a159a35e7b3a950fffeab29db1e48972992aa62\": rpc error: code = NotFound desc = could not find container \"ca506526a3181d2fcdd52f745a159a35e7b3a950fffeab29db1e48972992aa62\": container with ID starting with ca506526a3181d2fcdd52f745a159a35e7b3a950fffeab29db1e48972992aa62 not found: ID does not exist"
Sep 12 00:47:57.941737 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.941706    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"]
Sep 12 00:47:57.945498 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.945463    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-51ad044d-84d6d7b6b-rz57w"]
Sep 12 00:47:57.956035 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.956005    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"]
Sep 12 00:47:57.959474 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:57.959450    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-51ad044d-6d7dbdd8f7-wvjh8"]
Sep 12 00:47:58.273827 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:58.273785    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a7e34e4-4320-45b0-a743-af876d1fd760" path="/var/lib/kubelet/pods/0a7e34e4-4320-45b0-a743-af876d1fd760/volumes"
Sep 12 00:47:58.274337 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:58.274274    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358ff3a3-91b3-4e82-abc5-d838e57e4885" path="/var/lib/kubelet/pods/358ff3a3-91b3-4e82-abc5-d838e57e4885/volumes"
Sep 12 00:47:58.274706 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:47:58.274685    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9231c398-67f3-46a0-b8dd-3e6b3288dedf" path="/var/lib/kubelet/pods/9231c398-67f3-46a0-b8dd-3e6b3288dedf/volumes"
Sep 12 00:48:06.828953 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.828914    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"]
Sep 12 00:48:06.829411 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.829265    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9231c398-67f3-46a0-b8dd-3e6b3288dedf" containerName="payload-processing"
Sep 12 00:48:06.829411 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.829291    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9231c398-67f3-46a0-b8dd-3e6b3288dedf" containerName="payload-processing"
Sep 12 00:48:06.829411 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.829314    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358ff3a3-91b3-4e82-abc5-d838e57e4885" containerName="revoke-keys"
Sep 12 00:48:06.829411 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.829319    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="358ff3a3-91b3-4e82-abc5-d838e57e4885" containerName="revoke-keys"
Sep 12 00:48:06.829411 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.829328    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7e34e4-4320-45b0-a743-af876d1fd760" containerName="payload-pre-processing"
Sep 12 00:48:06.829411 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.829334    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7e34e4-4320-45b0-a743-af876d1fd760" containerName="payload-pre-processing"
Sep 12 00:48:06.829411 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.829390    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7e34e4-4320-45b0-a743-af876d1fd760" containerName="payload-pre-processing"
Sep 12 00:48:06.829411 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.829399    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="358ff3a3-91b3-4e82-abc5-d838e57e4885" containerName="revoke-keys"
Sep 12 00:48:06.829411 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.829406    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9231c398-67f3-46a0-b8dd-3e6b3288dedf" containerName="payload-processing"
Sep 12 00:48:06.832108 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.832088    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"
Sep 12 00:48:06.834656 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.834631    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f2beda59\""
Sep 12 00:48:06.843645 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.843610    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"]
Sep 12 00:48:06.884690 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.884649    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"]
Sep 12 00:48:06.887026 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.886994    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"
Sep 12 00:48:06.889364 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.889343    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f2beda59-dockercfg-g7d8q\""
Sep 12 00:48:06.897088 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.897066    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"]
Sep 12 00:48:06.921498 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.921467    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsbfx\" (UniqueName: \"kubernetes.io/projected/e0eca9f4-2a52-433e-82ab-443b58d36843-kube-api-access-bsbfx\") pod \"payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw\" (UID: \"e0eca9f4-2a52-433e-82ab-443b58d36843\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"
Sep 12 00:48:06.921673 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:06.921555    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0eca9f4-2a52-433e-82ab-443b58d36843-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw\" (UID: \"e0eca9f4-2a52-433e-82ab-443b58d36843\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"
Sep 12 00:48:07.022797 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.022747    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc05e63-e312-4300-9ca4-788cd32fcd91-plugins-config-volume\") pod \"payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc\" (UID: \"6bc05e63-e312-4300-9ca4-788cd32fcd91\") " pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"
Sep 12 00:48:07.022797 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.022797    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vkj\" (UniqueName: \"kubernetes.io/projected/6bc05e63-e312-4300-9ca4-788cd32fcd91-kube-api-access-k7vkj\") pod \"payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc\" (UID: \"6bc05e63-e312-4300-9ca4-788cd32fcd91\") " pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"
Sep 12 00:48:07.023074 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.022863    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsbfx\" (UniqueName: \"kubernetes.io/projected/e0eca9f4-2a52-433e-82ab-443b58d36843-kube-api-access-bsbfx\") pod \"payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw\" (UID: \"e0eca9f4-2a52-433e-82ab-443b58d36843\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"
Sep 12 00:48:07.023074 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.022939    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0eca9f4-2a52-433e-82ab-443b58d36843-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw\" (UID: \"e0eca9f4-2a52-433e-82ab-443b58d36843\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"
Sep 12 00:48:07.023633 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.023610    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0eca9f4-2a52-433e-82ab-443b58d36843-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw\" (UID: \"e0eca9f4-2a52-433e-82ab-443b58d36843\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"
Sep 12 00:48:07.031290 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.031256    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsbfx\" (UniqueName: \"kubernetes.io/projected/e0eca9f4-2a52-433e-82ab-443b58d36843-kube-api-access-bsbfx\") pod \"payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw\" (UID: \"e0eca9f4-2a52-433e-82ab-443b58d36843\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"
Sep 12 00:48:07.124407 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.124313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc05e63-e312-4300-9ca4-788cd32fcd91-plugins-config-volume\") pod \"payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc\" (UID: \"6bc05e63-e312-4300-9ca4-788cd32fcd91\") " pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"
Sep 12 00:48:07.124407 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.124365    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vkj\" (UniqueName: \"kubernetes.io/projected/6bc05e63-e312-4300-9ca4-788cd32fcd91-kube-api-access-k7vkj\") pod \"payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc\" (UID: \"6bc05e63-e312-4300-9ca4-788cd32fcd91\") " pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"
Sep 12 00:48:07.125051 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.125030    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc05e63-e312-4300-9ca4-788cd32fcd91-plugins-config-volume\") pod \"payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc\" (UID: \"6bc05e63-e312-4300-9ca4-788cd32fcd91\") " pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"
Sep 12 00:48:07.132165 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.132138    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vkj\" (UniqueName: \"kubernetes.io/projected/6bc05e63-e312-4300-9ca4-788cd32fcd91-kube-api-access-k7vkj\") pod \"payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc\" (UID: \"6bc05e63-e312-4300-9ca4-788cd32fcd91\") " pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"
Sep 12 00:48:07.143295 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.143268    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"
Sep 12 00:48:07.197873 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.197519    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"
Sep 12 00:48:07.413699 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.413668    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"]
Sep 12 00:48:07.415668 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:48:07.415635    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0eca9f4_2a52_433e_82ab_443b58d36843.slice/crio-532ab6a92b2db849b04ca93c1e29eff21d26a201ada2d7ed39e958f9d8aca924 WatchSource:0}: Error finding container 532ab6a92b2db849b04ca93c1e29eff21d26a201ada2d7ed39e958f9d8aca924: Status 404 returned error can't find the container with id 532ab6a92b2db849b04ca93c1e29eff21d26a201ada2d7ed39e958f9d8aca924
Sep 12 00:48:07.478627 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.478594    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"]
Sep 12 00:48:07.482582 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:48:07.482538    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc05e63_e312_4300_9ca4_788cd32fcd91.slice/crio-c6e9c97d62bffa1aeb07728a58535996e3dc401f170c06a2375369bc115cdf1d WatchSource:0}: Error finding container c6e9c97d62bffa1aeb07728a58535996e3dc401f170c06a2375369bc115cdf1d: Status 404 returned error can't find the container with id c6e9c97d62bffa1aeb07728a58535996e3dc401f170c06a2375369bc115cdf1d
Sep 12 00:48:07.972527 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.972484    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw" event={"ID":"e0eca9f4-2a52-433e-82ab-443b58d36843","Type":"ContainerStarted","Data":"7ea7b90fcafa4d5ca2d466a690369bfb49ceef2d6a3a7d2726c4be8e86475419"}
Sep 12 00:48:07.973043 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.972547    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw" event={"ID":"e0eca9f4-2a52-433e-82ab-443b58d36843","Type":"ContainerStarted","Data":"532ab6a92b2db849b04ca93c1e29eff21d26a201ada2d7ed39e958f9d8aca924"}
Sep 12 00:48:07.973043 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.972599    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"
Sep 12 00:48:07.974326 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.974295    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc" event={"ID":"6bc05e63-e312-4300-9ca4-788cd32fcd91","Type":"ContainerStarted","Data":"eb48b5509cfcdf7d29d4898a141a8169482fa71d76d3ea50dcc4078182ba260f"}
Sep 12 00:48:07.974464 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.974343    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc" event={"ID":"6bc05e63-e312-4300-9ca4-788cd32fcd91","Type":"ContainerStarted","Data":"c6e9c97d62bffa1aeb07728a58535996e3dc401f170c06a2375369bc115cdf1d"}
Sep 12 00:48:07.974464 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.974404    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"
Sep 12 00:48:07.988280 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:07.988225    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw" podStartSLOduration=1.988209179 podStartE2EDuration="1.988209179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:48:07.987252364 +0000 UTC m=+1120.191253187" watchObservedRunningTime="2026-09-12 00:48:07.988209179 +0000 UTC m=+1120.192210000"
Sep 12 00:48:08.003770 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:08.003720    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc" podStartSLOduration=2.003702966 podStartE2EDuration="2.003702966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:48:08.002152833 +0000 UTC m=+1120.206153657" watchObservedRunningTime="2026-09-12 00:48:08.003702966 +0000 UTC m=+1120.207703787"
Sep 12 00:48:18.981341 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:18.981306    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"
Sep 12 00:48:18.981799 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:18.981380    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"
Sep 12 00:48:23.458818 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:23.458781    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft"]
Sep 12 00:48:23.461430 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:23.461412    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft"
Sep 12 00:48:23.463879 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:23.463857    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:48:23.463879 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:23.463868    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:48:23.469139 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:23.469105    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft"]
Sep 12 00:48:23.590213 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:23.590171    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/867a74e2-3068-456e-abce-465654d9ce91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft\" (UID: \"867a74e2-3068-456e-abce-465654d9ce91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft"
Sep 12 00:48:23.691440 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:23.691404    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/867a74e2-3068-456e-abce-465654d9ce91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft\" (UID: \"867a74e2-3068-456e-abce-465654d9ce91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft"
Sep 12 00:48:23.692039 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:23.692017    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/867a74e2-3068-456e-abce-465654d9ce91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft\" (UID: \"867a74e2-3068-456e-abce-465654d9ce91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft"
Sep 12 00:48:23.774285 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:23.774246    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft"
Sep 12 00:48:24.040932 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:24.040905    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft"]
Sep 12 00:48:24.043433 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:48:24.043406    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867a74e2_3068_456e_abce_465654d9ce91.slice/crio-2be3ff2bcb1e889dc360de7e76882594b948063885acb3f3e6f1cae4c8b01d90 WatchSource:0}: Error finding container 2be3ff2bcb1e889dc360de7e76882594b948063885acb3f3e6f1cae4c8b01d90: Status 404 returned error can't find the container with id 2be3ff2bcb1e889dc360de7e76882594b948063885acb3f3e6f1cae4c8b01d90
Sep 12 00:48:24.063198 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:24.063173    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft" event={"ID":"867a74e2-3068-456e-abce-465654d9ce91","Type":"ContainerStarted","Data":"2be3ff2bcb1e889dc360de7e76882594b948063885acb3f3e6f1cae4c8b01d90"}
Sep 12 00:48:25.069923 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:25.069835    2567 generic.go:356] "Generic (PLEG): container finished" podID="867a74e2-3068-456e-abce-465654d9ce91" containerID="1654f145feb6e5f0ab5da619b8fec3762960af12eea7f14c4050c12d8351045d" exitCode=0
Sep 12 00:48:25.070284 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:25.069924    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft" event={"ID":"867a74e2-3068-456e-abce-465654d9ce91","Type":"ContainerDied","Data":"1654f145feb6e5f0ab5da619b8fec3762960af12eea7f14c4050c12d8351045d"}
Sep 12 00:48:26.204785 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.204757    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft"
Sep 12 00:48:26.213078 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.213052    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/867a74e2-3068-456e-abce-465654d9ce91-maas-api-service-ca\") pod \"867a74e2-3068-456e-abce-465654d9ce91\" (UID: \"867a74e2-3068-456e-abce-465654d9ce91\") "
Sep 12 00:48:26.213393 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.213370    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867a74e2-3068-456e-abce-465654d9ce91-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "867a74e2-3068-456e-abce-465654d9ce91" (UID: "867a74e2-3068-456e-abce-465654d9ce91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:48:26.276646 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.276610    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz"]
Sep 12 00:48:26.276988 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.276972    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="867a74e2-3068-456e-abce-465654d9ce91" containerName="revoke-keys"
Sep 12 00:48:26.276988 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.276987    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="867a74e2-3068-456e-abce-465654d9ce91" containerName="revoke-keys"
Sep 12 00:48:26.277131 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.277047    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="867a74e2-3068-456e-abce-465654d9ce91" containerName="revoke-keys"
Sep 12 00:48:26.279962 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.279937    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz"
Sep 12 00:48:26.286826 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.286802    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz"]
Sep 12 00:48:26.313843 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.313804    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89737ab-da27-4f66-befe-aa9e88944cb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz\" (UID: \"e89737ab-da27-4f66-befe-aa9e88944cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz"
Sep 12 00:48:26.314024 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.313973    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/867a74e2-3068-456e-abce-465654d9ce91-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:48:26.415049 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.414947    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89737ab-da27-4f66-befe-aa9e88944cb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz\" (UID: \"e89737ab-da27-4f66-befe-aa9e88944cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz"
Sep 12 00:48:26.415600 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.415551    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89737ab-da27-4f66-befe-aa9e88944cb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz\" (UID: \"e89737ab-da27-4f66-befe-aa9e88944cb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz"
Sep 12 00:48:26.596094 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.596045    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz"
Sep 12 00:48:26.866887 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:26.866859    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz"]
Sep 12 00:48:26.868954 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:48:26.868926    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89737ab_da27_4f66_befe_aa9e88944cb3.slice/crio-8a2eff46e3eac0669dd03045670a7f1286f1744dd6bf4470431cd69b6eaa99c9 WatchSource:0}: Error finding container 8a2eff46e3eac0669dd03045670a7f1286f1744dd6bf4470431cd69b6eaa99c9: Status 404 returned error can't find the container with id 8a2eff46e3eac0669dd03045670a7f1286f1744dd6bf4470431cd69b6eaa99c9
Sep 12 00:48:27.082293 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:27.082264    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft"
Sep 12 00:48:27.082453 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:27.082311    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft" event={"ID":"867a74e2-3068-456e-abce-465654d9ce91","Type":"ContainerDied","Data":"2be3ff2bcb1e889dc360de7e76882594b948063885acb3f3e6f1cae4c8b01d90"}
Sep 12 00:48:27.082453 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:27.082403    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2be3ff2bcb1e889dc360de7e76882594b948063885acb3f3e6f1cae4c8b01d90"
Sep 12 00:48:27.084002 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:27.083973    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz" event={"ID":"e89737ab-da27-4f66-befe-aa9e88944cb3","Type":"ContainerStarted","Data":"229ffa44dd46261c5498b7fe061c0556272f951b7f6b07924509de44cc126d96"}
Sep 12 00:48:27.084135 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:27.084017    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz" event={"ID":"e89737ab-da27-4f66-befe-aa9e88944cb3","Type":"ContainerStarted","Data":"8a2eff46e3eac0669dd03045670a7f1286f1744dd6bf4470431cd69b6eaa99c9"}
Sep 12 00:48:27.098388 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:27.098347    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz" podStartSLOduration=1.098335012 podStartE2EDuration="1.098335012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:48:27.097504183 +0000 UTC m=+1139.301505007" watchObservedRunningTime="2026-09-12 00:48:27.098335012 +0000 UTC m=+1139.302335879"
Sep 12 00:48:28.090683 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:28.090643    2567 generic.go:356] "Generic (PLEG): container finished" podID="e89737ab-da27-4f66-befe-aa9e88944cb3" containerID="229ffa44dd46261c5498b7fe061c0556272f951b7f6b07924509de44cc126d96" exitCode=0
Sep 12 00:48:28.091115 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:28.090738    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz" event={"ID":"e89737ab-da27-4f66-befe-aa9e88944cb3","Type":"ContainerDied","Data":"229ffa44dd46261c5498b7fe061c0556272f951b7f6b07924509de44cc126d96"}
Sep 12 00:48:29.235167 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:29.235134    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz"
Sep 12 00:48:29.343877 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:29.343841    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89737ab-da27-4f66-befe-aa9e88944cb3-maas-api-service-ca\") pod \"e89737ab-da27-4f66-befe-aa9e88944cb3\" (UID: \"e89737ab-da27-4f66-befe-aa9e88944cb3\") "
Sep 12 00:48:29.344223 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:29.344199    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89737ab-da27-4f66-befe-aa9e88944cb3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e89737ab-da27-4f66-befe-aa9e88944cb3" (UID: "e89737ab-da27-4f66-befe-aa9e88944cb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:48:29.445312 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:29.445219    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89737ab-da27-4f66-befe-aa9e88944cb3-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:48:30.109249 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:30.109218    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz"
Sep 12 00:48:30.109400 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:30.109245    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz" event={"ID":"e89737ab-da27-4f66-befe-aa9e88944cb3","Type":"ContainerDied","Data":"8a2eff46e3eac0669dd03045670a7f1286f1744dd6bf4470431cd69b6eaa99c9"}
Sep 12 00:48:30.109400 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:30.109323    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a2eff46e3eac0669dd03045670a7f1286f1744dd6bf4470431cd69b6eaa99c9"
Sep 12 00:48:33.624810 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:33.624774    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"]
Sep 12 00:48:33.625270 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:33.625005    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds" podUID="2d3b89ec-2c36-4409-840a-ebd1e85ebee8" containerName="payload-processing" containerID="cri-o://b8f69719c251bc7df85d36d9b6a32f0e77b1c07af9611c7c6e8c0f059df227b6" gracePeriod=30
Sep 12 00:48:33.768887 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:33.768846    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"]
Sep 12 00:48:33.769107 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:33.769071    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc" podUID="6bc05e63-e312-4300-9ca4-788cd32fcd91" containerName="payload-processing" containerID="cri-o://eb48b5509cfcdf7d29d4898a141a8169482fa71d76d3ea50dcc4078182ba260f" gracePeriod=30
Sep 12 00:48:33.779911 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:33.779879    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"]
Sep 12 00:48:33.780514 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:33.780150    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw" podUID="e0eca9f4-2a52-433e-82ab-443b58d36843" containerName="payload-pre-processing" containerID="cri-o://7ea7b90fcafa4d5ca2d466a690369bfb49ceef2d6a3a7d2726c4be8e86475419" gracePeriod=30
Sep 12 00:48:33.803958 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:33.803920    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft"]
Sep 12 00:48:33.806085 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:33.806061    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4b6ed7-kh4ft"]
Sep 12 00:48:33.915773 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:33.915738    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"
Sep 12 00:48:33.991240 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:33.991208    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b89ec-2c36-4409-840a-ebd1e85ebee8-plugins-config-volume\") pod \"2d3b89ec-2c36-4409-840a-ebd1e85ebee8\" (UID: \"2d3b89ec-2c36-4409-840a-ebd1e85ebee8\") "
Sep 12 00:48:33.991401 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:33.991280    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5h7nt\" (UniqueName: \"kubernetes.io/projected/2d3b89ec-2c36-4409-840a-ebd1e85ebee8-kube-api-access-5h7nt\") pod \"2d3b89ec-2c36-4409-840a-ebd1e85ebee8\" (UID: \"2d3b89ec-2c36-4409-840a-ebd1e85ebee8\") "
Sep 12 00:48:33.991707 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:33.991675    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3b89ec-2c36-4409-840a-ebd1e85ebee8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d3b89ec-2c36-4409-840a-ebd1e85ebee8" (UID: "2d3b89ec-2c36-4409-840a-ebd1e85ebee8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:48:33.993602 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:33.993553    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3b89ec-2c36-4409-840a-ebd1e85ebee8-kube-api-access-5h7nt" (OuterVolumeSpecName: "kube-api-access-5h7nt") pod "2d3b89ec-2c36-4409-840a-ebd1e85ebee8" (UID: "2d3b89ec-2c36-4409-840a-ebd1e85ebee8"). InnerVolumeSpecName "kube-api-access-5h7nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:48:34.070295 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.070262    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"
Sep 12 00:48:34.074212 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.074182    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"
Sep 12 00:48:34.092363 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.092179    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc05e63-e312-4300-9ca4-788cd32fcd91-plugins-config-volume\") pod \"6bc05e63-e312-4300-9ca4-788cd32fcd91\" (UID: \"6bc05e63-e312-4300-9ca4-788cd32fcd91\") "
Sep 12 00:48:34.092363 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.092293    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7vkj\" (UniqueName: \"kubernetes.io/projected/6bc05e63-e312-4300-9ca4-788cd32fcd91-kube-api-access-k7vkj\") pod \"6bc05e63-e312-4300-9ca4-788cd32fcd91\" (UID: \"6bc05e63-e312-4300-9ca4-788cd32fcd91\") "
Sep 12 00:48:34.092793 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.092766    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b89ec-2c36-4409-840a-ebd1e85ebee8-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:48:34.092994 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.092796    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h7nt\" (UniqueName: \"kubernetes.io/projected/2d3b89ec-2c36-4409-840a-ebd1e85ebee8-kube-api-access-5h7nt\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:48:34.092994 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.092951    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc05e63-e312-4300-9ca4-788cd32fcd91-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6bc05e63-e312-4300-9ca4-788cd32fcd91" (UID: "6bc05e63-e312-4300-9ca4-788cd32fcd91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:48:34.095096 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.095071    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc05e63-e312-4300-9ca4-788cd32fcd91-kube-api-access-k7vkj" (OuterVolumeSpecName: "kube-api-access-k7vkj") pod "6bc05e63-e312-4300-9ca4-788cd32fcd91" (UID: "6bc05e63-e312-4300-9ca4-788cd32fcd91"). InnerVolumeSpecName "kube-api-access-k7vkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:48:34.131008 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.130963    2567 generic.go:356] "Generic (PLEG): container finished" podID="e0eca9f4-2a52-433e-82ab-443b58d36843" containerID="7ea7b90fcafa4d5ca2d466a690369bfb49ceef2d6a3a7d2726c4be8e86475419" exitCode=0
Sep 12 00:48:34.131309 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.131028    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"
Sep 12 00:48:34.131309 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.131079    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw" event={"ID":"e0eca9f4-2a52-433e-82ab-443b58d36843","Type":"ContainerDied","Data":"7ea7b90fcafa4d5ca2d466a690369bfb49ceef2d6a3a7d2726c4be8e86475419"}
Sep 12 00:48:34.131309 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.131182    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw" event={"ID":"e0eca9f4-2a52-433e-82ab-443b58d36843","Type":"ContainerDied","Data":"532ab6a92b2db849b04ca93c1e29eff21d26a201ada2d7ed39e958f9d8aca924"}
Sep 12 00:48:34.131309 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.131213    2567 scope.go:117] "RemoveContainer" containerID="7ea7b90fcafa4d5ca2d466a690369bfb49ceef2d6a3a7d2726c4be8e86475419"
Sep 12 00:48:34.131686 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.131609    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz"]
Sep 12 00:48:34.132794 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.132769    2567 generic.go:356] "Generic (PLEG): container finished" podID="2d3b89ec-2c36-4409-840a-ebd1e85ebee8" containerID="b8f69719c251bc7df85d36d9b6a32f0e77b1c07af9611c7c6e8c0f059df227b6" exitCode=0
Sep 12 00:48:34.132863 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.132813    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"
Sep 12 00:48:34.132921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.132854    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds" event={"ID":"2d3b89ec-2c36-4409-840a-ebd1e85ebee8","Type":"ContainerDied","Data":"b8f69719c251bc7df85d36d9b6a32f0e77b1c07af9611c7c6e8c0f059df227b6"}
Sep 12 00:48:34.132974 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.132929    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds" event={"ID":"2d3b89ec-2c36-4409-840a-ebd1e85ebee8","Type":"ContainerDied","Data":"f821f0d92096191a66eb5bfbfbae0d4417e67572c044aa84881c8cc0d9c720a5"}
Sep 12 00:48:34.134239 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.134216    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f2beda59-dhsxz"]
Sep 12 00:48:34.134408 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.134383    2567 generic.go:356] "Generic (PLEG): container finished" podID="6bc05e63-e312-4300-9ca4-788cd32fcd91" containerID="eb48b5509cfcdf7d29d4898a141a8169482fa71d76d3ea50dcc4078182ba260f" exitCode=0
Sep 12 00:48:34.134483 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.134419    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"
Sep 12 00:48:34.134483 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.134467    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc" event={"ID":"6bc05e63-e312-4300-9ca4-788cd32fcd91","Type":"ContainerDied","Data":"eb48b5509cfcdf7d29d4898a141a8169482fa71d76d3ea50dcc4078182ba260f"}
Sep 12 00:48:34.134635 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.134513    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc" event={"ID":"6bc05e63-e312-4300-9ca4-788cd32fcd91","Type":"ContainerDied","Data":"c6e9c97d62bffa1aeb07728a58535996e3dc401f170c06a2375369bc115cdf1d"}
Sep 12 00:48:34.141732 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.141645    2567 scope.go:117] "RemoveContainer" containerID="7ea7b90fcafa4d5ca2d466a690369bfb49ceef2d6a3a7d2726c4be8e86475419"
Sep 12 00:48:34.142056 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:48:34.142022    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea7b90fcafa4d5ca2d466a690369bfb49ceef2d6a3a7d2726c4be8e86475419\": container with ID starting with 7ea7b90fcafa4d5ca2d466a690369bfb49ceef2d6a3a7d2726c4be8e86475419 not found: ID does not exist" containerID="7ea7b90fcafa4d5ca2d466a690369bfb49ceef2d6a3a7d2726c4be8e86475419"
Sep 12 00:48:34.142245 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.142066    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea7b90fcafa4d5ca2d466a690369bfb49ceef2d6a3a7d2726c4be8e86475419"} err="failed to get container status \"7ea7b90fcafa4d5ca2d466a690369bfb49ceef2d6a3a7d2726c4be8e86475419\": rpc error: code = NotFound desc = could not find container \"7ea7b90fcafa4d5ca2d466a690369bfb49ceef2d6a3a7d2726c4be8e86475419\": container with ID starting with 7ea7b90fcafa4d5ca2d466a690369bfb49ceef2d6a3a7d2726c4be8e86475419 not found: ID does not exist"
Sep 12 00:48:34.142245 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.142084    2567 scope.go:117] "RemoveContainer" containerID="b8f69719c251bc7df85d36d9b6a32f0e77b1c07af9611c7c6e8c0f059df227b6"
Sep 12 00:48:34.150251 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.150232    2567 scope.go:117] "RemoveContainer" containerID="b8f69719c251bc7df85d36d9b6a32f0e77b1c07af9611c7c6e8c0f059df227b6"
Sep 12 00:48:34.150482 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:48:34.150465    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f69719c251bc7df85d36d9b6a32f0e77b1c07af9611c7c6e8c0f059df227b6\": container with ID starting with b8f69719c251bc7df85d36d9b6a32f0e77b1c07af9611c7c6e8c0f059df227b6 not found: ID does not exist" containerID="b8f69719c251bc7df85d36d9b6a32f0e77b1c07af9611c7c6e8c0f059df227b6"
Sep 12 00:48:34.150533 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.150488    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f69719c251bc7df85d36d9b6a32f0e77b1c07af9611c7c6e8c0f059df227b6"} err="failed to get container status \"b8f69719c251bc7df85d36d9b6a32f0e77b1c07af9611c7c6e8c0f059df227b6\": rpc error: code = NotFound desc = could not find container \"b8f69719c251bc7df85d36d9b6a32f0e77b1c07af9611c7c6e8c0f059df227b6\": container with ID starting with b8f69719c251bc7df85d36d9b6a32f0e77b1c07af9611c7c6e8c0f059df227b6 not found: ID does not exist"
Sep 12 00:48:34.150533 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.150502    2567 scope.go:117] "RemoveContainer" containerID="eb48b5509cfcdf7d29d4898a141a8169482fa71d76d3ea50dcc4078182ba260f"
Sep 12 00:48:34.158105 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.158081    2567 scope.go:117] "RemoveContainer" containerID="eb48b5509cfcdf7d29d4898a141a8169482fa71d76d3ea50dcc4078182ba260f"
Sep 12 00:48:34.158962 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:48:34.158939    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb48b5509cfcdf7d29d4898a141a8169482fa71d76d3ea50dcc4078182ba260f\": container with ID starting with eb48b5509cfcdf7d29d4898a141a8169482fa71d76d3ea50dcc4078182ba260f not found: ID does not exist" containerID="eb48b5509cfcdf7d29d4898a141a8169482fa71d76d3ea50dcc4078182ba260f"
Sep 12 00:48:34.159078 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.158967    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb48b5509cfcdf7d29d4898a141a8169482fa71d76d3ea50dcc4078182ba260f"} err="failed to get container status \"eb48b5509cfcdf7d29d4898a141a8169482fa71d76d3ea50dcc4078182ba260f\": rpc error: code = NotFound desc = could not find container \"eb48b5509cfcdf7d29d4898a141a8169482fa71d76d3ea50dcc4078182ba260f\": container with ID starting with eb48b5509cfcdf7d29d4898a141a8169482fa71d76d3ea50dcc4078182ba260f not found: ID does not exist"
Sep 12 00:48:34.160538 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.160519    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"]
Sep 12 00:48:34.163994 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.163972    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f2beda59-68ff76d5b9-77jlc"]
Sep 12 00:48:34.174114 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.174091    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"]
Sep 12 00:48:34.179442 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.179422    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4b6ed7-6dc894869-wh5ds"]
Sep 12 00:48:34.194079 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.194060    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsbfx\" (UniqueName: \"kubernetes.io/projected/e0eca9f4-2a52-433e-82ab-443b58d36843-kube-api-access-bsbfx\") pod \"e0eca9f4-2a52-433e-82ab-443b58d36843\" (UID: \"e0eca9f4-2a52-433e-82ab-443b58d36843\") "
Sep 12 00:48:34.194148 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.194129    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0eca9f4-2a52-433e-82ab-443b58d36843-plugins-config-volume\") pod \"e0eca9f4-2a52-433e-82ab-443b58d36843\" (UID: \"e0eca9f4-2a52-433e-82ab-443b58d36843\") "
Sep 12 00:48:34.194492 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.194473    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0eca9f4-2a52-433e-82ab-443b58d36843-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0eca9f4-2a52-433e-82ab-443b58d36843" (UID: "e0eca9f4-2a52-433e-82ab-443b58d36843"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:48:34.194540 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.194526    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7vkj\" (UniqueName: \"kubernetes.io/projected/6bc05e63-e312-4300-9ca4-788cd32fcd91-kube-api-access-k7vkj\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:48:34.194605 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.194543    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bc05e63-e312-4300-9ca4-788cd32fcd91-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:48:34.196110 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.196091    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0eca9f4-2a52-433e-82ab-443b58d36843-kube-api-access-bsbfx" (OuterVolumeSpecName: "kube-api-access-bsbfx") pod "e0eca9f4-2a52-433e-82ab-443b58d36843" (UID: "e0eca9f4-2a52-433e-82ab-443b58d36843"). InnerVolumeSpecName "kube-api-access-bsbfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:48:34.273473 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.273436    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d3b89ec-2c36-4409-840a-ebd1e85ebee8" path="/var/lib/kubelet/pods/2d3b89ec-2c36-4409-840a-ebd1e85ebee8/volumes"
Sep 12 00:48:34.273861 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.273846    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bc05e63-e312-4300-9ca4-788cd32fcd91" path="/var/lib/kubelet/pods/6bc05e63-e312-4300-9ca4-788cd32fcd91/volumes"
Sep 12 00:48:34.274146 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.274134    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="867a74e2-3068-456e-abce-465654d9ce91" path="/var/lib/kubelet/pods/867a74e2-3068-456e-abce-465654d9ce91/volumes"
Sep 12 00:48:34.274410 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.274400    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89737ab-da27-4f66-befe-aa9e88944cb3" path="/var/lib/kubelet/pods/e89737ab-da27-4f66-befe-aa9e88944cb3/volumes"
Sep 12 00:48:34.295045 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.295019    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0eca9f4-2a52-433e-82ab-443b58d36843-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:48:34.295045 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.295045    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsbfx\" (UniqueName: \"kubernetes.io/projected/e0eca9f4-2a52-433e-82ab-443b58d36843-kube-api-access-bsbfx\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:48:34.446968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.446882    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"]
Sep 12 00:48:34.450865 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:34.450840    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f2beda59-f4849979d-qh4cw"]
Sep 12 00:48:36.273496 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:48:36.273463    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0eca9f4-2a52-433e-82ab-443b58d36843" path="/var/lib/kubelet/pods/e0eca9f4-2a52-433e-82ab-443b58d36843/volumes"
Sep 12 00:49:01.826763 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.826673    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"]
Sep 12 00:49:01.827231 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.827075    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89737ab-da27-4f66-befe-aa9e88944cb3" containerName="revoke-keys"
Sep 12 00:49:01.827231 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.827088    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89737ab-da27-4f66-befe-aa9e88944cb3" containerName="revoke-keys"
Sep 12 00:49:01.827231 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.827098    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0eca9f4-2a52-433e-82ab-443b58d36843" containerName="payload-pre-processing"
Sep 12 00:49:01.827231 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.827104    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0eca9f4-2a52-433e-82ab-443b58d36843" containerName="payload-pre-processing"
Sep 12 00:49:01.827231 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.827113    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d3b89ec-2c36-4409-840a-ebd1e85ebee8" containerName="payload-processing"
Sep 12 00:49:01.827231 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.827119    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d3b89ec-2c36-4409-840a-ebd1e85ebee8" containerName="payload-processing"
Sep 12 00:49:01.827231 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.827127    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc05e63-e312-4300-9ca4-788cd32fcd91" containerName="payload-processing"
Sep 12 00:49:01.827231 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.827132    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc05e63-e312-4300-9ca4-788cd32fcd91" containerName="payload-processing"
Sep 12 00:49:01.827231 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.827186    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d3b89ec-2c36-4409-840a-ebd1e85ebee8" containerName="payload-processing"
Sep 12 00:49:01.827231 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.827193    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89737ab-da27-4f66-befe-aa9e88944cb3" containerName="revoke-keys"
Sep 12 00:49:01.827231 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.827201    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0eca9f4-2a52-433e-82ab-443b58d36843" containerName="payload-pre-processing"
Sep 12 00:49:01.827231 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.827208    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bc05e63-e312-4300-9ca4-788cd32fcd91" containerName="payload-processing"
Sep 12 00:49:01.830081 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.830064    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:01.832985 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.832960    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-876808\"/\"premium-w4-876808-kserve-self-signed-certs\""
Sep 12 00:49:01.839758 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.839732    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"]
Sep 12 00:49:01.964437 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.964398    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-dshm\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:01.964437 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.964444    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-home\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:01.964709 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.964461    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-model-cache\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:01.964709 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.964515    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed476580-cb27-4a37-9ba4-f117f62232b6-tls-certs\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:01.964709 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.964594    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-tmp-dir\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:01.964709 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:01.964679    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m8lq\" (UniqueName: \"kubernetes.io/projected/ed476580-cb27-4a37-9ba4-f117f62232b6-kube-api-access-2m8lq\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.065200 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.065157    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-home\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.065342 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.065208    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-model-cache\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.065342 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.065261    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed476580-cb27-4a37-9ba4-f117f62232b6-tls-certs\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.065447 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.065349    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-tmp-dir\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.065447 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.065400    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m8lq\" (UniqueName: \"kubernetes.io/projected/ed476580-cb27-4a37-9ba4-f117f62232b6-kube-api-access-2m8lq\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.065447 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.065442    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-dshm\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.065695 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.065671    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-home\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.065695 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.065683    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-model-cache\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.065811 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.065761    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-tmp-dir\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.067760 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.067733    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-dshm\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.067964 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.067944    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed476580-cb27-4a37-9ba4-f117f62232b6-tls-certs\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.074907 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.074783    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m8lq\" (UniqueName: \"kubernetes.io/projected/ed476580-cb27-4a37-9ba4-f117f62232b6-kube-api-access-2m8lq\") pod \"premium-w4-876808-kserve-78fddb56d9-l5w69\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") " pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.140091 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.139992    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:02.411815 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:02.411784    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"]
Sep 12 00:49:02.414428 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:49:02.414397    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded476580_cb27_4a37_9ba4_f117f62232b6.slice/crio-6c4468089c1faecad0a395eb4a443edd019168df42eb236e1feb0490c9d7f1e5 WatchSource:0}: Error finding container 6c4468089c1faecad0a395eb4a443edd019168df42eb236e1feb0490c9d7f1e5: Status 404 returned error can't find the container with id 6c4468089c1faecad0a395eb4a443edd019168df42eb236e1feb0490c9d7f1e5
Sep 12 00:49:03.291206 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:03.291164    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69" event={"ID":"ed476580-cb27-4a37-9ba4-f117f62232b6","Type":"ContainerStarted","Data":"de5743abfdb8b741497a3e27c9edc858c504cada77e71cc136809867e7ccad4e"}
Sep 12 00:49:03.291206 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:03.291218    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69" event={"ID":"ed476580-cb27-4a37-9ba4-f117f62232b6","Type":"ContainerStarted","Data":"6c4468089c1faecad0a395eb4a443edd019168df42eb236e1feb0490c9d7f1e5"}
Sep 12 00:49:03.307952 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:03.307894    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69" podStartSLOduration=2.307876329 podStartE2EDuration="2.307876329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:03.306269018 +0000 UTC m=+1175.510269840" watchObservedRunningTime="2026-09-12 00:49:03.307876329 +0000 UTC m=+1175.511877150"
Sep 12 00:49:03.872370 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:03.872317    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"]
Sep 12 00:49:03.872602 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:03.872562    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" containerName="istio-proxy" containerID="cri-o://c0b0fd8c7fc7c83a3cadd94ec239000b35aedd018828d249333f9671dabfc87e" gracePeriod=30
Sep 12 00:49:03.964624 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:03.964589    2567 patch_prober.go:28] interesting pod/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 00:49:03.964793 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:03.964649    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 00:49:09.127556 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.127529    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:49:09.234765 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.234729    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57m6j\" (UniqueName: \"kubernetes.io/projected/872c43fb-40e8-4695-8de4-1d85948b786c-kube-api-access-57m6j\") pod \"872c43fb-40e8-4695-8de4-1d85948b786c\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") "
Sep 12 00:49:09.234964 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.234777    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-workload-socket\") pod \"872c43fb-40e8-4695-8de4-1d85948b786c\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") "
Sep 12 00:49:09.234964 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.234810    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872c43fb-40e8-4695-8de4-1d85948b786c-istio-podinfo\") pod \"872c43fb-40e8-4695-8de4-1d85948b786c\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") "
Sep 12 00:49:09.234964 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.234872    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872c43fb-40e8-4695-8de4-1d85948b786c-istiod-ca-cert\") pod \"872c43fb-40e8-4695-8de4-1d85948b786c\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") "
Sep 12 00:49:09.234964 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.234897    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-workload-certs\") pod \"872c43fb-40e8-4695-8de4-1d85948b786c\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") "
Sep 12 00:49:09.234964 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.234925    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-istio-envoy\") pod \"872c43fb-40e8-4695-8de4-1d85948b786c\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") "
Sep 12 00:49:09.234964 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.234963    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-credential-socket\") pod \"872c43fb-40e8-4695-8de4-1d85948b786c\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") "
Sep 12 00:49:09.235300 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.235093    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872c43fb-40e8-4695-8de4-1d85948b786c-istio-token\") pod \"872c43fb-40e8-4695-8de4-1d85948b786c\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") "
Sep 12 00:49:09.235300 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.235131    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-istio-data\") pod \"872c43fb-40e8-4695-8de4-1d85948b786c\" (UID: \"872c43fb-40e8-4695-8de4-1d85948b786c\") "
Sep 12 00:49:09.235300 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.235154    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "872c43fb-40e8-4695-8de4-1d85948b786c" (UID: "872c43fb-40e8-4695-8de4-1d85948b786c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:09.235456 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.235310    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "872c43fb-40e8-4695-8de4-1d85948b786c" (UID: "872c43fb-40e8-4695-8de4-1d85948b786c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:09.235456 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.235325    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872c43fb-40e8-4695-8de4-1d85948b786c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "872c43fb-40e8-4695-8de4-1d85948b786c" (UID: "872c43fb-40e8-4695-8de4-1d85948b786c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:49:09.235592 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.235508    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "872c43fb-40e8-4695-8de4-1d85948b786c" (UID: "872c43fb-40e8-4695-8de4-1d85948b786c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:09.235592 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.235553    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872c43fb-40e8-4695-8de4-1d85948b786c-istiod-ca-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:49:09.235592 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.235553    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-istio-data" (OuterVolumeSpecName: "istio-data") pod "872c43fb-40e8-4695-8de4-1d85948b786c" (UID: "872c43fb-40e8-4695-8de4-1d85948b786c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:09.235724 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.235604    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-credential-socket\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:49:09.235724 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.235624    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-workload-socket\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:49:09.237242 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.237211    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/872c43fb-40e8-4695-8de4-1d85948b786c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "872c43fb-40e8-4695-8de4-1d85948b786c" (UID: "872c43fb-40e8-4695-8de4-1d85948b786c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 00:49:09.237356 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.237264    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "872c43fb-40e8-4695-8de4-1d85948b786c" (UID: "872c43fb-40e8-4695-8de4-1d85948b786c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:09.237486 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.237469    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872c43fb-40e8-4695-8de4-1d85948b786c-kube-api-access-57m6j" (OuterVolumeSpecName: "kube-api-access-57m6j") pod "872c43fb-40e8-4695-8de4-1d85948b786c" (UID: "872c43fb-40e8-4695-8de4-1d85948b786c"). InnerVolumeSpecName "kube-api-access-57m6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:49:09.237527 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.237494    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872c43fb-40e8-4695-8de4-1d85948b786c-istio-token" (OuterVolumeSpecName: "istio-token") pod "872c43fb-40e8-4695-8de4-1d85948b786c" (UID: "872c43fb-40e8-4695-8de4-1d85948b786c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:49:09.322417 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.322377    2567 generic.go:356] "Generic (PLEG): container finished" podID="872c43fb-40e8-4695-8de4-1d85948b786c" containerID="c0b0fd8c7fc7c83a3cadd94ec239000b35aedd018828d249333f9671dabfc87e" exitCode=0
Sep 12 00:49:09.322647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.322435    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"
Sep 12 00:49:09.322647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.322468    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" event={"ID":"872c43fb-40e8-4695-8de4-1d85948b786c","Type":"ContainerDied","Data":"c0b0fd8c7fc7c83a3cadd94ec239000b35aedd018828d249333f9671dabfc87e"}
Sep 12 00:49:09.322647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.322537    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w" event={"ID":"872c43fb-40e8-4695-8de4-1d85948b786c","Type":"ContainerDied","Data":"98d6a400b33d6a97fd1cb11a0e4a8a89430fdb9633f959a689f0aa110174e6b2"}
Sep 12 00:49:09.322647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.322593    2567 scope.go:117] "RemoveContainer" containerID="c0b0fd8c7fc7c83a3cadd94ec239000b35aedd018828d249333f9671dabfc87e"
Sep 12 00:49:09.332100 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.332075    2567 scope.go:117] "RemoveContainer" containerID="e0aed1d07042242026bf4fcecc3a11e4c8c96bc01928d1038203c2bc56cdd055"
Sep 12 00:49:09.337026 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.337002    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-workload-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:49:09.337026 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.337024    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-istio-envoy\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:49:09.337151 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.337034    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872c43fb-40e8-4695-8de4-1d85948b786c-istio-token\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:49:09.337151 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.337043    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872c43fb-40e8-4695-8de4-1d85948b786c-istio-data\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:49:09.337151 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.337050    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57m6j\" (UniqueName: \"kubernetes.io/projected/872c43fb-40e8-4695-8de4-1d85948b786c-kube-api-access-57m6j\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:49:09.337151 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.337059    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872c43fb-40e8-4695-8de4-1d85948b786c-istio-podinfo\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:49:09.341109 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.341087    2567 scope.go:117] "RemoveContainer" containerID="c0b0fd8c7fc7c83a3cadd94ec239000b35aedd018828d249333f9671dabfc87e"
Sep 12 00:49:09.341633 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:49:09.341534    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b0fd8c7fc7c83a3cadd94ec239000b35aedd018828d249333f9671dabfc87e\": container with ID starting with c0b0fd8c7fc7c83a3cadd94ec239000b35aedd018828d249333f9671dabfc87e not found: ID does not exist" containerID="c0b0fd8c7fc7c83a3cadd94ec239000b35aedd018828d249333f9671dabfc87e"
Sep 12 00:49:09.341633 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.341590    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b0fd8c7fc7c83a3cadd94ec239000b35aedd018828d249333f9671dabfc87e"} err="failed to get container status \"c0b0fd8c7fc7c83a3cadd94ec239000b35aedd018828d249333f9671dabfc87e\": rpc error: code = NotFound desc = could not find container \"c0b0fd8c7fc7c83a3cadd94ec239000b35aedd018828d249333f9671dabfc87e\": container with ID starting with c0b0fd8c7fc7c83a3cadd94ec239000b35aedd018828d249333f9671dabfc87e not found: ID does not exist"
Sep 12 00:49:09.341633 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.341618    2567 scope.go:117] "RemoveContainer" containerID="e0aed1d07042242026bf4fcecc3a11e4c8c96bc01928d1038203c2bc56cdd055"
Sep 12 00:49:09.342083 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:49:09.342024    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0aed1d07042242026bf4fcecc3a11e4c8c96bc01928d1038203c2bc56cdd055\": container with ID starting with e0aed1d07042242026bf4fcecc3a11e4c8c96bc01928d1038203c2bc56cdd055 not found: ID does not exist" containerID="e0aed1d07042242026bf4fcecc3a11e4c8c96bc01928d1038203c2bc56cdd055"
Sep 12 00:49:09.342083 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.342059    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0aed1d07042242026bf4fcecc3a11e4c8c96bc01928d1038203c2bc56cdd055"} err="failed to get container status \"e0aed1d07042242026bf4fcecc3a11e4c8c96bc01928d1038203c2bc56cdd055\": rpc error: code = NotFound desc = could not find container \"e0aed1d07042242026bf4fcecc3a11e4c8c96bc01928d1038203c2bc56cdd055\": container with ID starting with e0aed1d07042242026bf4fcecc3a11e4c8c96bc01928d1038203c2bc56cdd055 not found: ID does not exist"
Sep 12 00:49:09.343940 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.343921    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"]
Sep 12 00:49:09.349889 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:09.349866    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-4b6ed7-openshift-default-77659899c8-jqn8w"]
Sep 12 00:49:10.274170 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:10.274124    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" path="/var/lib/kubelet/pods/872c43fb-40e8-4695-8de4-1d85948b786c/volumes"
Sep 12 00:49:11.941227 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:11.941187    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"]
Sep 12 00:49:11.941649 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:11.941542    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" containerName="istio-proxy"
Sep 12 00:49:11.941649 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:11.941554    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" containerName="istio-proxy"
Sep 12 00:49:11.941649 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:11.941658    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" containerName="istio-proxy"
Sep 12 00:49:11.941792 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:11.941668    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" containerName="istio-proxy"
Sep 12 00:49:11.946054 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:11.946034    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"
Sep 12 00:49:11.950412 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:11.950384    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c3f8d8-dockercfg-zr2g7\""
Sep 12 00:49:11.950722 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:11.950706    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c3f8d8\""
Sep 12 00:49:11.966361 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:11.966336    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"]
Sep 12 00:49:11.995093 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:11.995059    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"]
Sep 12 00:49:11.995424 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:11.995411    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" containerName="istio-proxy"
Sep 12 00:49:11.995472 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:11.995426    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="872c43fb-40e8-4695-8de4-1d85948b786c" containerName="istio-proxy"
Sep 12 00:49:11.998402 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:11.998385    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"
Sep 12 00:49:12.060028 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.059995    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"]
Sep 12 00:49:12.062563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.062524    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67\" (UID: \"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4\") " pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"
Sep 12 00:49:12.062827 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.062592    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdsh\" (UniqueName: \"kubernetes.io/projected/64261abc-72f8-49f0-9109-da740a9122d1-kube-api-access-zhdsh\") pod \"payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf\" (UID: \"64261abc-72f8-49f0-9109-da740a9122d1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"
Sep 12 00:49:12.062827 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.062720    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64261abc-72f8-49f0-9109-da740a9122d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf\" (UID: \"64261abc-72f8-49f0-9109-da740a9122d1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"
Sep 12 00:49:12.062827 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.062791    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gccf\" (UniqueName: \"kubernetes.io/projected/d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4-kube-api-access-9gccf\") pod \"payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67\" (UID: \"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4\") " pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"
Sep 12 00:49:12.140870 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.140835    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:12.141015 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.140888    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:12.152981 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.152954    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:12.163210 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.163182    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67\" (UID: \"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4\") " pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"
Sep 12 00:49:12.163371 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.163216    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdsh\" (UniqueName: \"kubernetes.io/projected/64261abc-72f8-49f0-9109-da740a9122d1-kube-api-access-zhdsh\") pod \"payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf\" (UID: \"64261abc-72f8-49f0-9109-da740a9122d1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"
Sep 12 00:49:12.163371 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.163302    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64261abc-72f8-49f0-9109-da740a9122d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf\" (UID: \"64261abc-72f8-49f0-9109-da740a9122d1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"
Sep 12 00:49:12.163371 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.163351    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gccf\" (UniqueName: \"kubernetes.io/projected/d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4-kube-api-access-9gccf\") pod \"payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67\" (UID: \"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4\") " pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"
Sep 12 00:49:12.164193 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.164164    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67\" (UID: \"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4\") " pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"
Sep 12 00:49:12.164250 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.164201    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64261abc-72f8-49f0-9109-da740a9122d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf\" (UID: \"64261abc-72f8-49f0-9109-da740a9122d1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"
Sep 12 00:49:12.175893 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.175863    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdsh\" (UniqueName: \"kubernetes.io/projected/64261abc-72f8-49f0-9109-da740a9122d1-kube-api-access-zhdsh\") pod \"payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf\" (UID: \"64261abc-72f8-49f0-9109-da740a9122d1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"
Sep 12 00:49:12.176178 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.176113    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gccf\" (UniqueName: \"kubernetes.io/projected/d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4-kube-api-access-9gccf\") pod \"payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67\" (UID: \"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4\") " pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"
Sep 12 00:49:12.255754 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.255718    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"
Sep 12 00:49:12.308122 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.307889    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"
Sep 12 00:49:12.364638 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.364605    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 00:49:12.533401 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.533373    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"]
Sep 12 00:49:12.535856 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:49:12.535829    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9eea309_f4df_4b7d_b0c2_6f9e058b2ec4.slice/crio-8b7beafe762941c39c4b89040e059bd51519dba3360aeb0f9f2fd611628caa26 WatchSource:0}: Error finding container 8b7beafe762941c39c4b89040e059bd51519dba3360aeb0f9f2fd611628caa26: Status 404 returned error can't find the container with id 8b7beafe762941c39c4b89040e059bd51519dba3360aeb0f9f2fd611628caa26
Sep 12 00:49:12.596946 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:12.596918    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"]
Sep 12 00:49:12.598716 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:49:12.598693    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64261abc_72f8_49f0_9109_da740a9122d1.slice/crio-f55868b2dade997e60099d929ea4b050a20a2084de3a1606bad5c2a416900d94 WatchSource:0}: Error finding container f55868b2dade997e60099d929ea4b050a20a2084de3a1606bad5c2a416900d94: Status 404 returned error can't find the container with id f55868b2dade997e60099d929ea4b050a20a2084de3a1606bad5c2a416900d94
Sep 12 00:49:13.352805 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:13.352764    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf" event={"ID":"64261abc-72f8-49f0-9109-da740a9122d1","Type":"ContainerStarted","Data":"d09d848a5ed7485b8e3ac538c9976bcef7780c6453446b2226b94106a576821d"}
Sep 12 00:49:13.353385 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:13.352845    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf" event={"ID":"64261abc-72f8-49f0-9109-da740a9122d1","Type":"ContainerStarted","Data":"f55868b2dade997e60099d929ea4b050a20a2084de3a1606bad5c2a416900d94"}
Sep 12 00:49:13.353385 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:13.352898    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"
Sep 12 00:49:13.354612 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:13.354583    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67" event={"ID":"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4","Type":"ContainerStarted","Data":"8f82b9dcdc5ca575495ed21598c9c8b13cb3529189764f0a3fd4b13fc40a97c4"}
Sep 12 00:49:13.354726 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:13.354632    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67" event={"ID":"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4","Type":"ContainerStarted","Data":"8b7beafe762941c39c4b89040e059bd51519dba3360aeb0f9f2fd611628caa26"}
Sep 12 00:49:13.354837 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:13.354818    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"
Sep 12 00:49:13.370703 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:13.370648    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf" podStartSLOduration=2.370630024 podStartE2EDuration="2.370630024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:13.369147963 +0000 UTC m=+1185.573148783" watchObservedRunningTime="2026-09-12 00:49:13.370630024 +0000 UTC m=+1185.574630847"
Sep 12 00:49:13.385360 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:13.385312    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67" podStartSLOduration=2.385295433 podStartE2EDuration="2.385295433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:13.384441194 +0000 UTC m=+1185.588442017" watchObservedRunningTime="2026-09-12 00:49:13.385295433 +0000 UTC m=+1185.589296321"
Sep 12 00:49:24.363033 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:24.363001    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"
Sep 12 00:49:24.363429 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:24.363076    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"
Sep 12 00:49:25.533025 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.532984    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"]
Sep 12 00:49:25.535867 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.535845    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.538687 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.538659    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4d485e\"/\"premium-w0-4d485e-kserve-self-signed-certs\""
Sep 12 00:49:25.548491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.548464    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"]
Sep 12 00:49:25.597444 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.597399    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-tmp-dir\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.597668 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.597454    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t7gf\" (UniqueName: \"kubernetes.io/projected/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-kube-api-access-2t7gf\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.597668 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.597505    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-dshm\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.597668 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.597655    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-model-cache\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.597789 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.597716    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-home\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.597789 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.597764    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-tls-certs\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.698757 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.698717    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-model-cache\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.698946 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.698764    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-home\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.698946 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.698811    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-tls-certs\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.698946 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.698878    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-tmp-dir\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.698946 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.698915    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t7gf\" (UniqueName: \"kubernetes.io/projected/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-kube-api-access-2t7gf\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.698946 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.698943    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-dshm\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.699219 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.699156    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-model-cache\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.699278 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.699212    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-home\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.699278 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.699234    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-tmp-dir\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.701089 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.701068    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-dshm\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.701317 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.701301    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-tls-certs\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.707439 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.707421    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t7gf\" (UniqueName: \"kubernetes.io/projected/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-kube-api-access-2t7gf\") pod \"premium-w0-4d485e-kserve-79479c44cd-g75nm\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") " pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:25.846136 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:25.846031    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:26.116607 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:26.116505    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"]
Sep 12 00:49:26.119713 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:49:26.119684    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd312b5d9_e69e_4c04_8c12_0003d3dd69e7.slice/crio-a6717fe771f7fa0ca8bf09c28a1575f1a9df0e812f4f6fc190c983e07416c61b WatchSource:0}: Error finding container a6717fe771f7fa0ca8bf09c28a1575f1a9df0e812f4f6fc190c983e07416c61b: Status 404 returned error can't find the container with id a6717fe771f7fa0ca8bf09c28a1575f1a9df0e812f4f6fc190c983e07416c61b
Sep 12 00:49:26.432258 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:26.432166    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm" event={"ID":"d312b5d9-e69e-4c04-8c12-0003d3dd69e7","Type":"ContainerStarted","Data":"10d8f7efdf2630412051250a5c73a18a36266d0a498bf4dc543ee6ec9abedfa2"}
Sep 12 00:49:26.432258 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:26.432218    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm" event={"ID":"d312b5d9-e69e-4c04-8c12-0003d3dd69e7","Type":"ContainerStarted","Data":"a6717fe771f7fa0ca8bf09c28a1575f1a9df0e812f4f6fc190c983e07416c61b"}
Sep 12 00:49:26.450637 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:26.450558    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm" podStartSLOduration=1.450540172 podStartE2EDuration="1.450540172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:26.448550269 +0000 UTC m=+1198.652551092" watchObservedRunningTime="2026-09-12 00:49:26.450540172 +0000 UTC m=+1198.654540993"
Sep 12 00:49:35.847014 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:35.846963    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:35.847565 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:35.847038    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:35.859093 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:35.859064    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:49:36.499213 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:49:36.499184    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:50:22.079910 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.079826    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"]
Sep 12 00:50:22.083388 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.083369    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"
Sep 12 00:50:22.086509 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.086483    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ee23f934\""
Sep 12 00:50:22.086646 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.086534    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ee23f934-dockercfg-wh9hf\""
Sep 12 00:50:22.098451 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.098423    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"]
Sep 12 00:50:22.101169 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.101145    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"]
Sep 12 00:50:22.104500 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.104477    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"
Sep 12 00:50:22.121052 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.121010    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"]
Sep 12 00:50:22.251072 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.251031    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1018696f-2682-4cde-b2e2-1624545836bb-plugins-config-volume\") pod \"payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q\" (UID: \"1018696f-2682-4cde-b2e2-1624545836bb\") " pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"
Sep 12 00:50:22.251277 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.251086    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcbtq\" (UniqueName: \"kubernetes.io/projected/0232479b-d213-4ca3-8463-5fadbd1d66d4-kube-api-access-bcbtq\") pod \"payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk\" (UID: \"0232479b-d213-4ca3-8463-5fadbd1d66d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"
Sep 12 00:50:22.251277 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.251178    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0232479b-d213-4ca3-8463-5fadbd1d66d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk\" (UID: \"0232479b-d213-4ca3-8463-5fadbd1d66d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"
Sep 12 00:50:22.251388 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.251300    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4rzb\" (UniqueName: \"kubernetes.io/projected/1018696f-2682-4cde-b2e2-1624545836bb-kube-api-access-z4rzb\") pod \"payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q\" (UID: \"1018696f-2682-4cde-b2e2-1624545836bb\") " pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"
Sep 12 00:50:22.352076 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.351975    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcbtq\" (UniqueName: \"kubernetes.io/projected/0232479b-d213-4ca3-8463-5fadbd1d66d4-kube-api-access-bcbtq\") pod \"payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk\" (UID: \"0232479b-d213-4ca3-8463-5fadbd1d66d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"
Sep 12 00:50:22.352076 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.352035    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0232479b-d213-4ca3-8463-5fadbd1d66d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk\" (UID: \"0232479b-d213-4ca3-8463-5fadbd1d66d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"
Sep 12 00:50:22.352349 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.352131    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4rzb\" (UniqueName: \"kubernetes.io/projected/1018696f-2682-4cde-b2e2-1624545836bb-kube-api-access-z4rzb\") pod \"payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q\" (UID: \"1018696f-2682-4cde-b2e2-1624545836bb\") " pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"
Sep 12 00:50:22.352349 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.352188    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1018696f-2682-4cde-b2e2-1624545836bb-plugins-config-volume\") pod \"payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q\" (UID: \"1018696f-2682-4cde-b2e2-1624545836bb\") " pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"
Sep 12 00:50:22.352949 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.352923    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1018696f-2682-4cde-b2e2-1624545836bb-plugins-config-volume\") pod \"payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q\" (UID: \"1018696f-2682-4cde-b2e2-1624545836bb\") " pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"
Sep 12 00:50:22.353048 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.352968    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0232479b-d213-4ca3-8463-5fadbd1d66d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk\" (UID: \"0232479b-d213-4ca3-8463-5fadbd1d66d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"
Sep 12 00:50:22.360963 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.360928    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcbtq\" (UniqueName: \"kubernetes.io/projected/0232479b-d213-4ca3-8463-5fadbd1d66d4-kube-api-access-bcbtq\") pod \"payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk\" (UID: \"0232479b-d213-4ca3-8463-5fadbd1d66d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"
Sep 12 00:50:22.361067 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.360965    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4rzb\" (UniqueName: \"kubernetes.io/projected/1018696f-2682-4cde-b2e2-1624545836bb-kube-api-access-z4rzb\") pod \"payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q\" (UID: \"1018696f-2682-4cde-b2e2-1624545836bb\") " pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"
Sep 12 00:50:22.394392 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.394355    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"
Sep 12 00:50:22.414722 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.414695    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"
Sep 12 00:50:22.675305 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.675273    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"]
Sep 12 00:50:22.677532 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:50:22.677504    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0232479b_d213_4ca3_8463_5fadbd1d66d4.slice/crio-e7f8eced3f639b36c95c691ef4a6029704f106a378a602eaa1d69eed85b09f37 WatchSource:0}: Error finding container e7f8eced3f639b36c95c691ef4a6029704f106a378a602eaa1d69eed85b09f37: Status 404 returned error can't find the container with id e7f8eced3f639b36c95c691ef4a6029704f106a378a602eaa1d69eed85b09f37
Sep 12 00:50:22.699153 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.699051    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"]
Sep 12 00:50:22.717038 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:22.717006    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk" event={"ID":"0232479b-d213-4ca3-8463-5fadbd1d66d4","Type":"ContainerStarted","Data":"e7f8eced3f639b36c95c691ef4a6029704f106a378a602eaa1d69eed85b09f37"}
Sep 12 00:50:22.719713 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:50:22.719690    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1018696f_2682_4cde_b2e2_1624545836bb.slice/crio-2c78a23fdac21465799e2d88eda00610d55a07557d6455f3c3e24969bf8b1651 WatchSource:0}: Error finding container 2c78a23fdac21465799e2d88eda00610d55a07557d6455f3c3e24969bf8b1651: Status 404 returned error can't find the container with id 2c78a23fdac21465799e2d88eda00610d55a07557d6455f3c3e24969bf8b1651
Sep 12 00:50:23.723683 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:23.723640    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q" event={"ID":"1018696f-2682-4cde-b2e2-1624545836bb","Type":"ContainerStarted","Data":"c78aab1a6e9647837a4fac284efd7734649a16da3d4e2a702d17f52d0d123f60"}
Sep 12 00:50:23.724152 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:23.723710    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q" event={"ID":"1018696f-2682-4cde-b2e2-1624545836bb","Type":"ContainerStarted","Data":"2c78a23fdac21465799e2d88eda00610d55a07557d6455f3c3e24969bf8b1651"}
Sep 12 00:50:23.724152 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:23.723737    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"
Sep 12 00:50:23.725134 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:23.725106    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk" event={"ID":"0232479b-d213-4ca3-8463-5fadbd1d66d4","Type":"ContainerStarted","Data":"73b8a11bc89ffa576bdd78801f082ddec6099df3ff3127cb55359a5407530f08"}
Sep 12 00:50:23.725245 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:23.725229    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"
Sep 12 00:50:23.740254 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:23.740202    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q" podStartSLOduration=1.740182398 podStartE2EDuration="1.740182398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:50:23.739735059 +0000 UTC m=+1255.943735884" watchObservedRunningTime="2026-09-12 00:50:23.740182398 +0000 UTC m=+1255.944183221"
Sep 12 00:50:23.756368 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:23.756309    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk" podStartSLOduration=1.756292265 podStartE2EDuration="1.756292265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:50:23.755557407 +0000 UTC m=+1255.959558239" watchObservedRunningTime="2026-09-12 00:50:23.756292265 +0000 UTC m=+1255.960293085"
Sep 12 00:50:34.731858 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:34.731826    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"
Sep 12 00:50:34.733455 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:34.733049    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"
Sep 12 00:50:55.259014 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:55.258954    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22"]
Sep 12 00:50:55.265309 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:55.265275    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22"
Sep 12 00:50:55.270315 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:55.270289    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:50:55.270456 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:55.270290    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:50:55.289599 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:55.289552    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22"]
Sep 12 00:50:55.373497 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:55.373449    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5038722-55ba-4cea-8551-0a5e24640d01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ee23f934-hnm22\" (UID: \"f5038722-55ba-4cea-8551-0a5e24640d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22"
Sep 12 00:50:55.474394 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:55.474343    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5038722-55ba-4cea-8551-0a5e24640d01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ee23f934-hnm22\" (UID: \"f5038722-55ba-4cea-8551-0a5e24640d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22"
Sep 12 00:50:55.475084 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:55.475061    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5038722-55ba-4cea-8551-0a5e24640d01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ee23f934-hnm22\" (UID: \"f5038722-55ba-4cea-8551-0a5e24640d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22"
Sep 12 00:50:55.575911 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:55.575807    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22"
Sep 12 00:50:55.869353 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:55.869269    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22"]
Sep 12 00:50:55.873664 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:50:55.873629    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5038722_55ba_4cea_8551_0a5e24640d01.slice/crio-dde3efa4a8c4242bcf5661fccb12bc40d74c84cf9add0e382cf016dd1986e742 WatchSource:0}: Error finding container dde3efa4a8c4242bcf5661fccb12bc40d74c84cf9add0e382cf016dd1986e742: Status 404 returned error can't find the container with id dde3efa4a8c4242bcf5661fccb12bc40d74c84cf9add0e382cf016dd1986e742
Sep 12 00:50:55.889089 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:55.889050    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22" event={"ID":"f5038722-55ba-4cea-8551-0a5e24640d01","Type":"ContainerStarted","Data":"dde3efa4a8c4242bcf5661fccb12bc40d74c84cf9add0e382cf016dd1986e742"}
Sep 12 00:50:56.895648 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:56.895546    2567 generic.go:356] "Generic (PLEG): container finished" podID="f5038722-55ba-4cea-8551-0a5e24640d01" containerID="cfdb4f0c368d882548356a7b073fb52e36da1af3ccc74ac767ee5315dc3a1086" exitCode=0
Sep 12 00:50:56.896023 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:56.895639    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22" event={"ID":"f5038722-55ba-4cea-8551-0a5e24640d01","Type":"ContainerDied","Data":"cfdb4f0c368d882548356a7b073fb52e36da1af3ccc74ac767ee5315dc3a1086"}
Sep 12 00:50:58.033765 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:58.033737    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22"
Sep 12 00:50:58.202892 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:58.202793    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5038722-55ba-4cea-8551-0a5e24640d01-maas-api-service-ca\") pod \"f5038722-55ba-4cea-8551-0a5e24640d01\" (UID: \"f5038722-55ba-4cea-8551-0a5e24640d01\") "
Sep 12 00:50:58.203188 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:58.203163    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5038722-55ba-4cea-8551-0a5e24640d01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5038722-55ba-4cea-8551-0a5e24640d01" (UID: "f5038722-55ba-4cea-8551-0a5e24640d01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:50:58.303803 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:58.303762    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5038722-55ba-4cea-8551-0a5e24640d01-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:50:58.907558 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:58.907528    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22"
Sep 12 00:50:58.907762 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:58.907599    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22" event={"ID":"f5038722-55ba-4cea-8551-0a5e24640d01","Type":"ContainerDied","Data":"dde3efa4a8c4242bcf5661fccb12bc40d74c84cf9add0e382cf016dd1986e742"}
Sep 12 00:50:58.907762 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:50:58.907651    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dde3efa4a8c4242bcf5661fccb12bc40d74c84cf9add0e382cf016dd1986e742"
Sep 12 00:51:06.276268 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.276218    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"]
Sep 12 00:51:06.276848 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.276503    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q" podUID="1018696f-2682-4cde-b2e2-1624545836bb" containerName="payload-processing" containerID="cri-o://c78aab1a6e9647837a4fac284efd7734649a16da3d4e2a702d17f52d0d123f60" gracePeriod=30
Sep 12 00:51:06.304398 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.304360    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"]
Sep 12 00:51:06.304744 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.304688    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk" podUID="0232479b-d213-4ca3-8463-5fadbd1d66d4" containerName="payload-pre-processing" containerID="cri-o://73b8a11bc89ffa576bdd78801f082ddec6099df3ff3127cb55359a5407530f08" gracePeriod=30
Sep 12 00:51:06.590452 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.590420    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"
Sep 12 00:51:06.592650 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.592625    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"
Sep 12 00:51:06.625986 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.625949    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22"]
Sep 12 00:51:06.630178 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.630141    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee23f934-hnm22"]
Sep 12 00:51:06.788714 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.788675    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4rzb\" (UniqueName: \"kubernetes.io/projected/1018696f-2682-4cde-b2e2-1624545836bb-kube-api-access-z4rzb\") pod \"1018696f-2682-4cde-b2e2-1624545836bb\" (UID: \"1018696f-2682-4cde-b2e2-1624545836bb\") "
Sep 12 00:51:06.788888 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.788788    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1018696f-2682-4cde-b2e2-1624545836bb-plugins-config-volume\") pod \"1018696f-2682-4cde-b2e2-1624545836bb\" (UID: \"1018696f-2682-4cde-b2e2-1624545836bb\") "
Sep 12 00:51:06.788888 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.788875    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0232479b-d213-4ca3-8463-5fadbd1d66d4-plugins-config-volume\") pod \"0232479b-d213-4ca3-8463-5fadbd1d66d4\" (UID: \"0232479b-d213-4ca3-8463-5fadbd1d66d4\") "
Sep 12 00:51:06.788969 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.788904    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcbtq\" (UniqueName: \"kubernetes.io/projected/0232479b-d213-4ca3-8463-5fadbd1d66d4-kube-api-access-bcbtq\") pod \"0232479b-d213-4ca3-8463-5fadbd1d66d4\" (UID: \"0232479b-d213-4ca3-8463-5fadbd1d66d4\") "
Sep 12 00:51:06.789069 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.789048    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1018696f-2682-4cde-b2e2-1624545836bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1018696f-2682-4cde-b2e2-1624545836bb" (UID: "1018696f-2682-4cde-b2e2-1624545836bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:51:06.789245 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.789224    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0232479b-d213-4ca3-8463-5fadbd1d66d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0232479b-d213-4ca3-8463-5fadbd1d66d4" (UID: "0232479b-d213-4ca3-8463-5fadbd1d66d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:51:06.789360 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.789343    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1018696f-2682-4cde-b2e2-1624545836bb-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:51:06.789428 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.789362    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0232479b-d213-4ca3-8463-5fadbd1d66d4-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:51:06.790894 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.790869    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0232479b-d213-4ca3-8463-5fadbd1d66d4-kube-api-access-bcbtq" (OuterVolumeSpecName: "kube-api-access-bcbtq") pod "0232479b-d213-4ca3-8463-5fadbd1d66d4" (UID: "0232479b-d213-4ca3-8463-5fadbd1d66d4"). InnerVolumeSpecName "kube-api-access-bcbtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:51:06.790993 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.790901    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1018696f-2682-4cde-b2e2-1624545836bb-kube-api-access-z4rzb" (OuterVolumeSpecName: "kube-api-access-z4rzb") pod "1018696f-2682-4cde-b2e2-1624545836bb" (UID: "1018696f-2682-4cde-b2e2-1624545836bb"). InnerVolumeSpecName "kube-api-access-z4rzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:51:06.890551 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.890459    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4rzb\" (UniqueName: \"kubernetes.io/projected/1018696f-2682-4cde-b2e2-1624545836bb-kube-api-access-z4rzb\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:51:06.890551 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.890490    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcbtq\" (UniqueName: \"kubernetes.io/projected/0232479b-d213-4ca3-8463-5fadbd1d66d4-kube-api-access-bcbtq\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:51:06.953770 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.953731    2567 generic.go:356] "Generic (PLEG): container finished" podID="1018696f-2682-4cde-b2e2-1624545836bb" containerID="c78aab1a6e9647837a4fac284efd7734649a16da3d4e2a702d17f52d0d123f60" exitCode=0
Sep 12 00:51:06.953972 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.953786    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"
Sep 12 00:51:06.953972 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.953815    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q" event={"ID":"1018696f-2682-4cde-b2e2-1624545836bb","Type":"ContainerDied","Data":"c78aab1a6e9647837a4fac284efd7734649a16da3d4e2a702d17f52d0d123f60"}
Sep 12 00:51:06.953972 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.953873    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q" event={"ID":"1018696f-2682-4cde-b2e2-1624545836bb","Type":"ContainerDied","Data":"2c78a23fdac21465799e2d88eda00610d55a07557d6455f3c3e24969bf8b1651"}
Sep 12 00:51:06.953972 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.953899    2567 scope.go:117] "RemoveContainer" containerID="c78aab1a6e9647837a4fac284efd7734649a16da3d4e2a702d17f52d0d123f60"
Sep 12 00:51:06.955223 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.955199    2567 generic.go:356] "Generic (PLEG): container finished" podID="0232479b-d213-4ca3-8463-5fadbd1d66d4" containerID="73b8a11bc89ffa576bdd78801f082ddec6099df3ff3127cb55359a5407530f08" exitCode=0
Sep 12 00:51:06.955347 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.955229    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk" event={"ID":"0232479b-d213-4ca3-8463-5fadbd1d66d4","Type":"ContainerDied","Data":"73b8a11bc89ffa576bdd78801f082ddec6099df3ff3127cb55359a5407530f08"}
Sep 12 00:51:06.955347 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.955251    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"
Sep 12 00:51:06.955347 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.955285    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk" event={"ID":"0232479b-d213-4ca3-8463-5fadbd1d66d4","Type":"ContainerDied","Data":"e7f8eced3f639b36c95c691ef4a6029704f106a378a602eaa1d69eed85b09f37"}
Sep 12 00:51:06.964154 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.964113    2567 scope.go:117] "RemoveContainer" containerID="c78aab1a6e9647837a4fac284efd7734649a16da3d4e2a702d17f52d0d123f60"
Sep 12 00:51:06.964481 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:51:06.964448    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78aab1a6e9647837a4fac284efd7734649a16da3d4e2a702d17f52d0d123f60\": container with ID starting with c78aab1a6e9647837a4fac284efd7734649a16da3d4e2a702d17f52d0d123f60 not found: ID does not exist" containerID="c78aab1a6e9647837a4fac284efd7734649a16da3d4e2a702d17f52d0d123f60"
Sep 12 00:51:06.964607 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.964486    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78aab1a6e9647837a4fac284efd7734649a16da3d4e2a702d17f52d0d123f60"} err="failed to get container status \"c78aab1a6e9647837a4fac284efd7734649a16da3d4e2a702d17f52d0d123f60\": rpc error: code = NotFound desc = could not find container \"c78aab1a6e9647837a4fac284efd7734649a16da3d4e2a702d17f52d0d123f60\": container with ID starting with c78aab1a6e9647837a4fac284efd7734649a16da3d4e2a702d17f52d0d123f60 not found: ID does not exist"
Sep 12 00:51:06.964607 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.964506    2567 scope.go:117] "RemoveContainer" containerID="73b8a11bc89ffa576bdd78801f082ddec6099df3ff3127cb55359a5407530f08"
Sep 12 00:51:06.972546 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.972529    2567 scope.go:117] "RemoveContainer" containerID="73b8a11bc89ffa576bdd78801f082ddec6099df3ff3127cb55359a5407530f08"
Sep 12 00:51:06.972856 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:51:06.972837    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b8a11bc89ffa576bdd78801f082ddec6099df3ff3127cb55359a5407530f08\": container with ID starting with 73b8a11bc89ffa576bdd78801f082ddec6099df3ff3127cb55359a5407530f08 not found: ID does not exist" containerID="73b8a11bc89ffa576bdd78801f082ddec6099df3ff3127cb55359a5407530f08"
Sep 12 00:51:06.972901 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.972863    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b8a11bc89ffa576bdd78801f082ddec6099df3ff3127cb55359a5407530f08"} err="failed to get container status \"73b8a11bc89ffa576bdd78801f082ddec6099df3ff3127cb55359a5407530f08\": rpc error: code = NotFound desc = could not find container \"73b8a11bc89ffa576bdd78801f082ddec6099df3ff3127cb55359a5407530f08\": container with ID starting with 73b8a11bc89ffa576bdd78801f082ddec6099df3ff3127cb55359a5407530f08 not found: ID does not exist"
Sep 12 00:51:06.986485 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.986459    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"]
Sep 12 00:51:06.996491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:06.996461    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ee23f934-5b9cf5577c-s484q"]
Sep 12 00:51:07.028154 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:07.028120    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"]
Sep 12 00:51:07.044494 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:07.044452    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ee23f934-5569bfc5f8-p54kk"]
Sep 12 00:51:08.273874 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:08.273839    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0232479b-d213-4ca3-8463-5fadbd1d66d4" path="/var/lib/kubelet/pods/0232479b-d213-4ca3-8463-5fadbd1d66d4/volumes"
Sep 12 00:51:08.274254 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:08.274219    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1018696f-2682-4cde-b2e2-1624545836bb" path="/var/lib/kubelet/pods/1018696f-2682-4cde-b2e2-1624545836bb/volumes"
Sep 12 00:51:08.274507 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:08.274494    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5038722-55ba-4cea-8551-0a5e24640d01" path="/var/lib/kubelet/pods/f5038722-55ba-4cea-8551-0a5e24640d01/volumes"
Sep 12 00:51:25.663407 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.663358    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"]
Sep 12 00:51:25.663966 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.663784    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1018696f-2682-4cde-b2e2-1624545836bb" containerName="payload-processing"
Sep 12 00:51:25.663966 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.663798    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1018696f-2682-4cde-b2e2-1624545836bb" containerName="payload-processing"
Sep 12 00:51:25.663966 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.663818    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0232479b-d213-4ca3-8463-5fadbd1d66d4" containerName="payload-pre-processing"
Sep 12 00:51:25.663966 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.663823    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0232479b-d213-4ca3-8463-5fadbd1d66d4" containerName="payload-pre-processing"
Sep 12 00:51:25.663966 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.663838    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5038722-55ba-4cea-8551-0a5e24640d01" containerName="revoke-keys"
Sep 12 00:51:25.663966 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.663845    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5038722-55ba-4cea-8551-0a5e24640d01" containerName="revoke-keys"
Sep 12 00:51:25.663966 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.663902    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5038722-55ba-4cea-8551-0a5e24640d01" containerName="revoke-keys"
Sep 12 00:51:25.663966 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.663912    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1018696f-2682-4cde-b2e2-1624545836bb" containerName="payload-processing"
Sep 12 00:51:25.663966 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.663919    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0232479b-d213-4ca3-8463-5fadbd1d66d4" containerName="payload-pre-processing"
Sep 12 00:51:25.668085 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.668065    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b"]
Sep 12 00:51:25.668209 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.668193    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.671540 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.671493    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b"
Sep 12 00:51:25.671925 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.671905    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-876808\"/\"e2e-conflict-a-f937eb5b-kserve-self-signed-certs\""
Sep 12 00:51:25.681085 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.681065    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:51:25.699392 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.699360    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:51:25.714402 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.714360    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b"]
Sep 12 00:51:25.717248 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.717222    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"]
Sep 12 00:51:25.774448 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.774404    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-home\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.774448 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.774448    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-dshm\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.774783 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.774481    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-tmp-dir\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.774783 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.774559    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnbbj\" (UniqueName: \"kubernetes.io/projected/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-kube-api-access-tnbbj\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.774783 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.774614    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-tls-certs\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.774783 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.774644    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1923ef30-7fd6-4de4-8946-aced6a6f65d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b\" (UID: \"1923ef30-7fd6-4de4-8946-aced6a6f65d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b"
Sep 12 00:51:25.775066 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.774787    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-model-cache\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.875924 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.875883    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-model-cache\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.876107 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.875941    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-home\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.876107 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.875961    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-dshm\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.876107 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.875989    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-tmp-dir\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.876107 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.876025    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbbj\" (UniqueName: \"kubernetes.io/projected/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-kube-api-access-tnbbj\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.876277 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.876164    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-tls-certs\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.876277 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.876202    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1923ef30-7fd6-4de4-8946-aced6a6f65d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b\" (UID: \"1923ef30-7fd6-4de4-8946-aced6a6f65d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b"
Sep 12 00:51:25.876370 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.876339    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-model-cache\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.876370 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.876353    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-home\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.876451 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.876419    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-tmp-dir\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.876941 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.876924    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1923ef30-7fd6-4de4-8946-aced6a6f65d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b\" (UID: \"1923ef30-7fd6-4de4-8946-aced6a6f65d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b"
Sep 12 00:51:25.878318 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.878298    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-dshm\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.878621 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.878606    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-tls-certs\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.893879 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.893851    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbbj\" (UniqueName: \"kubernetes.io/projected/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-kube-api-access-tnbbj\") pod \"e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") " pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.979380 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.979341    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:25.984637 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:25.984613    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b"
Sep 12 00:51:26.296227 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:26.296172    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b"]
Sep 12 00:51:26.298975 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:51:26.298941    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1923ef30_7fd6_4de4_8946_aced6a6f65d8.slice/crio-a76b783b8214f863db42a0cc36e2bfea3e641ee9031c052fa582abf554b4e6e6 WatchSource:0}: Error finding container a76b783b8214f863db42a0cc36e2bfea3e641ee9031c052fa582abf554b4e6e6: Status 404 returned error can't find the container with id a76b783b8214f863db42a0cc36e2bfea3e641ee9031c052fa582abf554b4e6e6
Sep 12 00:51:26.302214 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:26.302188    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"]
Sep 12 00:51:26.306518 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:51:26.306440    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65f31f55_918f_4cf2_99a8_6e8cc1f76f1d.slice/crio-fc9361dc017df344433138c319e6a5c02dcc0996d8bad84f2da7bc96ce571b3f WatchSource:0}: Error finding container fc9361dc017df344433138c319e6a5c02dcc0996d8bad84f2da7bc96ce571b3f: Status 404 returned error can't find the container with id fc9361dc017df344433138c319e6a5c02dcc0996d8bad84f2da7bc96ce571b3f
Sep 12 00:51:27.065406 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:27.065364    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh" event={"ID":"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d","Type":"ContainerStarted","Data":"874a552c7d2edaf3ee7e8bb4e4ad7e4a3e1d272d7ed3fd0aefddd0aefed8dc28"}
Sep 12 00:51:27.065855 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:27.065414    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh" event={"ID":"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d","Type":"ContainerStarted","Data":"fc9361dc017df344433138c319e6a5c02dcc0996d8bad84f2da7bc96ce571b3f"}
Sep 12 00:51:27.066936 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:27.066905    2567 generic.go:356] "Generic (PLEG): container finished" podID="1923ef30-7fd6-4de4-8946-aced6a6f65d8" containerID="c646e818aaa0e7fb3e43ece703c60c3a5e075bf402a5f8db085093855eaf747e" exitCode=0
Sep 12 00:51:27.067045 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:27.066985    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b" event={"ID":"1923ef30-7fd6-4de4-8946-aced6a6f65d8","Type":"ContainerDied","Data":"c646e818aaa0e7fb3e43ece703c60c3a5e075bf402a5f8db085093855eaf747e"}
Sep 12 00:51:27.067045 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:27.067032    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b" event={"ID":"1923ef30-7fd6-4de4-8946-aced6a6f65d8","Type":"ContainerStarted","Data":"a76b783b8214f863db42a0cc36e2bfea3e641ee9031c052fa582abf554b4e6e6"}
Sep 12 00:51:27.142151 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:27.142093    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh" podStartSLOduration=2.142071194 podStartE2EDuration="2.142071194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:51:27.141112012 +0000 UTC m=+1319.345112834" watchObservedRunningTime="2026-09-12 00:51:27.142071194 +0000 UTC m=+1319.346072015"
Sep 12 00:51:28.212610 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:28.212584    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b"
Sep 12 00:51:28.302689 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:28.302655    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1923ef30-7fd6-4de4-8946-aced6a6f65d8-maas-api-service-ca\") pod \"1923ef30-7fd6-4de4-8946-aced6a6f65d8\" (UID: \"1923ef30-7fd6-4de4-8946-aced6a6f65d8\") "
Sep 12 00:51:28.303028 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:28.303001    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1923ef30-7fd6-4de4-8946-aced6a6f65d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1923ef30-7fd6-4de4-8946-aced6a6f65d8" (UID: "1923ef30-7fd6-4de4-8946-aced6a6f65d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:51:28.403890 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:28.403802    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1923ef30-7fd6-4de4-8946-aced6a6f65d8-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:51:29.079494 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:29.079406    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b"
Sep 12 00:51:29.079494 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:29.079439    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b" event={"ID":"1923ef30-7fd6-4de4-8946-aced6a6f65d8","Type":"ContainerDied","Data":"a76b783b8214f863db42a0cc36e2bfea3e641ee9031c052fa582abf554b4e6e6"}
Sep 12 00:51:29.079708 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:29.079513    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a76b783b8214f863db42a0cc36e2bfea3e641ee9031c052fa582abf554b4e6e6"
Sep 12 00:51:35.979684 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:35.979645    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:35.980101 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:35.979696    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:35.992535 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:35.992506    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:36.130305 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:36.130269    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:51:36.888372 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:36.888334    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"]
Sep 12 00:51:36.888714 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:36.888660    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67" podUID="d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4" containerName="payload-processing" containerID="cri-o://8f82b9dcdc5ca575495ed21598c9c8b13cb3529189764f0a3fd4b13fc40a97c4" gracePeriod=30
Sep 12 00:51:36.899914 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:36.899876    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"]
Sep 12 00:51:36.900274 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:36.900240    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf" podUID="64261abc-72f8-49f0-9109-da740a9122d1" containerName="payload-pre-processing" containerID="cri-o://d09d848a5ed7485b8e3ac538c9976bcef7780c6453446b2226b94106a576821d" gracePeriod=30
Sep 12 00:51:37.127328 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.127287    2567 generic.go:356] "Generic (PLEG): container finished" podID="d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4" containerID="8f82b9dcdc5ca575495ed21598c9c8b13cb3529189764f0a3fd4b13fc40a97c4" exitCode=0
Sep 12 00:51:37.127843 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.127379    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67" event={"ID":"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4","Type":"ContainerDied","Data":"8f82b9dcdc5ca575495ed21598c9c8b13cb3529189764f0a3fd4b13fc40a97c4"}
Sep 12 00:51:37.129461 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.129437    2567 generic.go:356] "Generic (PLEG): container finished" podID="64261abc-72f8-49f0-9109-da740a9122d1" containerID="d09d848a5ed7485b8e3ac538c9976bcef7780c6453446b2226b94106a576821d" exitCode=0
Sep 12 00:51:37.129600 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.129516    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf" event={"ID":"64261abc-72f8-49f0-9109-da740a9122d1","Type":"ContainerDied","Data":"d09d848a5ed7485b8e3ac538c9976bcef7780c6453446b2226b94106a576821d"}
Sep 12 00:51:37.190057 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.190031    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"
Sep 12 00:51:37.193126 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.193104    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"
Sep 12 00:51:37.296030 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.295991    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gccf\" (UniqueName: \"kubernetes.io/projected/d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4-kube-api-access-9gccf\") pod \"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4\" (UID: \"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4\") "
Sep 12 00:51:37.296236 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.296046    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4-plugins-config-volume\") pod \"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4\" (UID: \"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4\") "
Sep 12 00:51:37.296236 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.296077    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64261abc-72f8-49f0-9109-da740a9122d1-plugins-config-volume\") pod \"64261abc-72f8-49f0-9109-da740a9122d1\" (UID: \"64261abc-72f8-49f0-9109-da740a9122d1\") "
Sep 12 00:51:37.296236 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.296145    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhdsh\" (UniqueName: \"kubernetes.io/projected/64261abc-72f8-49f0-9109-da740a9122d1-kube-api-access-zhdsh\") pod \"64261abc-72f8-49f0-9109-da740a9122d1\" (UID: \"64261abc-72f8-49f0-9109-da740a9122d1\") "
Sep 12 00:51:37.296551 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.296519    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4" (UID: "d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:51:37.296650 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.296527    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64261abc-72f8-49f0-9109-da740a9122d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64261abc-72f8-49f0-9109-da740a9122d1" (UID: "64261abc-72f8-49f0-9109-da740a9122d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:51:37.298441 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.298415    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64261abc-72f8-49f0-9109-da740a9122d1-kube-api-access-zhdsh" (OuterVolumeSpecName: "kube-api-access-zhdsh") pod "64261abc-72f8-49f0-9109-da740a9122d1" (UID: "64261abc-72f8-49f0-9109-da740a9122d1"). InnerVolumeSpecName "kube-api-access-zhdsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:51:37.298769 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.298746    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4-kube-api-access-9gccf" (OuterVolumeSpecName: "kube-api-access-9gccf") pod "d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4" (UID: "d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4"). InnerVolumeSpecName "kube-api-access-9gccf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:51:37.358934 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.358898    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b"]
Sep 12 00:51:37.368357 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.368326    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c3f8d8-pp76b"]
Sep 12 00:51:37.397940 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.397899    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gccf\" (UniqueName: \"kubernetes.io/projected/d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4-kube-api-access-9gccf\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:51:37.397940 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.397932    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:51:37.397940 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.397950    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64261abc-72f8-49f0-9109-da740a9122d1-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:51:37.398186 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:37.397962    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhdsh\" (UniqueName: \"kubernetes.io/projected/64261abc-72f8-49f0-9109-da740a9122d1-kube-api-access-zhdsh\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:51:38.136850 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:38.136820    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"
Sep 12 00:51:38.137345 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:38.136840    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67" event={"ID":"d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4","Type":"ContainerDied","Data":"8b7beafe762941c39c4b89040e059bd51519dba3360aeb0f9f2fd611628caa26"}
Sep 12 00:51:38.137345 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:38.136957    2567 scope.go:117] "RemoveContainer" containerID="8f82b9dcdc5ca575495ed21598c9c8b13cb3529189764f0a3fd4b13fc40a97c4"
Sep 12 00:51:38.138594 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:38.138550    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf" event={"ID":"64261abc-72f8-49f0-9109-da740a9122d1","Type":"ContainerDied","Data":"f55868b2dade997e60099d929ea4b050a20a2084de3a1606bad5c2a416900d94"}
Sep 12 00:51:38.138717 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:38.138606    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"
Sep 12 00:51:38.148817 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:38.148792    2567 scope.go:117] "RemoveContainer" containerID="d09d848a5ed7485b8e3ac538c9976bcef7780c6453446b2226b94106a576821d"
Sep 12 00:51:38.195220 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:38.195184    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"]
Sep 12 00:51:38.196992 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:38.196963    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c3f8d8-7894ccff57-slm67"]
Sep 12 00:51:38.248309 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:38.248274    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"]
Sep 12 00:51:38.249942 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:38.249921    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c3f8d8-7659c6cc8b-4l2jf"]
Sep 12 00:51:38.274110 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:38.274075    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1923ef30-7fd6-4de4-8946-aced6a6f65d8" path="/var/lib/kubelet/pods/1923ef30-7fd6-4de4-8946-aced6a6f65d8/volumes"
Sep 12 00:51:38.274523 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:38.274507    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64261abc-72f8-49f0-9109-da740a9122d1" path="/var/lib/kubelet/pods/64261abc-72f8-49f0-9109-da740a9122d1/volumes"
Sep 12 00:51:38.274911 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:38.274897    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4" path="/var/lib/kubelet/pods/d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4/volumes"
Sep 12 00:51:46.365777 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.365733    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"]
Sep 12 00:51:46.366183 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.366086    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64261abc-72f8-49f0-9109-da740a9122d1" containerName="payload-pre-processing"
Sep 12 00:51:46.366183 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.366097    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="64261abc-72f8-49f0-9109-da740a9122d1" containerName="payload-pre-processing"
Sep 12 00:51:46.366183 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.366114    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4" containerName="payload-processing"
Sep 12 00:51:46.366183 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.366121    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4" containerName="payload-processing"
Sep 12 00:51:46.366183 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.366137    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1923ef30-7fd6-4de4-8946-aced6a6f65d8" containerName="revoke-keys"
Sep 12 00:51:46.366183 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.366143    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1923ef30-7fd6-4de4-8946-aced6a6f65d8" containerName="revoke-keys"
Sep 12 00:51:46.366381 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.366200    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="64261abc-72f8-49f0-9109-da740a9122d1" containerName="payload-pre-processing"
Sep 12 00:51:46.366381 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.366208    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9eea309-f4df-4b7d-b0c2-6f9e058b2ec4" containerName="payload-processing"
Sep 12 00:51:46.366381 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.366214    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1923ef30-7fd6-4de4-8946-aced6a6f65d8" containerName="revoke-keys"
Sep 12 00:51:46.370567 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.370543    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"
Sep 12 00:51:46.375042 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.375022    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-088c83\""
Sep 12 00:51:46.385611 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.385551    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qz5p\" (UniqueName: \"kubernetes.io/projected/57c84d1e-44c3-4942-a621-63fe6c439c0d-kube-api-access-5qz5p\") pod \"payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6\" (UID: \"57c84d1e-44c3-4942-a621-63fe6c439c0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"
Sep 12 00:51:46.385794 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.385661    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c84d1e-44c3-4942-a621-63fe6c439c0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6\" (UID: \"57c84d1e-44c3-4942-a621-63fe6c439c0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"
Sep 12 00:51:46.387529 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.387499    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"]
Sep 12 00:51:46.432321 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.432282    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"]
Sep 12 00:51:46.435582 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.435550    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"
Sep 12 00:51:46.439441 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:51:46.439403    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="71d398c5-af9c-4f61-adbd-f5f0d945e00f" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"
Sep 12 00:51:46.440121 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:51:46.440095    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-shared-a-w2-088c83-dockercfg-42xzl\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-088c83-dockercfg-42xzl\"" type="*v1.Secret"
Sep 12 00:51:46.486876 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.486831    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qz5p\" (UniqueName: \"kubernetes.io/projected/57c84d1e-44c3-4942-a621-63fe6c439c0d-kube-api-access-5qz5p\") pod \"payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6\" (UID: \"57c84d1e-44c3-4942-a621-63fe6c439c0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"
Sep 12 00:51:46.487096 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.486888    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85phc\" (UniqueName: \"kubernetes.io/projected/71d398c5-af9c-4f61-adbd-f5f0d945e00f-kube-api-access-85phc\") pod \"payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l\" (UID: \"71d398c5-af9c-4f61-adbd-f5f0d945e00f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"
Sep 12 00:51:46.487096 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.486918    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c84d1e-44c3-4942-a621-63fe6c439c0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6\" (UID: \"57c84d1e-44c3-4942-a621-63fe6c439c0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"
Sep 12 00:51:46.487096 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.487074    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d398c5-af9c-4f61-adbd-f5f0d945e00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l\" (UID: \"71d398c5-af9c-4f61-adbd-f5f0d945e00f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"
Sep 12 00:51:46.487658 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.487639    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c84d1e-44c3-4942-a621-63fe6c439c0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6\" (UID: \"57c84d1e-44c3-4942-a621-63fe6c439c0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"
Sep 12 00:51:46.497203 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.497060    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"]
Sep 12 00:51:46.505302 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.505267    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qz5p\" (UniqueName: \"kubernetes.io/projected/57c84d1e-44c3-4942-a621-63fe6c439c0d-kube-api-access-5qz5p\") pod \"payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6\" (UID: \"57c84d1e-44c3-4942-a621-63fe6c439c0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"
Sep 12 00:51:46.588636 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.588594    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85phc\" (UniqueName: \"kubernetes.io/projected/71d398c5-af9c-4f61-adbd-f5f0d945e00f-kube-api-access-85phc\") pod \"payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l\" (UID: \"71d398c5-af9c-4f61-adbd-f5f0d945e00f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"
Sep 12 00:51:46.588827 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.588720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d398c5-af9c-4f61-adbd-f5f0d945e00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l\" (UID: \"71d398c5-af9c-4f61-adbd-f5f0d945e00f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"
Sep 12 00:51:46.589425 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.589402    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d398c5-af9c-4f61-adbd-f5f0d945e00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l\" (UID: \"71d398c5-af9c-4f61-adbd-f5f0d945e00f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"
Sep 12 00:51:46.604286 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.604252    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85phc\" (UniqueName: \"kubernetes.io/projected/71d398c5-af9c-4f61-adbd-f5f0d945e00f-kube-api-access-85phc\") pod \"payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l\" (UID: \"71d398c5-af9c-4f61-adbd-f5f0d945e00f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"
Sep 12 00:51:46.681120 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.681031    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"
Sep 12 00:51:46.961162 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:46.961088    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"]
Sep 12 00:51:46.963725 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:51:46.963695    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c84d1e_44c3_4942_a621_63fe6c439c0d.slice/crio-8b3844f737bab0a13c579058562c9f44ceb3a6fc99c81d2e9f14a89dfa74d79d WatchSource:0}: Error finding container 8b3844f737bab0a13c579058562c9f44ceb3a6fc99c81d2e9f14a89dfa74d79d: Status 404 returned error can't find the container with id 8b3844f737bab0a13c579058562c9f44ceb3a6fc99c81d2e9f14a89dfa74d79d
Sep 12 00:51:47.189488 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:47.189445    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6" event={"ID":"57c84d1e-44c3-4942-a621-63fe6c439c0d","Type":"ContainerStarted","Data":"1c379e0e4f671e011d1ea4d23d10a22d5e509a58e35e6781fb0d1f5a211b8407"}
Sep 12 00:51:47.189727 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:47.189513    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6" event={"ID":"57c84d1e-44c3-4942-a621-63fe6c439c0d","Type":"ContainerStarted","Data":"8b3844f737bab0a13c579058562c9f44ceb3a6fc99c81d2e9f14a89dfa74d79d"}
Sep 12 00:51:47.189727 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:47.189593    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"
Sep 12 00:51:47.212635 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:47.212528    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6" podStartSLOduration=1.212510336 podStartE2EDuration="1.212510336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:51:47.210473006 +0000 UTC m=+1339.414473824" watchObservedRunningTime="2026-09-12 00:51:47.212510336 +0000 UTC m=+1339.416511158"
Sep 12 00:51:47.520461 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:47.520419    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-088c83-dockercfg-42xzl\""
Sep 12 00:51:47.527439 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:47.527420    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"
Sep 12 00:51:47.803727 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:47.803630    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"]
Sep 12 00:51:47.814692 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:51:47.814654    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d398c5_af9c_4f61_adbd_f5f0d945e00f.slice/crio-ded2ad2b1ab1cbdb36d1f0be71836d29bb306b0cc9ae5431597ea8e3d939994d WatchSource:0}: Error finding container ded2ad2b1ab1cbdb36d1f0be71836d29bb306b0cc9ae5431597ea8e3d939994d: Status 404 returned error can't find the container with id ded2ad2b1ab1cbdb36d1f0be71836d29bb306b0cc9ae5431597ea8e3d939994d
Sep 12 00:51:48.197101 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:48.196988    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l" event={"ID":"71d398c5-af9c-4f61-adbd-f5f0d945e00f","Type":"ContainerStarted","Data":"7f5284b8285959107914b74cf29f47c6dc2e7c2bd57c1db0ebf6a1e0e47d5194"}
Sep 12 00:51:48.197101 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:48.197056    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"
Sep 12 00:51:48.197101 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:48.197078    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l" event={"ID":"71d398c5-af9c-4f61-adbd-f5f0d945e00f","Type":"ContainerStarted","Data":"ded2ad2b1ab1cbdb36d1f0be71836d29bb306b0cc9ae5431597ea8e3d939994d"}
Sep 12 00:51:48.219792 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:48.219734    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l" podStartSLOduration=2.219712398 podStartE2EDuration="2.219712398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:51:48.218965463 +0000 UTC m=+1340.422966287" watchObservedRunningTime="2026-09-12 00:51:48.219712398 +0000 UTC m=+1340.423713223"
Sep 12 00:51:58.199468 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:58.199427    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"
Sep 12 00:51:59.206376 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:51:59.206339    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"
Sep 12 00:52:16.090005 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.089965    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"]
Sep 12 00:52:16.093426 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.093404    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"
Sep 12 00:52:16.096399 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.096370    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c3ca75-dockercfg-xvksh\""
Sep 12 00:52:16.096692 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.096674    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c3ca75\""
Sep 12 00:52:16.097770 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.097750    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"]
Sep 12 00:52:16.100986 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.100967    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"
Sep 12 00:52:16.113955 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.113927    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"]
Sep 12 00:52:16.119997 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.119888    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"]
Sep 12 00:52:16.186423 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.186376    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzmrn\" (UniqueName: \"kubernetes.io/projected/6b583281-a094-4abe-9a47-da7dea65f629-kube-api-access-hzmrn\") pod \"payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb\" (UID: \"6b583281-a094-4abe-9a47-da7dea65f629\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"
Sep 12 00:52:16.186664 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.186482    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d9321a3-4b8e-4f87-b165-4cb4000ec575-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql\" (UID: \"1d9321a3-4b8e-4f87-b165-4cb4000ec575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"
Sep 12 00:52:16.186664 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.186536    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b583281-a094-4abe-9a47-da7dea65f629-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb\" (UID: \"6b583281-a094-4abe-9a47-da7dea65f629\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"
Sep 12 00:52:16.186799 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.186667    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr6lh\" (UniqueName: \"kubernetes.io/projected/1d9321a3-4b8e-4f87-b165-4cb4000ec575-kube-api-access-vr6lh\") pod \"payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql\" (UID: \"1d9321a3-4b8e-4f87-b165-4cb4000ec575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"
Sep 12 00:52:16.288157 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.288119    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzmrn\" (UniqueName: \"kubernetes.io/projected/6b583281-a094-4abe-9a47-da7dea65f629-kube-api-access-hzmrn\") pod \"payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb\" (UID: \"6b583281-a094-4abe-9a47-da7dea65f629\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"
Sep 12 00:52:16.288346 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.288198    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d9321a3-4b8e-4f87-b165-4cb4000ec575-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql\" (UID: \"1d9321a3-4b8e-4f87-b165-4cb4000ec575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"
Sep 12 00:52:16.288346 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.288241    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b583281-a094-4abe-9a47-da7dea65f629-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb\" (UID: \"6b583281-a094-4abe-9a47-da7dea65f629\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"
Sep 12 00:52:16.288441 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.288372    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr6lh\" (UniqueName: \"kubernetes.io/projected/1d9321a3-4b8e-4f87-b165-4cb4000ec575-kube-api-access-vr6lh\") pod \"payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql\" (UID: \"1d9321a3-4b8e-4f87-b165-4cb4000ec575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"
Sep 12 00:52:16.289060 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.289040    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b583281-a094-4abe-9a47-da7dea65f629-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb\" (UID: \"6b583281-a094-4abe-9a47-da7dea65f629\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"
Sep 12 00:52:16.289181 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.289157    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d9321a3-4b8e-4f87-b165-4cb4000ec575-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql\" (UID: \"1d9321a3-4b8e-4f87-b165-4cb4000ec575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"
Sep 12 00:52:16.301617 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.301553    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr6lh\" (UniqueName: \"kubernetes.io/projected/1d9321a3-4b8e-4f87-b165-4cb4000ec575-kube-api-access-vr6lh\") pod \"payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql\" (UID: \"1d9321a3-4b8e-4f87-b165-4cb4000ec575\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"
Sep 12 00:52:16.301792 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.301634    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzmrn\" (UniqueName: \"kubernetes.io/projected/6b583281-a094-4abe-9a47-da7dea65f629-kube-api-access-hzmrn\") pod \"payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb\" (UID: \"6b583281-a094-4abe-9a47-da7dea65f629\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"
Sep 12 00:52:16.404599 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.404492    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"
Sep 12 00:52:16.413947 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.413918    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"
Sep 12 00:52:16.694760 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.694720    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"]
Sep 12 00:52:16.696588 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:52:16.696541    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9321a3_4b8e_4f87_b165_4cb4000ec575.slice/crio-3be40cc400b97e6ac598227f8d6963f217bad5c7f3bcbad007dc183b1666e478 WatchSource:0}: Error finding container 3be40cc400b97e6ac598227f8d6963f217bad5c7f3bcbad007dc183b1666e478: Status 404 returned error can't find the container with id 3be40cc400b97e6ac598227f8d6963f217bad5c7f3bcbad007dc183b1666e478
Sep 12 00:52:16.721246 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:16.721216    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"]
Sep 12 00:52:16.722911 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:52:16.722887    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b583281_a094_4abe_9a47_da7dea65f629.slice/crio-856e9c020085762d505e6c84646b5a6e38c7bef659740d354fd580b405bf1eef WatchSource:0}: Error finding container 856e9c020085762d505e6c84646b5a6e38c7bef659740d354fd580b405bf1eef: Status 404 returned error can't find the container with id 856e9c020085762d505e6c84646b5a6e38c7bef659740d354fd580b405bf1eef
Sep 12 00:52:17.360518 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:17.360468    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb" event={"ID":"6b583281-a094-4abe-9a47-da7dea65f629","Type":"ContainerStarted","Data":"495cbc9fadf7f638a55c0aa08458db3db804bec0ce90bb8a27688aa91bca687f"}
Sep 12 00:52:17.361041 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:17.360603    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb" event={"ID":"6b583281-a094-4abe-9a47-da7dea65f629","Type":"ContainerStarted","Data":"856e9c020085762d505e6c84646b5a6e38c7bef659740d354fd580b405bf1eef"}
Sep 12 00:52:17.361041 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:17.360637    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"
Sep 12 00:52:17.362221 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:17.362193    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql" event={"ID":"1d9321a3-4b8e-4f87-b165-4cb4000ec575","Type":"ContainerStarted","Data":"77e2cc695703a10814386d0c00529fe8c0c5cac5580b8c55a30cb8156dd27cdb"}
Sep 12 00:52:17.362352 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:17.362245    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql" event={"ID":"1d9321a3-4b8e-4f87-b165-4cb4000ec575","Type":"ContainerStarted","Data":"3be40cc400b97e6ac598227f8d6963f217bad5c7f3bcbad007dc183b1666e478"}
Sep 12 00:52:17.362352 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:17.362290    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"
Sep 12 00:52:17.482471 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:17.482411    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql" podStartSLOduration=1.482392658 podStartE2EDuration="1.482392658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:52:17.481992003 +0000 UTC m=+1369.685992828" watchObservedRunningTime="2026-09-12 00:52:17.482392658 +0000 UTC m=+1369.686393479"
Sep 12 00:52:17.483951 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:17.483909    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb" podStartSLOduration=1.483894457 podStartE2EDuration="1.483894457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:52:17.433447562 +0000 UTC m=+1369.637448383" watchObservedRunningTime="2026-09-12 00:52:17.483894457 +0000 UTC m=+1369.687895330"
Sep 12 00:52:28.370792 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:28.370753    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"
Sep 12 00:52:28.371235 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:28.370842    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"
Sep 12 00:52:45.023046 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:45.022997    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24"]
Sep 12 00:52:45.029246 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:45.029216    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24"
Sep 12 00:52:45.033862 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:45.033382    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:52:45.033862 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:45.033382    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:52:45.035689 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:45.035664    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24"]
Sep 12 00:52:45.179855 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:45.179805    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d52b15-4fd3-44cd-930b-5cb2a27774e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-81df64-88p24\" (UID: \"a3d52b15-4fd3-44cd-930b-5cb2a27774e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24"
Sep 12 00:52:45.281218 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:45.281114    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d52b15-4fd3-44cd-930b-5cb2a27774e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-81df64-88p24\" (UID: \"a3d52b15-4fd3-44cd-930b-5cb2a27774e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24"
Sep 12 00:52:45.281758 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:45.281740    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d52b15-4fd3-44cd-930b-5cb2a27774e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-81df64-88p24\" (UID: \"a3d52b15-4fd3-44cd-930b-5cb2a27774e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24"
Sep 12 00:52:45.341801 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:45.341761    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24"
Sep 12 00:52:45.619729 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:45.619702    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24"]
Sep 12 00:52:45.622916 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:52:45.622889    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d52b15_4fd3_44cd_930b_5cb2a27774e7.slice/crio-75d1267e4dc265cc5f009f8bcc35c2a19af37206ce8b265fae55b3ff5a30159c WatchSource:0}: Error finding container 75d1267e4dc265cc5f009f8bcc35c2a19af37206ce8b265fae55b3ff5a30159c: Status 404 returned error can't find the container with id 75d1267e4dc265cc5f009f8bcc35c2a19af37206ce8b265fae55b3ff5a30159c
Sep 12 00:52:46.519006 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:46.518967    2567 generic.go:356] "Generic (PLEG): container finished" podID="a3d52b15-4fd3-44cd-930b-5cb2a27774e7" containerID="9fa261da73b0e852bccdb01cd80f86f4c8ae416b1b95db28ddad6b8d4b232793" exitCode=0
Sep 12 00:52:46.519406 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:46.519056    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24" event={"ID":"a3d52b15-4fd3-44cd-930b-5cb2a27774e7","Type":"ContainerDied","Data":"9fa261da73b0e852bccdb01cd80f86f4c8ae416b1b95db28ddad6b8d4b232793"}
Sep 12 00:52:46.519406 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:46.519107    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24" event={"ID":"a3d52b15-4fd3-44cd-930b-5cb2a27774e7","Type":"ContainerStarted","Data":"75d1267e4dc265cc5f009f8bcc35c2a19af37206ce8b265fae55b3ff5a30159c"}
Sep 12 00:52:47.666149 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:47.666118    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24"
Sep 12 00:52:47.807491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:47.807387    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d52b15-4fd3-44cd-930b-5cb2a27774e7-maas-api-service-ca\") pod \"a3d52b15-4fd3-44cd-930b-5cb2a27774e7\" (UID: \"a3d52b15-4fd3-44cd-930b-5cb2a27774e7\") "
Sep 12 00:52:47.807882 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:47.807850    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d52b15-4fd3-44cd-930b-5cb2a27774e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3d52b15-4fd3-44cd-930b-5cb2a27774e7" (UID: "a3d52b15-4fd3-44cd-930b-5cb2a27774e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:52:47.909077 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:47.909038    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3d52b15-4fd3-44cd-930b-5cb2a27774e7-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:52:48.531728 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:48.531629    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24"
Sep 12 00:52:48.531728 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:48.531632    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24" event={"ID":"a3d52b15-4fd3-44cd-930b-5cb2a27774e7","Type":"ContainerDied","Data":"75d1267e4dc265cc5f009f8bcc35c2a19af37206ce8b265fae55b3ff5a30159c"}
Sep 12 00:52:48.531920 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:48.531773    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75d1267e4dc265cc5f009f8bcc35c2a19af37206ce8b265fae55b3ff5a30159c"
Sep 12 00:52:56.556288 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:56.556246    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24"]
Sep 12 00:52:56.562716 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:56.562675    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-81df64-88p24"]
Sep 12 00:52:57.349070 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.349035    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"]
Sep 12 00:52:57.349454 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.349432    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d52b15-4fd3-44cd-930b-5cb2a27774e7" containerName="revoke-keys"
Sep 12 00:52:57.349454 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.349445    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d52b15-4fd3-44cd-930b-5cb2a27774e7" containerName="revoke-keys"
Sep 12 00:52:57.349547 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.349537    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d52b15-4fd3-44cd-930b-5cb2a27774e7" containerName="revoke-keys"
Sep 12 00:52:57.352912 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.352889    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"
Sep 12 00:52:57.355693 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.355666    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5306e6ca\""
Sep 12 00:52:57.355811 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.355740    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5306e6ca-dockercfg-ccsj6\""
Sep 12 00:52:57.364814 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.364788    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"]
Sep 12 00:52:57.510466 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.510422    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1522dc9d-0dbc-4830-b20e-e8eeb82108af-plugins-config-volume\") pod \"payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr\" (UID: \"1522dc9d-0dbc-4830-b20e-e8eeb82108af\") " pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"
Sep 12 00:52:57.510699 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.510479    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8xx\" (UniqueName: \"kubernetes.io/projected/1522dc9d-0dbc-4830-b20e-e8eeb82108af-kube-api-access-cs8xx\") pod \"payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr\" (UID: \"1522dc9d-0dbc-4830-b20e-e8eeb82108af\") " pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"
Sep 12 00:52:57.611621 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.611486    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1522dc9d-0dbc-4830-b20e-e8eeb82108af-plugins-config-volume\") pod \"payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr\" (UID: \"1522dc9d-0dbc-4830-b20e-e8eeb82108af\") " pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"
Sep 12 00:52:57.611621 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.611540    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8xx\" (UniqueName: \"kubernetes.io/projected/1522dc9d-0dbc-4830-b20e-e8eeb82108af-kube-api-access-cs8xx\") pod \"payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr\" (UID: \"1522dc9d-0dbc-4830-b20e-e8eeb82108af\") " pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"
Sep 12 00:52:57.612352 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.612323    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1522dc9d-0dbc-4830-b20e-e8eeb82108af-plugins-config-volume\") pod \"payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr\" (UID: \"1522dc9d-0dbc-4830-b20e-e8eeb82108af\") " pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"
Sep 12 00:52:57.621812 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.621779    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8xx\" (UniqueName: \"kubernetes.io/projected/1522dc9d-0dbc-4830-b20e-e8eeb82108af-kube-api-access-cs8xx\") pod \"payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr\" (UID: \"1522dc9d-0dbc-4830-b20e-e8eeb82108af\") " pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"
Sep 12 00:52:57.664465 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.664426    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"
Sep 12 00:52:57.943958 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:57.943930    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"]
Sep 12 00:52:57.946183 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:52:57.946156    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1522dc9d_0dbc_4830_b20e_e8eeb82108af.slice/crio-1ea1b263d6468cf68e662695d885e57b4a4385015c1b7b5a7b6c8b92f8d07036 WatchSource:0}: Error finding container 1ea1b263d6468cf68e662695d885e57b4a4385015c1b7b5a7b6c8b92f8d07036: Status 404 returned error can't find the container with id 1ea1b263d6468cf68e662695d885e57b4a4385015c1b7b5a7b6c8b92f8d07036
Sep 12 00:52:58.274442 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:58.274408    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3d52b15-4fd3-44cd-930b-5cb2a27774e7" path="/var/lib/kubelet/pods/a3d52b15-4fd3-44cd-930b-5cb2a27774e7/volumes"
Sep 12 00:52:58.595257 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:58.595148    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr" event={"ID":"1522dc9d-0dbc-4830-b20e-e8eeb82108af","Type":"ContainerStarted","Data":"eeb808315412392fbd5a87e4019e219acb8acceae4e22a2441bc0d3114068ae7"}
Sep 12 00:52:58.595257 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:58.595214    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"
Sep 12 00:52:58.595257 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:58.595242    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr" event={"ID":"1522dc9d-0dbc-4830-b20e-e8eeb82108af","Type":"ContainerStarted","Data":"1ea1b263d6468cf68e662695d885e57b4a4385015c1b7b5a7b6c8b92f8d07036"}
Sep 12 00:52:58.617592 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:52:58.615194    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr" podStartSLOduration=1.615171429 podStartE2EDuration="1.615171429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:52:58.611772192 +0000 UTC m=+1410.815773015" watchObservedRunningTime="2026-09-12 00:52:58.615171429 +0000 UTC m=+1410.819172252"
Sep 12 00:53:03.757627 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:03.757589    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs"]
Sep 12 00:53:03.761315 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:03.761293    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs"
Sep 12 00:53:03.763903 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:03.763874    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:53:03.763903 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:03.763875    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:53:03.769250 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:03.769204    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3d1538-374f-4ec7-9b87-ff9de4789d90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs\" (UID: \"ee3d1538-374f-4ec7-9b87-ff9de4789d90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs"
Sep 12 00:53:03.770285 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:03.770247    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs"]
Sep 12 00:53:03.870270 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:03.870227    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3d1538-374f-4ec7-9b87-ff9de4789d90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs\" (UID: \"ee3d1538-374f-4ec7-9b87-ff9de4789d90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs"
Sep 12 00:53:03.870893 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:03.870874    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3d1538-374f-4ec7-9b87-ff9de4789d90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs\" (UID: \"ee3d1538-374f-4ec7-9b87-ff9de4789d90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs"
Sep 12 00:53:04.075111 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:04.075017    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs"
Sep 12 00:53:04.348907 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:04.348778    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs"]
Sep 12 00:53:04.351470 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:53:04.351437    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3d1538_374f_4ec7_9b87_ff9de4789d90.slice/crio-59ef3749d1b04dbce9fd33ac6c5a045205803fc67214ba0ce0778e62e2c528c7 WatchSource:0}: Error finding container 59ef3749d1b04dbce9fd33ac6c5a045205803fc67214ba0ce0778e62e2c528c7: Status 404 returned error can't find the container with id 59ef3749d1b04dbce9fd33ac6c5a045205803fc67214ba0ce0778e62e2c528c7
Sep 12 00:53:04.635879 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:04.635785    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs" event={"ID":"ee3d1538-374f-4ec7-9b87-ff9de4789d90","Type":"ContainerStarted","Data":"59ef3749d1b04dbce9fd33ac6c5a045205803fc67214ba0ce0778e62e2c528c7"}
Sep 12 00:53:05.643644 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:05.643604    2567 generic.go:356] "Generic (PLEG): container finished" podID="ee3d1538-374f-4ec7-9b87-ff9de4789d90" containerID="f69769360674f688013150b703638a1648bf5956993e93f82e2835b6830b9a94" exitCode=0
Sep 12 00:53:05.644067 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:05.643686    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs" event={"ID":"ee3d1538-374f-4ec7-9b87-ff9de4789d90","Type":"ContainerDied","Data":"f69769360674f688013150b703638a1648bf5956993e93f82e2835b6830b9a94"}
Sep 12 00:53:06.782152 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:06.782122    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs"
Sep 12 00:53:06.803080 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:06.803044    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3d1538-374f-4ec7-9b87-ff9de4789d90-maas-api-service-ca\") pod \"ee3d1538-374f-4ec7-9b87-ff9de4789d90\" (UID: \"ee3d1538-374f-4ec7-9b87-ff9de4789d90\") "
Sep 12 00:53:06.803466 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:06.803439    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3d1538-374f-4ec7-9b87-ff9de4789d90-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ee3d1538-374f-4ec7-9b87-ff9de4789d90" (UID: "ee3d1538-374f-4ec7-9b87-ff9de4789d90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:53:06.904444 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:06.904355    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3d1538-374f-4ec7-9b87-ff9de4789d90-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:53:07.655820 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:07.655784    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs"
Sep 12 00:53:07.656008 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:07.655795    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs" event={"ID":"ee3d1538-374f-4ec7-9b87-ff9de4789d90","Type":"ContainerDied","Data":"59ef3749d1b04dbce9fd33ac6c5a045205803fc67214ba0ce0778e62e2c528c7"}
Sep 12 00:53:07.656008 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:07.655913    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59ef3749d1b04dbce9fd33ac6c5a045205803fc67214ba0ce0778e62e2c528c7"
Sep 12 00:53:09.603474 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:09.603438    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"
Sep 12 00:53:14.543625 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:14.543563    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs"]
Sep 12 00:53:14.546634 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:14.546604    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb6e74-4ttgs"]
Sep 12 00:53:16.274470 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:16.274428    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee3d1538-374f-4ec7-9b87-ff9de4789d90" path="/var/lib/kubelet/pods/ee3d1538-374f-4ec7-9b87-ff9de4789d90/volumes"
Sep 12 00:53:28.772863 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:28.772828    2567 scope.go:117] "RemoveContainer" containerID="da6558c2eceddff0807d5d685dd2a3f31fd2e7f81d0183174bcd886ac7ea1bca"
Sep 12 00:53:28.782020 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:28.781985    2567 scope.go:117] "RemoveContainer" containerID="c8545d6d9a0b1eba863721efb5b8b04c5734af0558ab39cde917f587f5fe3064"
Sep 12 00:53:56.993024 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:56.992983    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"]
Sep 12 00:53:56.993663 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:53:56.993237    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br" podUID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerName="istio-proxy" containerID="cri-o://ea6741d5cd678b1c8387db25439ad85c9661f6731ab7a4c28394ce1f563f298c" gracePeriod=30
Sep 12 00:54:02.253067 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.253035    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:54:02.284110 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284030    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-envoy\") pod \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") "
Sep 12 00:54:02.284110 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284063    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmc5q\" (UniqueName: \"kubernetes.io/projected/d5c5e730-811c-4c4c-9f65-c907f374fd2e-kube-api-access-wmc5q\") pod \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") "
Sep 12 00:54:02.284110 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284086    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-credential-socket\") pod \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") "
Sep 12 00:54:02.284402 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284114    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-workload-socket\") pod \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") "
Sep 12 00:54:02.284402 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284146    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-podinfo\") pod \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") "
Sep 12 00:54:02.284402 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284166    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-data\") pod \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") "
Sep 12 00:54:02.284402 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284199    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-workload-certs\") pod \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") "
Sep 12 00:54:02.284402 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284251    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-token\") pod \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") "
Sep 12 00:54:02.284402 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284276    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istiod-ca-cert\") pod \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\" (UID: \"d5c5e730-811c-4c4c-9f65-c907f374fd2e\") "
Sep 12 00:54:02.284751 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284422    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5c5e730-811c-4c4c-9f65-c907f374fd2e" (UID: "d5c5e730-811c-4c4c-9f65-c907f374fd2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:54:02.284751 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284442    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5c5e730-811c-4c4c-9f65-c907f374fd2e" (UID: "d5c5e730-811c-4c4c-9f65-c907f374fd2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:54:02.284751 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284721    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5c5e730-811c-4c4c-9f65-c907f374fd2e" (UID: "d5c5e730-811c-4c4c-9f65-c907f374fd2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:54:02.284919 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284739    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5c5e730-811c-4c4c-9f65-c907f374fd2e" (UID: "d5c5e730-811c-4c4c-9f65-c907f374fd2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:54:02.284919 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284851    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-workload-socket\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:54:02.284919 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284875    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-credential-socket\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:54:02.285086 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.284971    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5c5e730-811c-4c4c-9f65-c907f374fd2e" (UID: "d5c5e730-811c-4c4c-9f65-c907f374fd2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:54:02.286862 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.286831    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5c5e730-811c-4c4c-9f65-c907f374fd2e" (UID: "d5c5e730-811c-4c4c-9f65-c907f374fd2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 00:54:02.286976 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.286845    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c5e730-811c-4c4c-9f65-c907f374fd2e-kube-api-access-wmc5q" (OuterVolumeSpecName: "kube-api-access-wmc5q") pod "d5c5e730-811c-4c4c-9f65-c907f374fd2e" (UID: "d5c5e730-811c-4c4c-9f65-c907f374fd2e"). InnerVolumeSpecName "kube-api-access-wmc5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:54:02.287211 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.287192    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5c5e730-811c-4c4c-9f65-c907f374fd2e" (UID: "d5c5e730-811c-4c4c-9f65-c907f374fd2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:54:02.287257 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.287194    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5c5e730-811c-4c4c-9f65-c907f374fd2e" (UID: "d5c5e730-811c-4c4c-9f65-c907f374fd2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:54:02.385408 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.385371    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-envoy\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:54:02.385408 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.385406    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmc5q\" (UniqueName: \"kubernetes.io/projected/d5c5e730-811c-4c4c-9f65-c907f374fd2e-kube-api-access-wmc5q\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:54:02.385408 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.385417    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-podinfo\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:54:02.385811 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.385426    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-data\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:54:02.385811 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.385435    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e730-811c-4c4c-9f65-c907f374fd2e-workload-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:54:02.385811 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.385443    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istio-token\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:54:02.385811 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.385450    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c5e730-811c-4c4c-9f65-c907f374fd2e-istiod-ca-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:54:02.966746 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.966708    2567 generic.go:356] "Generic (PLEG): container finished" podID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerID="ea6741d5cd678b1c8387db25439ad85c9661f6731ab7a4c28394ce1f563f298c" exitCode=0
Sep 12 00:54:02.966968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.966760    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br" event={"ID":"d5c5e730-811c-4c4c-9f65-c907f374fd2e","Type":"ContainerDied","Data":"ea6741d5cd678b1c8387db25439ad85c9661f6731ab7a4c28394ce1f563f298c"}
Sep 12 00:54:02.966968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.966778    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"
Sep 12 00:54:02.966968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.966805    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br" event={"ID":"d5c5e730-811c-4c4c-9f65-c907f374fd2e","Type":"ContainerDied","Data":"89b6293dcd63f2e73dec42c98100935a464afd0d04c08faac319c245ebcf765d"}
Sep 12 00:54:02.966968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.966825    2567 scope.go:117] "RemoveContainer" containerID="ea6741d5cd678b1c8387db25439ad85c9661f6731ab7a4c28394ce1f563f298c"
Sep 12 00:54:02.977122 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.977100    2567 scope.go:117] "RemoveContainer" containerID="330fd9f149bbe6c1d2d5218c5050db88e183d69b45cbc6e9d485163bfe26f533"
Sep 12 00:54:02.986386 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.986365    2567 scope.go:117] "RemoveContainer" containerID="ea6741d5cd678b1c8387db25439ad85c9661f6731ab7a4c28394ce1f563f298c"
Sep 12 00:54:02.986706 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:54:02.986682    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6741d5cd678b1c8387db25439ad85c9661f6731ab7a4c28394ce1f563f298c\": container with ID starting with ea6741d5cd678b1c8387db25439ad85c9661f6731ab7a4c28394ce1f563f298c not found: ID does not exist" containerID="ea6741d5cd678b1c8387db25439ad85c9661f6731ab7a4c28394ce1f563f298c"
Sep 12 00:54:02.986762 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.986716    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6741d5cd678b1c8387db25439ad85c9661f6731ab7a4c28394ce1f563f298c"} err="failed to get container status \"ea6741d5cd678b1c8387db25439ad85c9661f6731ab7a4c28394ce1f563f298c\": rpc error: code = NotFound desc = could not find container \"ea6741d5cd678b1c8387db25439ad85c9661f6731ab7a4c28394ce1f563f298c\": container with ID starting with ea6741d5cd678b1c8387db25439ad85c9661f6731ab7a4c28394ce1f563f298c not found: ID does not exist"
Sep 12 00:54:02.986762 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.986734    2567 scope.go:117] "RemoveContainer" containerID="330fd9f149bbe6c1d2d5218c5050db88e183d69b45cbc6e9d485163bfe26f533"
Sep 12 00:54:02.987015 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:54:02.986997    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330fd9f149bbe6c1d2d5218c5050db88e183d69b45cbc6e9d485163bfe26f533\": container with ID starting with 330fd9f149bbe6c1d2d5218c5050db88e183d69b45cbc6e9d485163bfe26f533 not found: ID does not exist" containerID="330fd9f149bbe6c1d2d5218c5050db88e183d69b45cbc6e9d485163bfe26f533"
Sep 12 00:54:02.987068 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.987019    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330fd9f149bbe6c1d2d5218c5050db88e183d69b45cbc6e9d485163bfe26f533"} err="failed to get container status \"330fd9f149bbe6c1d2d5218c5050db88e183d69b45cbc6e9d485163bfe26f533\": rpc error: code = NotFound desc = could not find container \"330fd9f149bbe6c1d2d5218c5050db88e183d69b45cbc6e9d485163bfe26f533\": container with ID starting with 330fd9f149bbe6c1d2d5218c5050db88e183d69b45cbc6e9d485163bfe26f533 not found: ID does not exist"
Sep 12 00:54:02.991102 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.991081    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"]
Sep 12 00:54:02.995946 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:02.995922    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bb6e74-openshift-default-68cb6df57b-nh7br"]
Sep 12 00:54:04.273979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:04.273946    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" path="/var/lib/kubelet/pods/d5c5e730-811c-4c4c-9f65-c907f374fd2e/volumes"
Sep 12 00:54:28.807378 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:28.807338    2567 scope.go:117] "RemoveContainer" containerID="1654f145feb6e5f0ab5da619b8fec3762960af12eea7f14c4050c12d8351045d"
Sep 12 00:54:28.816180 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:28.816158    2567 scope.go:117] "RemoveContainer" containerID="229ffa44dd46261c5498b7fe061c0556272f951b7f6b07924509de44cc126d96"
Sep 12 00:54:28.825017 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:28.824993    2567 scope.go:117] "RemoveContainer" containerID="925b6505e1c4e4e8f49c2e2bd9d421d3188abb8402b81dc0c3bc09fe915e3ad9"
Sep 12 00:54:39.712369 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.712329    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"]
Sep 12 00:54:39.712836 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.712781    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerName="istio-proxy"
Sep 12 00:54:39.712836 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.712794    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerName="istio-proxy"
Sep 12 00:54:39.712836 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.712806    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee3d1538-374f-4ec7-9b87-ff9de4789d90" containerName="revoke-keys"
Sep 12 00:54:39.712836 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.712812    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee3d1538-374f-4ec7-9b87-ff9de4789d90" containerName="revoke-keys"
Sep 12 00:54:39.712968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.712872    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerName="istio-proxy"
Sep 12 00:54:39.712968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.712883    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerName="istio-proxy"
Sep 12 00:54:39.712968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.712889    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee3d1538-374f-4ec7-9b87-ff9de4789d90" containerName="revoke-keys"
Sep 12 00:54:39.717421 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.717395    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"
Sep 12 00:54:39.723827 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.723792    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-dd25dd-dockercfg-5bsk4\""
Sep 12 00:54:39.724046 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.724028    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-dd25dd\""
Sep 12 00:54:39.732178 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.732144    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"]
Sep 12 00:54:39.862029 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.861983    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d63161a-12bb-4a88-8a2a-840a4cad8eb3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4\" (UID: \"6d63161a-12bb-4a88-8a2a-840a4cad8eb3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"
Sep 12 00:54:39.862224 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.862067    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5gl\" (UniqueName: \"kubernetes.io/projected/6d63161a-12bb-4a88-8a2a-840a4cad8eb3-kube-api-access-4h5gl\") pod \"payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4\" (UID: \"6d63161a-12bb-4a88-8a2a-840a4cad8eb3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"
Sep 12 00:54:39.963679 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.963560    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5gl\" (UniqueName: \"kubernetes.io/projected/6d63161a-12bb-4a88-8a2a-840a4cad8eb3-kube-api-access-4h5gl\") pod \"payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4\" (UID: \"6d63161a-12bb-4a88-8a2a-840a4cad8eb3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"
Sep 12 00:54:39.963679 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.963678    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d63161a-12bb-4a88-8a2a-840a4cad8eb3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4\" (UID: \"6d63161a-12bb-4a88-8a2a-840a4cad8eb3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"
Sep 12 00:54:39.964366 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.964344    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d63161a-12bb-4a88-8a2a-840a4cad8eb3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4\" (UID: \"6d63161a-12bb-4a88-8a2a-840a4cad8eb3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"
Sep 12 00:54:39.971624 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:39.971596    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5gl\" (UniqueName: \"kubernetes.io/projected/6d63161a-12bb-4a88-8a2a-840a4cad8eb3-kube-api-access-4h5gl\") pod \"payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4\" (UID: \"6d63161a-12bb-4a88-8a2a-840a4cad8eb3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"
Sep 12 00:54:40.028227 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:40.028185    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"
Sep 12 00:54:40.300725 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:40.300586    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"]
Sep 12 00:54:40.303116 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:54:40.303079    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d63161a_12bb_4a88_8a2a_840a4cad8eb3.slice/crio-26489d81552041ea66419d48568d8bc3dab02f389235fde3d085ef3ebe7dd725 WatchSource:0}: Error finding container 26489d81552041ea66419d48568d8bc3dab02f389235fde3d085ef3ebe7dd725: Status 404 returned error can't find the container with id 26489d81552041ea66419d48568d8bc3dab02f389235fde3d085ef3ebe7dd725
Sep 12 00:54:41.176999 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:41.176961    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4" event={"ID":"6d63161a-12bb-4a88-8a2a-840a4cad8eb3","Type":"ContainerStarted","Data":"975b76c35043f9c889d09efef9af2494a75a6978a64a306e49ba0d58d11c87ef"}
Sep 12 00:54:41.177418 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:41.177017    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4" event={"ID":"6d63161a-12bb-4a88-8a2a-840a4cad8eb3","Type":"ContainerStarted","Data":"26489d81552041ea66419d48568d8bc3dab02f389235fde3d085ef3ebe7dd725"}
Sep 12 00:54:41.177418 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:41.177162    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"
Sep 12 00:54:41.194767 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:41.194708    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4" podStartSLOduration=2.194691764 podStartE2EDuration="2.194691764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:54:41.193960554 +0000 UTC m=+1513.397961376" watchObservedRunningTime="2026-09-12 00:54:41.194691764 +0000 UTC m=+1513.398692586"
Sep 12 00:54:52.184851 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:54:52.184755    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"
Sep 12 00:55:32.811274 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:32.811235    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"]
Sep 12 00:55:33.831840 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:33.831807    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs"]
Sep 12 00:55:33.832328 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:33.832240    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerName="istio-proxy"
Sep 12 00:55:33.832328 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:33.832253    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c5e730-811c-4c4c-9f65-c907f374fd2e" containerName="istio-proxy"
Sep 12 00:55:33.842315 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:33.842280    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs"
Sep 12 00:55:33.844807 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:33.844776    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs"]
Sep 12 00:55:33.845312 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:33.845285    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:55:33.845312 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:33.845300    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:55:33.897070 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:33.897026    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d72b0a-be7d-451e-bd78-31f51cbf41b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs\" (UID: \"92d72b0a-be7d-451e-bd78-31f51cbf41b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs"
Sep 12 00:55:33.997793 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:33.997750    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d72b0a-be7d-451e-bd78-31f51cbf41b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs\" (UID: \"92d72b0a-be7d-451e-bd78-31f51cbf41b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs"
Sep 12 00:55:33.998405 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:33.998383    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d72b0a-be7d-451e-bd78-31f51cbf41b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs\" (UID: \"92d72b0a-be7d-451e-bd78-31f51cbf41b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs"
Sep 12 00:55:34.155499 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:34.155404    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs"
Sep 12 00:55:34.438380 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:55:34.438347    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d72b0a_be7d_451e_bd78_31f51cbf41b4.slice/crio-2126556a4caabda0c0fcbbc1cd5c00498acf1e68b01ec137dd82e17f138c7081 WatchSource:0}: Error finding container 2126556a4caabda0c0fcbbc1cd5c00498acf1e68b01ec137dd82e17f138c7081: Status 404 returned error can't find the container with id 2126556a4caabda0c0fcbbc1cd5c00498acf1e68b01ec137dd82e17f138c7081
Sep 12 00:55:34.439923 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:34.439718    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs"]
Sep 12 00:55:34.454880 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:34.454847    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs" event={"ID":"92d72b0a-be7d-451e-bd78-31f51cbf41b4","Type":"ContainerStarted","Data":"2126556a4caabda0c0fcbbc1cd5c00498acf1e68b01ec137dd82e17f138c7081"}
Sep 12 00:55:35.462983 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:35.462948    2567 generic.go:356] "Generic (PLEG): container finished" podID="92d72b0a-be7d-451e-bd78-31f51cbf41b4" containerID="d146b7608cb601c3a89e011186c50e7702f500a2bbafb290e5e52cd81c07384e" exitCode=0
Sep 12 00:55:35.463384 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:35.463038    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs" event={"ID":"92d72b0a-be7d-451e-bd78-31f51cbf41b4","Type":"ContainerDied","Data":"d146b7608cb601c3a89e011186c50e7702f500a2bbafb290e5e52cd81c07384e"}
Sep 12 00:55:36.607683 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:36.607653    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs"
Sep 12 00:55:36.725932 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:36.725894    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d72b0a-be7d-451e-bd78-31f51cbf41b4-maas-api-service-ca\") pod \"92d72b0a-be7d-451e-bd78-31f51cbf41b4\" (UID: \"92d72b0a-be7d-451e-bd78-31f51cbf41b4\") "
Sep 12 00:55:36.726299 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:36.726275    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d72b0a-be7d-451e-bd78-31f51cbf41b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92d72b0a-be7d-451e-bd78-31f51cbf41b4" (UID: "92d72b0a-be7d-451e-bd78-31f51cbf41b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:55:36.827092 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:36.827050    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d72b0a-be7d-451e-bd78-31f51cbf41b4-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:55:37.475185 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:37.475152    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs"
Sep 12 00:55:37.475386 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:37.475187    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs" event={"ID":"92d72b0a-be7d-451e-bd78-31f51cbf41b4","Type":"ContainerDied","Data":"2126556a4caabda0c0fcbbc1cd5c00498acf1e68b01ec137dd82e17f138c7081"}
Sep 12 00:55:37.475386 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:37.475235    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2126556a4caabda0c0fcbbc1cd5c00498acf1e68b01ec137dd82e17f138c7081"
Sep 12 00:55:47.830804 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:47.830748    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh" podUID="65f31f55-918f-4cf2-99a8-6e8cc1f76f1d" containerName="main" containerID="cri-o://874a552c7d2edaf3ee7e8bb4e4ad7e4a3e1d272d7ed3fd0aefddd0aefed8dc28" gracePeriod=45
Sep 12 00:55:48.086596 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.086513    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:55:48.243059 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.243023    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-tmp-dir\") pod \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") "
Sep 12 00:55:48.243264 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.243086    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-home\") pod \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") "
Sep 12 00:55:48.243264 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.243121    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnbbj\" (UniqueName: \"kubernetes.io/projected/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-kube-api-access-tnbbj\") pod \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") "
Sep 12 00:55:48.243264 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.243153    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-dshm\") pod \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") "
Sep 12 00:55:48.243264 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.243221    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65f31f55-918f-4cf2-99a8-6e8cc1f76f1d" (UID: "65f31f55-918f-4cf2-99a8-6e8cc1f76f1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:55:48.243264 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.243230    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-tls-certs\") pod \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") "
Sep 12 00:55:48.243554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.243303    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-model-cache\") pod \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\" (UID: \"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d\") "
Sep 12 00:55:48.243554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.243359    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-home" (OuterVolumeSpecName: "home") pod "65f31f55-918f-4cf2-99a8-6e8cc1f76f1d" (UID: "65f31f55-918f-4cf2-99a8-6e8cc1f76f1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:55:48.243554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.243544    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-model-cache" (OuterVolumeSpecName: "model-cache") pod "65f31f55-918f-4cf2-99a8-6e8cc1f76f1d" (UID: "65f31f55-918f-4cf2-99a8-6e8cc1f76f1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:55:48.243727 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.243686    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-tmp-dir\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:55:48.243727 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.243707    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-home\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:55:48.243727 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.243719    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-model-cache\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:55:48.245609 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.245564    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65f31f55-918f-4cf2-99a8-6e8cc1f76f1d" (UID: "65f31f55-918f-4cf2-99a8-6e8cc1f76f1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:55:48.245776 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.245749    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-dshm" (OuterVolumeSpecName: "dshm") pod "65f31f55-918f-4cf2-99a8-6e8cc1f76f1d" (UID: "65f31f55-918f-4cf2-99a8-6e8cc1f76f1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:55:48.245899 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.245757    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-kube-api-access-tnbbj" (OuterVolumeSpecName: "kube-api-access-tnbbj") pod "65f31f55-918f-4cf2-99a8-6e8cc1f76f1d" (UID: "65f31f55-918f-4cf2-99a8-6e8cc1f76f1d"). InnerVolumeSpecName "kube-api-access-tnbbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:55:48.344484 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.344398    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-tls-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:55:48.344484 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.344431    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnbbj\" (UniqueName: \"kubernetes.io/projected/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-kube-api-access-tnbbj\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:55:48.344484 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.344441    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d-dshm\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:55:48.541287 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.541251    2567 generic.go:356] "Generic (PLEG): container finished" podID="65f31f55-918f-4cf2-99a8-6e8cc1f76f1d" containerID="874a552c7d2edaf3ee7e8bb4e4ad7e4a3e1d272d7ed3fd0aefddd0aefed8dc28" exitCode=0
Sep 12 00:55:48.541492 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.541294    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"
Sep 12 00:55:48.541492 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.541341    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh" event={"ID":"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d","Type":"ContainerDied","Data":"874a552c7d2edaf3ee7e8bb4e4ad7e4a3e1d272d7ed3fd0aefddd0aefed8dc28"}
Sep 12 00:55:48.541492 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.541392    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh" event={"ID":"65f31f55-918f-4cf2-99a8-6e8cc1f76f1d","Type":"ContainerDied","Data":"fc9361dc017df344433138c319e6a5c02dcc0996d8bad84f2da7bc96ce571b3f"}
Sep 12 00:55:48.541492 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.541416    2567 scope.go:117] "RemoveContainer" containerID="874a552c7d2edaf3ee7e8bb4e4ad7e4a3e1d272d7ed3fd0aefddd0aefed8dc28"
Sep 12 00:55:48.550144 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.550127    2567 scope.go:117] "RemoveContainer" containerID="874a552c7d2edaf3ee7e8bb4e4ad7e4a3e1d272d7ed3fd0aefddd0aefed8dc28"
Sep 12 00:55:48.550406 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:55:48.550387    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874a552c7d2edaf3ee7e8bb4e4ad7e4a3e1d272d7ed3fd0aefddd0aefed8dc28\": container with ID starting with 874a552c7d2edaf3ee7e8bb4e4ad7e4a3e1d272d7ed3fd0aefddd0aefed8dc28 not found: ID does not exist" containerID="874a552c7d2edaf3ee7e8bb4e4ad7e4a3e1d272d7ed3fd0aefddd0aefed8dc28"
Sep 12 00:55:48.550476 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.550419    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874a552c7d2edaf3ee7e8bb4e4ad7e4a3e1d272d7ed3fd0aefddd0aefed8dc28"} err="failed to get container status \"874a552c7d2edaf3ee7e8bb4e4ad7e4a3e1d272d7ed3fd0aefddd0aefed8dc28\": rpc error: code = NotFound desc = could not find container \"874a552c7d2edaf3ee7e8bb4e4ad7e4a3e1d272d7ed3fd0aefddd0aefed8dc28\": container with ID starting with 874a552c7d2edaf3ee7e8bb4e4ad7e4a3e1d272d7ed3fd0aefddd0aefed8dc28 not found: ID does not exist"
Sep 12 00:55:48.557623 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.557599    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"]
Sep 12 00:55:48.561850 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:48.561828    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-876808/e2e-conflict-a-f937eb5b-kserve-784b9468c6-4l9vh"]
Sep 12 00:55:49.372756 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:49.372721    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"]
Sep 12 00:55:49.373237 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:49.372999    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr" podUID="1522dc9d-0dbc-4830-b20e-e8eeb82108af" containerName="payload-processing" containerID="cri-o://eeb808315412392fbd5a87e4019e219acb8acceae4e22a2441bc0d3114068ae7" gracePeriod=30
Sep 12 00:55:49.550876 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:49.550839    2567 generic.go:356] "Generic (PLEG): container finished" podID="1522dc9d-0dbc-4830-b20e-e8eeb82108af" containerID="eeb808315412392fbd5a87e4019e219acb8acceae4e22a2441bc0d3114068ae7" exitCode=0
Sep 12 00:55:49.551028 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:49.550908    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr" event={"ID":"1522dc9d-0dbc-4830-b20e-e8eeb82108af","Type":"ContainerDied","Data":"eeb808315412392fbd5a87e4019e219acb8acceae4e22a2441bc0d3114068ae7"}
Sep 12 00:55:49.573725 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:49.573690    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs"]
Sep 12 00:55:49.576915 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:49.576880    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5306e6ca-hkhbs"]
Sep 12 00:55:49.638735 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:49.638616    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"
Sep 12 00:55:49.759450 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:49.759417    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1522dc9d-0dbc-4830-b20e-e8eeb82108af-plugins-config-volume\") pod \"1522dc9d-0dbc-4830-b20e-e8eeb82108af\" (UID: \"1522dc9d-0dbc-4830-b20e-e8eeb82108af\") "
Sep 12 00:55:49.759653 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:49.759469    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs8xx\" (UniqueName: \"kubernetes.io/projected/1522dc9d-0dbc-4830-b20e-e8eeb82108af-kube-api-access-cs8xx\") pod \"1522dc9d-0dbc-4830-b20e-e8eeb82108af\" (UID: \"1522dc9d-0dbc-4830-b20e-e8eeb82108af\") "
Sep 12 00:55:49.759919 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:49.759889    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1522dc9d-0dbc-4830-b20e-e8eeb82108af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1522dc9d-0dbc-4830-b20e-e8eeb82108af" (UID: "1522dc9d-0dbc-4830-b20e-e8eeb82108af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:55:49.761647 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:49.761621    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1522dc9d-0dbc-4830-b20e-e8eeb82108af-kube-api-access-cs8xx" (OuterVolumeSpecName: "kube-api-access-cs8xx") pod "1522dc9d-0dbc-4830-b20e-e8eeb82108af" (UID: "1522dc9d-0dbc-4830-b20e-e8eeb82108af"). InnerVolumeSpecName "kube-api-access-cs8xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:55:49.860842 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:49.860800    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1522dc9d-0dbc-4830-b20e-e8eeb82108af-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:55:49.860842 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:49.860832    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs8xx\" (UniqueName: \"kubernetes.io/projected/1522dc9d-0dbc-4830-b20e-e8eeb82108af-kube-api-access-cs8xx\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:55:50.274476 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:50.274440    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f31f55-918f-4cf2-99a8-6e8cc1f76f1d" path="/var/lib/kubelet/pods/65f31f55-918f-4cf2-99a8-6e8cc1f76f1d/volumes"
Sep 12 00:55:50.274831 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:50.274816    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92d72b0a-be7d-451e-bd78-31f51cbf41b4" path="/var/lib/kubelet/pods/92d72b0a-be7d-451e-bd78-31f51cbf41b4/volumes"
Sep 12 00:55:50.558335 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:50.558252    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"
Sep 12 00:55:50.558335 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:50.558291    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr" event={"ID":"1522dc9d-0dbc-4830-b20e-e8eeb82108af","Type":"ContainerDied","Data":"1ea1b263d6468cf68e662695d885e57b4a4385015c1b7b5a7b6c8b92f8d07036"}
Sep 12 00:55:50.558811 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:50.558366    2567 scope.go:117] "RemoveContainer" containerID="eeb808315412392fbd5a87e4019e219acb8acceae4e22a2441bc0d3114068ae7"
Sep 12 00:55:50.574904 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:50.574874    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"]
Sep 12 00:55:50.583502 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:50.583468    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr"]
Sep 12 00:55:50.602179 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:50.602136    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5306e6ca-7595f6f657-w5glr" podUID="1522dc9d-0dbc-4830-b20e-e8eeb82108af" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.64:9004: i/o timeout"
Sep 12 00:55:52.273684 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:52.273649    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1522dc9d-0dbc-4830-b20e-e8eeb82108af" path="/var/lib/kubelet/pods/1522dc9d-0dbc-4830-b20e-e8eeb82108af/volumes"
Sep 12 00:55:54.342393 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.342357    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"]
Sep 12 00:55:54.342803 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.342733    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1522dc9d-0dbc-4830-b20e-e8eeb82108af" containerName="payload-processing"
Sep 12 00:55:54.342803 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.342745    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1522dc9d-0dbc-4830-b20e-e8eeb82108af" containerName="payload-processing"
Sep 12 00:55:54.342803 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.342764    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d72b0a-be7d-451e-bd78-31f51cbf41b4" containerName="revoke-keys"
Sep 12 00:55:54.342803 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.342770    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d72b0a-be7d-451e-bd78-31f51cbf41b4" containerName="revoke-keys"
Sep 12 00:55:54.342803 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.342779    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f31f55-918f-4cf2-99a8-6e8cc1f76f1d" containerName="main"
Sep 12 00:55:54.342803 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.342784    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f31f55-918f-4cf2-99a8-6e8cc1f76f1d" containerName="main"
Sep 12 00:55:54.343007 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.342843    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f31f55-918f-4cf2-99a8-6e8cc1f76f1d" containerName="main"
Sep 12 00:55:54.343007 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.342851    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d72b0a-be7d-451e-bd78-31f51cbf41b4" containerName="revoke-keys"
Sep 12 00:55:54.343007 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.342860    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1522dc9d-0dbc-4830-b20e-e8eeb82108af" containerName="payload-processing"
Sep 12 00:55:54.347229 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.347209    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.350050 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.350018    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-876808\"/\"distinct-w4-876808-kserve-self-signed-certs\""
Sep 12 00:55:54.367416 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.367390    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"]
Sep 12 00:55:54.507727 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.507689    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-dshm\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.507932 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.507740    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-model-cache\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.507932 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.507841    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-home\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.508026 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.507978    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hr5f\" (UniqueName: \"kubernetes.io/projected/827f9d20-74d3-4601-a68a-927b164ff00f-kube-api-access-5hr5f\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.508026 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.508025    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827f9d20-74d3-4601-a68a-927b164ff00f-tls-certs\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.508112 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.508046    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-tmp-dir\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.609272 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.609171    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hr5f\" (UniqueName: \"kubernetes.io/projected/827f9d20-74d3-4601-a68a-927b164ff00f-kube-api-access-5hr5f\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.609272 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.609231    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827f9d20-74d3-4601-a68a-927b164ff00f-tls-certs\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.609272 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.609260    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-tmp-dir\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.609588 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.609281    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-dshm\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.609588 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.609306    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-model-cache\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.609588 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.609343    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-home\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.609808 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.609761    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-model-cache\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.609894 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.609815    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-home\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.609894 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.609776    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-tmp-dir\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.611717 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.611691    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-dshm\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.612001 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.611979    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827f9d20-74d3-4601-a68a-927b164ff00f-tls-certs\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.618853 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.618828    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hr5f\" (UniqueName: \"kubernetes.io/projected/827f9d20-74d3-4601-a68a-927b164ff00f-kube-api-access-5hr5f\") pod \"distinct-w4-876808-kserve-85cdc976d4-5mfz7\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") " pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.656706 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.656666    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:55:54.935798 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:54.935715    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"]
Sep 12 00:55:54.938926 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:55:54.938897    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod827f9d20_74d3_4601_a68a_927b164ff00f.slice/crio-54283699241c2af69124cb46dd5f72a690d6e335c96c63ae75c45356b4549f3c WatchSource:0}: Error finding container 54283699241c2af69124cb46dd5f72a690d6e335c96c63ae75c45356b4549f3c: Status 404 returned error can't find the container with id 54283699241c2af69124cb46dd5f72a690d6e335c96c63ae75c45356b4549f3c
Sep 12 00:55:55.590681 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:55.590644    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7" event={"ID":"827f9d20-74d3-4601-a68a-927b164ff00f","Type":"ContainerStarted","Data":"4c2a54e26fd47091629632cb5923b54fb06165159b309b11d1659feac9f40c4f"}
Sep 12 00:55:55.591101 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:55.590694    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7" event={"ID":"827f9d20-74d3-4601-a68a-927b164ff00f","Type":"ContainerStarted","Data":"54283699241c2af69124cb46dd5f72a690d6e335c96c63ae75c45356b4549f3c"}
Sep 12 00:55:55.613035 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:55:55.612983    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7" podStartSLOduration=1.612965207 podStartE2EDuration="1.612965207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:55:55.611391566 +0000 UTC m=+1587.815392389" watchObservedRunningTime="2026-09-12 00:55:55.612965207 +0000 UTC m=+1587.816966028"
Sep 12 00:56:04.657458 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:56:04.657416    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:56:04.658019 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:56:04.657477    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:56:04.670672 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:56:04.670641    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:56:05.657072 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:56:05.657041    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 00:57:28.881220 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:28.881179    2567 scope.go:117] "RemoveContainer" containerID="c646e818aaa0e7fb3e43ece703c60c3a5e075bf402a5f8db085093855eaf747e"
Sep 12 00:57:28.893911 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:28.893883    2567 scope.go:117] "RemoveContainer" containerID="cfdb4f0c368d882548356a7b073fb52e36da1af3ccc74ac767ee5315dc3a1086"
Sep 12 00:57:46.336845 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.336801    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"]
Sep 12 00:57:46.340360 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.340336    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.343340 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.343317    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4d485e\"/\"embedding-w0-4d485e-kserve-self-signed-certs\""
Sep 12 00:57:46.354482 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.354456    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"]
Sep 12 00:57:46.450733 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.450694    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-model-cache\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.450947 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.450757    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p27qr\" (UniqueName: \"kubernetes.io/projected/f794a452-52ef-435e-86dc-8f6a39f7ba22-kube-api-access-p27qr\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.450947 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.450866    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f794a452-52ef-435e-86dc-8f6a39f7ba22-tls-certs\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.451087 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.450945    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-home\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.451087 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.451005    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-tmp-dir\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.451087 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.451044    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-dshm\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.552065 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.552021    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f794a452-52ef-435e-86dc-8f6a39f7ba22-tls-certs\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.552269 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.552087    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-home\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.552269 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.552125    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-tmp-dir\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.552269 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.552147    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-dshm\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.552269 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.552176    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-model-cache\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.552269 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.552227    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p27qr\" (UniqueName: \"kubernetes.io/projected/f794a452-52ef-435e-86dc-8f6a39f7ba22-kube-api-access-p27qr\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.552664 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.552626    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-home\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.552664 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.552652    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-tmp-dir\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.552842 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.552719    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-model-cache\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.554439 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.554417    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-dshm\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.554752 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.554733    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f794a452-52ef-435e-86dc-8f6a39f7ba22-tls-certs\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.562724 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.562698    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p27qr\" (UniqueName: \"kubernetes.io/projected/f794a452-52ef-435e-86dc-8f6a39f7ba22-kube-api-access-p27qr\") pod \"embedding-w0-4d485e-kserve-5bbccc5558-vnnlc\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") " pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.649862 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.649765    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:46.925889 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:46.925864    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"]
Sep 12 00:57:46.928526 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:57:46.928495    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf794a452_52ef_435e_86dc_8f6a39f7ba22.slice/crio-4725db0eca0bb79ea3bc6c72a0f10f96fe1f11078d23050cb56ff245ccaad645 WatchSource:0}: Error finding container 4725db0eca0bb79ea3bc6c72a0f10f96fe1f11078d23050cb56ff245ccaad645: Status 404 returned error can't find the container with id 4725db0eca0bb79ea3bc6c72a0f10f96fe1f11078d23050cb56ff245ccaad645
Sep 12 00:57:47.196728 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:47.196636    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc" event={"ID":"f794a452-52ef-435e-86dc-8f6a39f7ba22","Type":"ContainerStarted","Data":"f41dd5913b8a50af40a3996b351ea3ae6016b23c016e943ee8c6b4e4bba05653"}
Sep 12 00:57:47.196728 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:47.196692    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc" event={"ID":"f794a452-52ef-435e-86dc-8f6a39f7ba22","Type":"ContainerStarted","Data":"4725db0eca0bb79ea3bc6c72a0f10f96fe1f11078d23050cb56ff245ccaad645"}
Sep 12 00:57:47.218031 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:47.217949    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc" podStartSLOduration=1.217930785 podStartE2EDuration="1.217930785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:57:47.215493705 +0000 UTC m=+1699.419494554" watchObservedRunningTime="2026-09-12 00:57:47.217930785 +0000 UTC m=+1699.421931607"
Sep 12 00:57:52.375300 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:52.375208    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"]
Sep 12 00:57:52.378929 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:52.378905    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"
Sep 12 00:57:52.381535 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:52.381505    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-04f8e8\""
Sep 12 00:57:52.381702 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:52.381609    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-04f8e8-dockercfg-mg845\""
Sep 12 00:57:52.389856 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:52.389829    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"]
Sep 12 00:57:52.514043 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:52.514007    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc30cab0-2db2-4733-a500-f5ba203c731c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw\" (UID: \"dc30cab0-2db2-4733-a500-f5ba203c731c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"
Sep 12 00:57:52.514232 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:52.514088    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb9wb\" (UniqueName: \"kubernetes.io/projected/dc30cab0-2db2-4733-a500-f5ba203c731c-kube-api-access-tb9wb\") pod \"payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw\" (UID: \"dc30cab0-2db2-4733-a500-f5ba203c731c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"
Sep 12 00:57:52.614710 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:52.614673    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb9wb\" (UniqueName: \"kubernetes.io/projected/dc30cab0-2db2-4733-a500-f5ba203c731c-kube-api-access-tb9wb\") pod \"payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw\" (UID: \"dc30cab0-2db2-4733-a500-f5ba203c731c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"
Sep 12 00:57:52.614892 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:52.614776    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc30cab0-2db2-4733-a500-f5ba203c731c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw\" (UID: \"dc30cab0-2db2-4733-a500-f5ba203c731c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"
Sep 12 00:57:52.615501 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:52.615479    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc30cab0-2db2-4733-a500-f5ba203c731c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw\" (UID: \"dc30cab0-2db2-4733-a500-f5ba203c731c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"
Sep 12 00:57:52.623884 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:52.623851    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb9wb\" (UniqueName: \"kubernetes.io/projected/dc30cab0-2db2-4733-a500-f5ba203c731c-kube-api-access-tb9wb\") pod \"payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw\" (UID: \"dc30cab0-2db2-4733-a500-f5ba203c731c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"
Sep 12 00:57:52.691379 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:52.691283    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"
Sep 12 00:57:52.967939 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:52.967843    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"]
Sep 12 00:57:52.973714 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:57:52.973683    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc30cab0_2db2_4733_a500_f5ba203c731c.slice/crio-c90ecee8f403eb2888fd1a6e620633217081778e44008b074c623366e712f3e0 WatchSource:0}: Error finding container c90ecee8f403eb2888fd1a6e620633217081778e44008b074c623366e712f3e0: Status 404 returned error can't find the container with id c90ecee8f403eb2888fd1a6e620633217081778e44008b074c623366e712f3e0
Sep 12 00:57:53.236630 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:53.236562    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw" event={"ID":"dc30cab0-2db2-4733-a500-f5ba203c731c","Type":"ContainerStarted","Data":"e459d2ac58420c89ed7423b7c23c04f5d5eef8aee57bc51e8c3fb04fcece3620"}
Sep 12 00:57:53.236817 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:53.236656    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw" event={"ID":"dc30cab0-2db2-4733-a500-f5ba203c731c","Type":"ContainerStarted","Data":"c90ecee8f403eb2888fd1a6e620633217081778e44008b074c623366e712f3e0"}
Sep 12 00:57:53.236817 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:53.236771    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"
Sep 12 00:57:53.255984 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:53.255922    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw" podStartSLOduration=1.2559034 podStartE2EDuration="1.2559034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:57:53.255283648 +0000 UTC m=+1705.459284481" watchObservedRunningTime="2026-09-12 00:57:53.2559034 +0000 UTC m=+1705.459904221"
Sep 12 00:57:56.650421 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:56.650375    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:56.651003 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:56.650521    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:56.663377 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:56.663343    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:57:57.268830 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:57:57.268797    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:58:04.244918 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:04.244878    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"
Sep 12 00:58:22.552921 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.552882    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"]
Sep 12 00:58:22.556539 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.556519    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"
Sep 12 00:58:22.559414 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.559397    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c14a20\""
Sep 12 00:58:22.567181 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.567147    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"]
Sep 12 00:58:22.574332 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.574301    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"]
Sep 12 00:58:22.577808 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.577785    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"
Sep 12 00:58:22.580479 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.580454    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c14a20-dockercfg-hgs72\""
Sep 12 00:58:22.588920 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.588894    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"]
Sep 12 00:58:22.618264 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.618230    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d52f8\" (UniqueName: \"kubernetes.io/projected/ada8a8fe-16e4-4511-b623-9e48f321a297-kube-api-access-d52f8\") pod \"payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf\" (UID: \"ada8a8fe-16e4-4511-b623-9e48f321a297\") " pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"
Sep 12 00:58:22.618452 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.618288    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b5c1a5-1e45-4278-b45e-fe7b3debeaca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6\" (UID: \"10b5c1a5-1e45-4278-b45e-fe7b3debeaca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"
Sep 12 00:58:22.618452 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.618377    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada8a8fe-16e4-4511-b623-9e48f321a297-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf\" (UID: \"ada8a8fe-16e4-4511-b623-9e48f321a297\") " pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"
Sep 12 00:58:22.618610 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.618520    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfkhc\" (UniqueName: \"kubernetes.io/projected/10b5c1a5-1e45-4278-b45e-fe7b3debeaca-kube-api-access-cfkhc\") pod \"payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6\" (UID: \"10b5c1a5-1e45-4278-b45e-fe7b3debeaca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"
Sep 12 00:58:22.719698 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.719660    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada8a8fe-16e4-4511-b623-9e48f321a297-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf\" (UID: \"ada8a8fe-16e4-4511-b623-9e48f321a297\") " pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"
Sep 12 00:58:22.719885 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.719759    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfkhc\" (UniqueName: \"kubernetes.io/projected/10b5c1a5-1e45-4278-b45e-fe7b3debeaca-kube-api-access-cfkhc\") pod \"payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6\" (UID: \"10b5c1a5-1e45-4278-b45e-fe7b3debeaca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"
Sep 12 00:58:22.719885 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.719793    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d52f8\" (UniqueName: \"kubernetes.io/projected/ada8a8fe-16e4-4511-b623-9e48f321a297-kube-api-access-d52f8\") pod \"payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf\" (UID: \"ada8a8fe-16e4-4511-b623-9e48f321a297\") " pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"
Sep 12 00:58:22.719885 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.719833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b5c1a5-1e45-4278-b45e-fe7b3debeaca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6\" (UID: \"10b5c1a5-1e45-4278-b45e-fe7b3debeaca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"
Sep 12 00:58:22.720462 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.720442    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada8a8fe-16e4-4511-b623-9e48f321a297-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf\" (UID: \"ada8a8fe-16e4-4511-b623-9e48f321a297\") " pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"
Sep 12 00:58:22.720563 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.720545    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b5c1a5-1e45-4278-b45e-fe7b3debeaca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6\" (UID: \"10b5c1a5-1e45-4278-b45e-fe7b3debeaca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"
Sep 12 00:58:22.727749 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.727726    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d52f8\" (UniqueName: \"kubernetes.io/projected/ada8a8fe-16e4-4511-b623-9e48f321a297-kube-api-access-d52f8\") pod \"payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf\" (UID: \"ada8a8fe-16e4-4511-b623-9e48f321a297\") " pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"
Sep 12 00:58:22.727837 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.727783    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfkhc\" (UniqueName: \"kubernetes.io/projected/10b5c1a5-1e45-4278-b45e-fe7b3debeaca-kube-api-access-cfkhc\") pod \"payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6\" (UID: \"10b5c1a5-1e45-4278-b45e-fe7b3debeaca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"
Sep 12 00:58:22.867909 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.867800    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"
Sep 12 00:58:22.889805 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:22.889768    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"
Sep 12 00:58:23.151446 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:23.151416    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"]
Sep 12 00:58:23.153256 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:58:23.153229    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b5c1a5_1e45_4278_b45e_fe7b3debeaca.slice/crio-52baebcccff5504931b6c23a47608df6cc202bdb8c48998a8a680edaae3ceee0 WatchSource:0}: Error finding container 52baebcccff5504931b6c23a47608df6cc202bdb8c48998a8a680edaae3ceee0: Status 404 returned error can't find the container with id 52baebcccff5504931b6c23a47608df6cc202bdb8c48998a8a680edaae3ceee0
Sep 12 00:58:23.185908 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:23.185883    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"]
Sep 12 00:58:23.195809 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:58:23.195773    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada8a8fe_16e4_4511_b623_9e48f321a297.slice/crio-6feb981cde758a041edea7518ee70d0742a1c1cee7a40623627ed0c50cb484f9 WatchSource:0}: Error finding container 6feb981cde758a041edea7518ee70d0742a1c1cee7a40623627ed0c50cb484f9: Status 404 returned error can't find the container with id 6feb981cde758a041edea7518ee70d0742a1c1cee7a40623627ed0c50cb484f9
Sep 12 00:58:23.397122 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:23.397083    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6" event={"ID":"10b5c1a5-1e45-4278-b45e-fe7b3debeaca","Type":"ContainerStarted","Data":"7b838b6a047ec2882fe5caa2b6f0c70b52d9c8159a93e21ef3ff462150cf8c4e"}
Sep 12 00:58:23.397347 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:23.397158    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"
Sep 12 00:58:23.397347 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:23.397180    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6" event={"ID":"10b5c1a5-1e45-4278-b45e-fe7b3debeaca","Type":"ContainerStarted","Data":"52baebcccff5504931b6c23a47608df6cc202bdb8c48998a8a680edaae3ceee0"}
Sep 12 00:58:23.398810 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:23.398785    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf" event={"ID":"ada8a8fe-16e4-4511-b623-9e48f321a297","Type":"ContainerStarted","Data":"91c66f1eda2db91d335a3228adef51ec346ff09143c4871a01e20406e02b6dea"}
Sep 12 00:58:23.398897 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:23.398841    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"
Sep 12 00:58:23.398897 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:23.398859    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf" event={"ID":"ada8a8fe-16e4-4511-b623-9e48f321a297","Type":"ContainerStarted","Data":"6feb981cde758a041edea7518ee70d0742a1c1cee7a40623627ed0c50cb484f9"}
Sep 12 00:58:23.413078 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:23.412981    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6" podStartSLOduration=1.41296332 podStartE2EDuration="1.41296332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:58:23.41171641 +0000 UTC m=+1735.615717235" watchObservedRunningTime="2026-09-12 00:58:23.41296332 +0000 UTC m=+1735.616964141"
Sep 12 00:58:23.429442 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:23.429384    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf" podStartSLOduration=1.4293679799999999 podStartE2EDuration="1.42936798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:58:23.428176349 +0000 UTC m=+1735.632177172" watchObservedRunningTime="2026-09-12 00:58:23.42936798 +0000 UTC m=+1735.633368856"
Sep 12 00:58:34.405842 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:34.405805    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"
Sep 12 00:58:34.406321 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:34.406042    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"
Sep 12 00:58:47.968017 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:47.967974    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd"]
Sep 12 00:58:47.971466 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:47.971444    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd"
Sep 12 00:58:47.974006 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:47.973978    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:58:47.974149 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:47.974126    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:58:47.979459 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:47.979422    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37763a4-18e7-4665-99a3-6f8928bcaebd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd\" (UID: \"a37763a4-18e7-4665-99a3-6f8928bcaebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd"
Sep 12 00:58:47.979786 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:47.979761    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd"]
Sep 12 00:58:48.080048 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:48.079992    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37763a4-18e7-4665-99a3-6f8928bcaebd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd\" (UID: \"a37763a4-18e7-4665-99a3-6f8928bcaebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd"
Sep 12 00:58:48.080770 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:48.080745    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37763a4-18e7-4665-99a3-6f8928bcaebd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd\" (UID: \"a37763a4-18e7-4665-99a3-6f8928bcaebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd"
Sep 12 00:58:48.284138 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:48.284099    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd"
Sep 12 00:58:48.565123 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:48.565032    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd"]
Sep 12 00:58:48.568660 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:58:48.568624    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37763a4_18e7_4665_99a3_6f8928bcaebd.slice/crio-6dc933a886b38a8d3d9f5da85823896d4d357e2c22cabb709817918707d4b3c6 WatchSource:0}: Error finding container 6dc933a886b38a8d3d9f5da85823896d4d357e2c22cabb709817918707d4b3c6: Status 404 returned error can't find the container with id 6dc933a886b38a8d3d9f5da85823896d4d357e2c22cabb709817918707d4b3c6
Sep 12 00:58:49.546979 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:49.546888    2567 generic.go:356] "Generic (PLEG): container finished" podID="a37763a4-18e7-4665-99a3-6f8928bcaebd" containerID="37160fbd8d09544f8381b151ea9020b37ef916ddb94e1fd4d545a93548befba2" exitCode=0
Sep 12 00:58:49.547365 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:49.546972    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd" event={"ID":"a37763a4-18e7-4665-99a3-6f8928bcaebd","Type":"ContainerDied","Data":"37160fbd8d09544f8381b151ea9020b37ef916ddb94e1fd4d545a93548befba2"}
Sep 12 00:58:49.547365 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:49.547036    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd" event={"ID":"a37763a4-18e7-4665-99a3-6f8928bcaebd","Type":"ContainerStarted","Data":"6dc933a886b38a8d3d9f5da85823896d4d357e2c22cabb709817918707d4b3c6"}
Sep 12 00:58:50.701808 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:50.701774    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd"
Sep 12 00:58:50.810464 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:50.810365    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37763a4-18e7-4665-99a3-6f8928bcaebd-maas-api-service-ca\") pod \"a37763a4-18e7-4665-99a3-6f8928bcaebd\" (UID: \"a37763a4-18e7-4665-99a3-6f8928bcaebd\") "
Sep 12 00:58:50.810846 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:50.810823    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37763a4-18e7-4665-99a3-6f8928bcaebd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a37763a4-18e7-4665-99a3-6f8928bcaebd" (UID: "a37763a4-18e7-4665-99a3-6f8928bcaebd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:58:50.911421 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:50.911378    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37763a4-18e7-4665-99a3-6f8928bcaebd-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:58:51.562227 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:51.562196    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd"
Sep 12 00:58:51.562227 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:51.562209    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd" event={"ID":"a37763a4-18e7-4665-99a3-6f8928bcaebd","Type":"ContainerDied","Data":"6dc933a886b38a8d3d9f5da85823896d4d357e2c22cabb709817918707d4b3c6"}
Sep 12 00:58:51.562516 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:58:51.562273    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dc933a886b38a8d3d9f5da85823896d4d357e2c22cabb709817918707d4b3c6"
Sep 12 00:59:03.707631 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:03.707596    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"]
Sep 12 00:59:03.708189 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:03.707842    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw" podUID="dc30cab0-2db2-4733-a500-f5ba203c731c" containerName="payload-pre-processing" containerID="cri-o://e459d2ac58420c89ed7423b7c23c04f5d5eef8aee57bc51e8c3fb04fcece3620" gracePeriod=30
Sep 12 00:59:03.908764 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:03.908727    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd"]
Sep 12 00:59:03.912643 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:03.912613    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-04f8e8-7hmnd"]
Sep 12 00:59:03.966033 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:03.965948    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"
Sep 12 00:59:04.045950 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.045904    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb9wb\" (UniqueName: \"kubernetes.io/projected/dc30cab0-2db2-4733-a500-f5ba203c731c-kube-api-access-tb9wb\") pod \"dc30cab0-2db2-4733-a500-f5ba203c731c\" (UID: \"dc30cab0-2db2-4733-a500-f5ba203c731c\") "
Sep 12 00:59:04.045950 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.045959    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc30cab0-2db2-4733-a500-f5ba203c731c-plugins-config-volume\") pod \"dc30cab0-2db2-4733-a500-f5ba203c731c\" (UID: \"dc30cab0-2db2-4733-a500-f5ba203c731c\") "
Sep 12 00:59:04.046369 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.046343    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc30cab0-2db2-4733-a500-f5ba203c731c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dc30cab0-2db2-4733-a500-f5ba203c731c" (UID: "dc30cab0-2db2-4733-a500-f5ba203c731c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:59:04.048235 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.048211    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc30cab0-2db2-4733-a500-f5ba203c731c-kube-api-access-tb9wb" (OuterVolumeSpecName: "kube-api-access-tb9wb") pod "dc30cab0-2db2-4733-a500-f5ba203c731c" (UID: "dc30cab0-2db2-4733-a500-f5ba203c731c"). InnerVolumeSpecName "kube-api-access-tb9wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:59:04.146810 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.146770    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb9wb\" (UniqueName: \"kubernetes.io/projected/dc30cab0-2db2-4733-a500-f5ba203c731c-kube-api-access-tb9wb\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:04.146810 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.146803    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc30cab0-2db2-4733-a500-f5ba203c731c-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:04.274123 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.274085    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a37763a4-18e7-4665-99a3-6f8928bcaebd" path="/var/lib/kubelet/pods/a37763a4-18e7-4665-99a3-6f8928bcaebd/volumes"
Sep 12 00:59:04.639778 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.639684    2567 generic.go:356] "Generic (PLEG): container finished" podID="dc30cab0-2db2-4733-a500-f5ba203c731c" containerID="e459d2ac58420c89ed7423b7c23c04f5d5eef8aee57bc51e8c3fb04fcece3620" exitCode=0
Sep 12 00:59:04.639778 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.639732    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"
Sep 12 00:59:04.640018 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.639775    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw" event={"ID":"dc30cab0-2db2-4733-a500-f5ba203c731c","Type":"ContainerDied","Data":"e459d2ac58420c89ed7423b7c23c04f5d5eef8aee57bc51e8c3fb04fcece3620"}
Sep 12 00:59:04.640018 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.639829    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw" event={"ID":"dc30cab0-2db2-4733-a500-f5ba203c731c","Type":"ContainerDied","Data":"c90ecee8f403eb2888fd1a6e620633217081778e44008b074c623366e712f3e0"}
Sep 12 00:59:04.640018 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.639852    2567 scope.go:117] "RemoveContainer" containerID="e459d2ac58420c89ed7423b7c23c04f5d5eef8aee57bc51e8c3fb04fcece3620"
Sep 12 00:59:04.649132 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.649107    2567 scope.go:117] "RemoveContainer" containerID="e459d2ac58420c89ed7423b7c23c04f5d5eef8aee57bc51e8c3fb04fcece3620"
Sep 12 00:59:04.649457 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:59:04.649434    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e459d2ac58420c89ed7423b7c23c04f5d5eef8aee57bc51e8c3fb04fcece3620\": container with ID starting with e459d2ac58420c89ed7423b7c23c04f5d5eef8aee57bc51e8c3fb04fcece3620 not found: ID does not exist" containerID="e459d2ac58420c89ed7423b7c23c04f5d5eef8aee57bc51e8c3fb04fcece3620"
Sep 12 00:59:04.649545 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.649471    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e459d2ac58420c89ed7423b7c23c04f5d5eef8aee57bc51e8c3fb04fcece3620"} err="failed to get container status \"e459d2ac58420c89ed7423b7c23c04f5d5eef8aee57bc51e8c3fb04fcece3620\": rpc error: code = NotFound desc = could not find container \"e459d2ac58420c89ed7423b7c23c04f5d5eef8aee57bc51e8c3fb04fcece3620\": container with ID starting with e459d2ac58420c89ed7423b7c23c04f5d5eef8aee57bc51e8c3fb04fcece3620 not found: ID does not exist"
Sep 12 00:59:04.659245 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.659215    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"]
Sep 12 00:59:04.663757 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:04.663730    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-04f8e8-77b7bbcbd6-94svw"]
Sep 12 00:59:06.274316 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:06.274275    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc30cab0-2db2-4733-a500-f5ba203c731c" path="/var/lib/kubelet/pods/dc30cab0-2db2-4733-a500-f5ba203c731c/volumes"
Sep 12 00:59:11.096777 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.096735    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t"]
Sep 12 00:59:11.097181 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.097137    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a37763a4-18e7-4665-99a3-6f8928bcaebd" containerName="revoke-keys"
Sep 12 00:59:11.097181 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.097149    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a37763a4-18e7-4665-99a3-6f8928bcaebd" containerName="revoke-keys"
Sep 12 00:59:11.097181 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.097175    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc30cab0-2db2-4733-a500-f5ba203c731c" containerName="payload-pre-processing"
Sep 12 00:59:11.097181 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.097182    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc30cab0-2db2-4733-a500-f5ba203c731c" containerName="payload-pre-processing"
Sep 12 00:59:11.097317 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.097239    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a37763a4-18e7-4665-99a3-6f8928bcaebd" containerName="revoke-keys"
Sep 12 00:59:11.097317 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.097249    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc30cab0-2db2-4733-a500-f5ba203c731c" containerName="payload-pre-processing"
Sep 12 00:59:11.101770 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.101742    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t"
Sep 12 00:59:11.105176 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.105153    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:59:11.105498 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.105472    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:59:11.124748 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.124710    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t"]
Sep 12 00:59:11.226480 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.226435    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ac255d-6d41-40f9-8c2a-3b709da77827-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t\" (UID: \"78ac255d-6d41-40f9-8c2a-3b709da77827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t"
Sep 12 00:59:11.327520 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.327479    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ac255d-6d41-40f9-8c2a-3b709da77827-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t\" (UID: \"78ac255d-6d41-40f9-8c2a-3b709da77827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t"
Sep 12 00:59:11.328194 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.328169    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ac255d-6d41-40f9-8c2a-3b709da77827-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t\" (UID: \"78ac255d-6d41-40f9-8c2a-3b709da77827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t"
Sep 12 00:59:11.412872 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.412775    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t"
Sep 12 00:59:11.688147 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:11.688121    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t"]
Sep 12 00:59:11.690376 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:59:11.690346    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ac255d_6d41_40f9_8c2a_3b709da77827.slice/crio-0bebc14802e3a74aa09bfde98ddd49dc3df061ddd583c29cc3f4c0ba35559a09 WatchSource:0}: Error finding container 0bebc14802e3a74aa09bfde98ddd49dc3df061ddd583c29cc3f4c0ba35559a09: Status 404 returned error can't find the container with id 0bebc14802e3a74aa09bfde98ddd49dc3df061ddd583c29cc3f4c0ba35559a09
Sep 12 00:59:12.686510 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:12.686400    2567 generic.go:356] "Generic (PLEG): container finished" podID="78ac255d-6d41-40f9-8c2a-3b709da77827" containerID="363e7cacc49c6f81d5704d23d2ff2bc1a7554b094482092644732d2cc08cca31" exitCode=0
Sep 12 00:59:12.686510 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:12.686492    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t" event={"ID":"78ac255d-6d41-40f9-8c2a-3b709da77827","Type":"ContainerDied","Data":"363e7cacc49c6f81d5704d23d2ff2bc1a7554b094482092644732d2cc08cca31"}
Sep 12 00:59:12.687022 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:12.686588    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t" event={"ID":"78ac255d-6d41-40f9-8c2a-3b709da77827","Type":"ContainerStarted","Data":"0bebc14802e3a74aa09bfde98ddd49dc3df061ddd583c29cc3f4c0ba35559a09"}
Sep 12 00:59:13.854468 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:13.854434    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t"
Sep 12 00:59:13.956066 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:13.955961    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ac255d-6d41-40f9-8c2a-3b709da77827-maas-api-service-ca\") pod \"78ac255d-6d41-40f9-8c2a-3b709da77827\" (UID: \"78ac255d-6d41-40f9-8c2a-3b709da77827\") "
Sep 12 00:59:13.956352 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:13.956327    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ac255d-6d41-40f9-8c2a-3b709da77827-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78ac255d-6d41-40f9-8c2a-3b709da77827" (UID: "78ac255d-6d41-40f9-8c2a-3b709da77827"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:59:14.057045 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:14.057003    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ac255d-6d41-40f9-8c2a-3b709da77827-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:14.700227 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:14.700141    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t"
Sep 12 00:59:14.700227 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:14.700187    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t" event={"ID":"78ac255d-6d41-40f9-8c2a-3b709da77827","Type":"ContainerDied","Data":"0bebc14802e3a74aa09bfde98ddd49dc3df061ddd583c29cc3f4c0ba35559a09"}
Sep 12 00:59:14.700425 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:14.700238    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bebc14802e3a74aa09bfde98ddd49dc3df061ddd583c29cc3f4c0ba35559a09"
Sep 12 00:59:26.753645 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:26.753608    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"]
Sep 12 00:59:26.754154 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:26.753840    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf" podUID="ada8a8fe-16e4-4511-b623-9e48f321a297" containerName="payload-processing" containerID="cri-o://91c66f1eda2db91d335a3228adef51ec346ff09143c4871a01e20406e02b6dea" gracePeriod=30
Sep 12 00:59:26.761851 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:26.761816    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"]
Sep 12 00:59:26.762976 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:26.762073    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6" podUID="10b5c1a5-1e45-4278-b45e-fe7b3debeaca" containerName="payload-pre-processing" containerID="cri-o://7b838b6a047ec2882fe5caa2b6f0c70b52d9c8159a93e21ef3ff462150cf8c4e" gracePeriod=30
Sep 12 00:59:26.918703 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:26.918530    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t"]
Sep 12 00:59:26.921263 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:26.921230    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c14a20-vgw2t"]
Sep 12 00:59:27.035289 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.035249    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"
Sep 12 00:59:27.066420 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.066390    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"
Sep 12 00:59:27.088822 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.088788    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b5c1a5-1e45-4278-b45e-fe7b3debeaca-plugins-config-volume\") pod \"10b5c1a5-1e45-4278-b45e-fe7b3debeaca\" (UID: \"10b5c1a5-1e45-4278-b45e-fe7b3debeaca\") "
Sep 12 00:59:27.089020 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.088840    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada8a8fe-16e4-4511-b623-9e48f321a297-plugins-config-volume\") pod \"ada8a8fe-16e4-4511-b623-9e48f321a297\" (UID: \"ada8a8fe-16e4-4511-b623-9e48f321a297\") "
Sep 12 00:59:27.089020 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.088917    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfkhc\" (UniqueName: \"kubernetes.io/projected/10b5c1a5-1e45-4278-b45e-fe7b3debeaca-kube-api-access-cfkhc\") pod \"10b5c1a5-1e45-4278-b45e-fe7b3debeaca\" (UID: \"10b5c1a5-1e45-4278-b45e-fe7b3debeaca\") "
Sep 12 00:59:27.089020 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.088957    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d52f8\" (UniqueName: \"kubernetes.io/projected/ada8a8fe-16e4-4511-b623-9e48f321a297-kube-api-access-d52f8\") pod \"ada8a8fe-16e4-4511-b623-9e48f321a297\" (UID: \"ada8a8fe-16e4-4511-b623-9e48f321a297\") "
Sep 12 00:59:27.089714 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.089430    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada8a8fe-16e4-4511-b623-9e48f321a297-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ada8a8fe-16e4-4511-b623-9e48f321a297" (UID: "ada8a8fe-16e4-4511-b623-9e48f321a297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:59:27.089714 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.089507    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b5c1a5-1e45-4278-b45e-fe7b3debeaca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "10b5c1a5-1e45-4278-b45e-fe7b3debeaca" (UID: "10b5c1a5-1e45-4278-b45e-fe7b3debeaca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:59:27.091919 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.091889    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada8a8fe-16e4-4511-b623-9e48f321a297-kube-api-access-d52f8" (OuterVolumeSpecName: "kube-api-access-d52f8") pod "ada8a8fe-16e4-4511-b623-9e48f321a297" (UID: "ada8a8fe-16e4-4511-b623-9e48f321a297"). InnerVolumeSpecName "kube-api-access-d52f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:59:27.092009 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.091934    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b5c1a5-1e45-4278-b45e-fe7b3debeaca-kube-api-access-cfkhc" (OuterVolumeSpecName: "kube-api-access-cfkhc") pod "10b5c1a5-1e45-4278-b45e-fe7b3debeaca" (UID: "10b5c1a5-1e45-4278-b45e-fe7b3debeaca"). InnerVolumeSpecName "kube-api-access-cfkhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:59:27.190224 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.190109    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b5c1a5-1e45-4278-b45e-fe7b3debeaca-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:27.190224 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.190161    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada8a8fe-16e4-4511-b623-9e48f321a297-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:27.190224 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.190171    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfkhc\" (UniqueName: \"kubernetes.io/projected/10b5c1a5-1e45-4278-b45e-fe7b3debeaca-kube-api-access-cfkhc\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:27.190224 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.190181    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d52f8\" (UniqueName: \"kubernetes.io/projected/ada8a8fe-16e4-4511-b623-9e48f321a297-kube-api-access-d52f8\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:27.781785 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.781748    2567 generic.go:356] "Generic (PLEG): container finished" podID="ada8a8fe-16e4-4511-b623-9e48f321a297" containerID="91c66f1eda2db91d335a3228adef51ec346ff09143c4871a01e20406e02b6dea" exitCode=0
Sep 12 00:59:27.782275 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.781796    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"
Sep 12 00:59:27.782275 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.781837    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf" event={"ID":"ada8a8fe-16e4-4511-b623-9e48f321a297","Type":"ContainerDied","Data":"91c66f1eda2db91d335a3228adef51ec346ff09143c4871a01e20406e02b6dea"}
Sep 12 00:59:27.782275 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.781893    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf" event={"ID":"ada8a8fe-16e4-4511-b623-9e48f321a297","Type":"ContainerDied","Data":"6feb981cde758a041edea7518ee70d0742a1c1cee7a40623627ed0c50cb484f9"}
Sep 12 00:59:27.782275 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.781920    2567 scope.go:117] "RemoveContainer" containerID="91c66f1eda2db91d335a3228adef51ec346ff09143c4871a01e20406e02b6dea"
Sep 12 00:59:27.783460 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.783429    2567 generic.go:356] "Generic (PLEG): container finished" podID="10b5c1a5-1e45-4278-b45e-fe7b3debeaca" containerID="7b838b6a047ec2882fe5caa2b6f0c70b52d9c8159a93e21ef3ff462150cf8c4e" exitCode=0
Sep 12 00:59:27.783533 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.783481    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"
Sep 12 00:59:27.783533 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.783511    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6" event={"ID":"10b5c1a5-1e45-4278-b45e-fe7b3debeaca","Type":"ContainerDied","Data":"7b838b6a047ec2882fe5caa2b6f0c70b52d9c8159a93e21ef3ff462150cf8c4e"}
Sep 12 00:59:27.783640 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.783560    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6" event={"ID":"10b5c1a5-1e45-4278-b45e-fe7b3debeaca","Type":"ContainerDied","Data":"52baebcccff5504931b6c23a47608df6cc202bdb8c48998a8a680edaae3ceee0"}
Sep 12 00:59:27.792941 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.792917    2567 scope.go:117] "RemoveContainer" containerID="91c66f1eda2db91d335a3228adef51ec346ff09143c4871a01e20406e02b6dea"
Sep 12 00:59:27.793308 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:59:27.793275    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c66f1eda2db91d335a3228adef51ec346ff09143c4871a01e20406e02b6dea\": container with ID starting with 91c66f1eda2db91d335a3228adef51ec346ff09143c4871a01e20406e02b6dea not found: ID does not exist" containerID="91c66f1eda2db91d335a3228adef51ec346ff09143c4871a01e20406e02b6dea"
Sep 12 00:59:27.793432 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.793319    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c66f1eda2db91d335a3228adef51ec346ff09143c4871a01e20406e02b6dea"} err="failed to get container status \"91c66f1eda2db91d335a3228adef51ec346ff09143c4871a01e20406e02b6dea\": rpc error: code = NotFound desc = could not find container \"91c66f1eda2db91d335a3228adef51ec346ff09143c4871a01e20406e02b6dea\": container with ID starting with 91c66f1eda2db91d335a3228adef51ec346ff09143c4871a01e20406e02b6dea not found: ID does not exist"
Sep 12 00:59:27.793432 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.793338    2567 scope.go:117] "RemoveContainer" containerID="7b838b6a047ec2882fe5caa2b6f0c70b52d9c8159a93e21ef3ff462150cf8c4e"
Sep 12 00:59:27.802340 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.802308    2567 scope.go:117] "RemoveContainer" containerID="7b838b6a047ec2882fe5caa2b6f0c70b52d9c8159a93e21ef3ff462150cf8c4e"
Sep 12 00:59:27.802796 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:59:27.802769    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b838b6a047ec2882fe5caa2b6f0c70b52d9c8159a93e21ef3ff462150cf8c4e\": container with ID starting with 7b838b6a047ec2882fe5caa2b6f0c70b52d9c8159a93e21ef3ff462150cf8c4e not found: ID does not exist" containerID="7b838b6a047ec2882fe5caa2b6f0c70b52d9c8159a93e21ef3ff462150cf8c4e"
Sep 12 00:59:27.802902 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.802804    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b838b6a047ec2882fe5caa2b6f0c70b52d9c8159a93e21ef3ff462150cf8c4e"} err="failed to get container status \"7b838b6a047ec2882fe5caa2b6f0c70b52d9c8159a93e21ef3ff462150cf8c4e\": rpc error: code = NotFound desc = could not find container \"7b838b6a047ec2882fe5caa2b6f0c70b52d9c8159a93e21ef3ff462150cf8c4e\": container with ID starting with 7b838b6a047ec2882fe5caa2b6f0c70b52d9c8159a93e21ef3ff462150cf8c4e not found: ID does not exist"
Sep 12 00:59:27.805857 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.805825    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"]
Sep 12 00:59:27.811070 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.811040    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c14a20-689f56858c-qgfrf"]
Sep 12 00:59:27.821824 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.821790    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"]
Sep 12 00:59:27.826546 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:27.826513    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c14a20-6d949db59d-qjqd6"]
Sep 12 00:59:28.280896 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:28.280860    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10b5c1a5-1e45-4278-b45e-fe7b3debeaca" path="/var/lib/kubelet/pods/10b5c1a5-1e45-4278-b45e-fe7b3debeaca/volumes"
Sep 12 00:59:28.281222 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:28.281209    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78ac255d-6d41-40f9-8c2a-3b709da77827" path="/var/lib/kubelet/pods/78ac255d-6d41-40f9-8c2a-3b709da77827/volumes"
Sep 12 00:59:28.281551 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:28.281539    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ada8a8fe-16e4-4511-b623-9e48f321a297" path="/var/lib/kubelet/pods/ada8a8fe-16e4-4511-b623-9e48f321a297/volumes"
Sep 12 00:59:28.937480 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:28.937439    2567 scope.go:117] "RemoveContainer" containerID="f69769360674f688013150b703638a1648bf5956993e93f82e2835b6830b9a94"
Sep 12 00:59:28.947671 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:28.947649    2567 scope.go:117] "RemoveContainer" containerID="9fa261da73b0e852bccdb01cd80f86f4c8ae416b1b95db28ddad6b8d4b232793"
Sep 12 00:59:30.382040 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.381998    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq"]
Sep 12 00:59:30.382454 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.382395    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10b5c1a5-1e45-4278-b45e-fe7b3debeaca" containerName="payload-pre-processing"
Sep 12 00:59:30.382454 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.382407    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="10b5c1a5-1e45-4278-b45e-fe7b3debeaca" containerName="payload-pre-processing"
Sep 12 00:59:30.382454 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.382419    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ada8a8fe-16e4-4511-b623-9e48f321a297" containerName="payload-processing"
Sep 12 00:59:30.382454 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.382425    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ada8a8fe-16e4-4511-b623-9e48f321a297" containerName="payload-processing"
Sep 12 00:59:30.382454 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.382450    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78ac255d-6d41-40f9-8c2a-3b709da77827" containerName="revoke-keys"
Sep 12 00:59:30.382454 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.382458    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="78ac255d-6d41-40f9-8c2a-3b709da77827" containerName="revoke-keys"
Sep 12 00:59:30.382712 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.382527    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="10b5c1a5-1e45-4278-b45e-fe7b3debeaca" containerName="payload-pre-processing"
Sep 12 00:59:30.382712 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.382538    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ada8a8fe-16e4-4511-b623-9e48f321a297" containerName="payload-processing"
Sep 12 00:59:30.382712 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.382544    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="78ac255d-6d41-40f9-8c2a-3b709da77827" containerName="revoke-keys"
Sep 12 00:59:30.387271 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.387252    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq"
Sep 12 00:59:30.390100 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.390071    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:59:30.390303 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.390122    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:59:30.392369 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.392338    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq"]
Sep 12 00:59:30.414369 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.414328    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e6be17-8ded-4a74-9d4b-e1cb46017aa6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq\" (UID: \"d9e6be17-8ded-4a74-9d4b-e1cb46017aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq"
Sep 12 00:59:30.515050 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.514993    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e6be17-8ded-4a74-9d4b-e1cb46017aa6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq\" (UID: \"d9e6be17-8ded-4a74-9d4b-e1cb46017aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq"
Sep 12 00:59:30.515615 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.515588    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e6be17-8ded-4a74-9d4b-e1cb46017aa6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq\" (UID: \"d9e6be17-8ded-4a74-9d4b-e1cb46017aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq"
Sep 12 00:59:30.699890 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.699783    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq"
Sep 12 00:59:30.972852 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:30.972819    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq"]
Sep 12 00:59:30.975005 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:59:30.974973    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e6be17_8ded_4a74_9d4b_e1cb46017aa6.slice/crio-59520ed7067bdf513bd77e1be715c5cb7c27361b14deab276f79fd039b169bd8 WatchSource:0}: Error finding container 59520ed7067bdf513bd77e1be715c5cb7c27361b14deab276f79fd039b169bd8: Status 404 returned error can't find the container with id 59520ed7067bdf513bd77e1be715c5cb7c27361b14deab276f79fd039b169bd8
Sep 12 00:59:31.808449 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:31.808415    2567 generic.go:356] "Generic (PLEG): container finished" podID="d9e6be17-8ded-4a74-9d4b-e1cb46017aa6" containerID="c46a9de9cc191e97efa8310196a1363caebb64aef5b496ba4d8130f6c488acd7" exitCode=0
Sep 12 00:59:31.808906 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:31.808503    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq" event={"ID":"d9e6be17-8ded-4a74-9d4b-e1cb46017aa6","Type":"ContainerDied","Data":"c46a9de9cc191e97efa8310196a1363caebb64aef5b496ba4d8130f6c488acd7"}
Sep 12 00:59:31.808906 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:31.808567    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq" event={"ID":"d9e6be17-8ded-4a74-9d4b-e1cb46017aa6","Type":"ContainerStarted","Data":"59520ed7067bdf513bd77e1be715c5cb7c27361b14deab276f79fd039b169bd8"}
Sep 12 00:59:32.959447 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:32.959420    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq"
Sep 12 00:59:33.035641 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:33.035602    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e6be17-8ded-4a74-9d4b-e1cb46017aa6-maas-api-service-ca\") pod \"d9e6be17-8ded-4a74-9d4b-e1cb46017aa6\" (UID: \"d9e6be17-8ded-4a74-9d4b-e1cb46017aa6\") "
Sep 12 00:59:33.036015 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:33.035986    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e6be17-8ded-4a74-9d4b-e1cb46017aa6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9e6be17-8ded-4a74-9d4b-e1cb46017aa6" (UID: "d9e6be17-8ded-4a74-9d4b-e1cb46017aa6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:59:33.137279 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:33.137183    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e6be17-8ded-4a74-9d4b-e1cb46017aa6-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:33.821784 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:33.821750    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq"
Sep 12 00:59:33.821968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:33.821739    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq" event={"ID":"d9e6be17-8ded-4a74-9d4b-e1cb46017aa6","Type":"ContainerDied","Data":"59520ed7067bdf513bd77e1be715c5cb7c27361b14deab276f79fd039b169bd8"}
Sep 12 00:59:33.821968 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:33.821881    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59520ed7067bdf513bd77e1be715c5cb7c27361b14deab276f79fd039b169bd8"
Sep 12 00:59:34.170181 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.170094    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b"]
Sep 12 00:59:34.170558 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.170463    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e6be17-8ded-4a74-9d4b-e1cb46017aa6" containerName="revoke-keys"
Sep 12 00:59:34.170558 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.170474    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e6be17-8ded-4a74-9d4b-e1cb46017aa6" containerName="revoke-keys"
Sep 12 00:59:34.170670 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.170561    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e6be17-8ded-4a74-9d4b-e1cb46017aa6" containerName="revoke-keys"
Sep 12 00:59:34.173185 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.173163    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b"
Sep 12 00:59:34.176372 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.176341    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:59:34.176654 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.176636    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:59:34.190562 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.190531    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b"]
Sep 12 00:59:34.248847 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.248797    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/403cdf9e-c1dc-4827-8b3b-a6d970713cee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b\" (UID: \"403cdf9e-c1dc-4827-8b3b-a6d970713cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b"
Sep 12 00:59:34.349348 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.349308    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/403cdf9e-c1dc-4827-8b3b-a6d970713cee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b\" (UID: \"403cdf9e-c1dc-4827-8b3b-a6d970713cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b"
Sep 12 00:59:34.350099 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.350074    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/403cdf9e-c1dc-4827-8b3b-a6d970713cee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b\" (UID: \"403cdf9e-c1dc-4827-8b3b-a6d970713cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b"
Sep 12 00:59:34.483286 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.483249    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b"
Sep 12 00:59:34.759869 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.759840    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b"]
Sep 12 00:59:34.762542 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:59:34.762512    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403cdf9e_c1dc_4827_8b3b_a6d970713cee.slice/crio-eed07bc5cb22dba122d621ec60ab6d369013979194423fba73bfedce00730db6 WatchSource:0}: Error finding container eed07bc5cb22dba122d621ec60ab6d369013979194423fba73bfedce00730db6: Status 404 returned error can't find the container with id eed07bc5cb22dba122d621ec60ab6d369013979194423fba73bfedce00730db6
Sep 12 00:59:34.828355 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:34.828318    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b" event={"ID":"403cdf9e-c1dc-4827-8b3b-a6d970713cee","Type":"ContainerStarted","Data":"eed07bc5cb22dba122d621ec60ab6d369013979194423fba73bfedce00730db6"}
Sep 12 00:59:35.275791 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:35.275749    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"]
Sep 12 00:59:35.285686 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:35.285654    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"]
Sep 12 00:59:35.296455 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:35.296379    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"]
Sep 12 00:59:35.836498 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:35.836465    2567 generic.go:356] "Generic (PLEG): container finished" podID="403cdf9e-c1dc-4827-8b3b-a6d970713cee" containerID="ad4d0c1d675a7e7db03fb635bd89ea81ffe79009e9bc71c3064084a1d6eac8c4" exitCode=0
Sep 12 00:59:35.836723 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:35.836514    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b" event={"ID":"403cdf9e-c1dc-4827-8b3b-a6d970713cee","Type":"ContainerDied","Data":"ad4d0c1d675a7e7db03fb635bd89ea81ffe79009e9bc71c3064084a1d6eac8c4"}
Sep 12 00:59:36.980611 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:36.980581    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b"
Sep 12 00:59:37.079313 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:37.079215    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/403cdf9e-c1dc-4827-8b3b-a6d970713cee-maas-api-service-ca\") pod \"403cdf9e-c1dc-4827-8b3b-a6d970713cee\" (UID: \"403cdf9e-c1dc-4827-8b3b-a6d970713cee\") "
Sep 12 00:59:37.079621 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:37.079566    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403cdf9e-c1dc-4827-8b3b-a6d970713cee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "403cdf9e-c1dc-4827-8b3b-a6d970713cee" (UID: "403cdf9e-c1dc-4827-8b3b-a6d970713cee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:59:37.180021 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:37.179986    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/403cdf9e-c1dc-4827-8b3b-a6d970713cee-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:37.847920 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:37.847890    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b"
Sep 12 00:59:37.847920 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:37.847896    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b" event={"ID":"403cdf9e-c1dc-4827-8b3b-a6d970713cee","Type":"ContainerDied","Data":"eed07bc5cb22dba122d621ec60ab6d369013979194423fba73bfedce00730db6"}
Sep 12 00:59:37.848139 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:37.847942    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eed07bc5cb22dba122d621ec60ab6d369013979194423fba73bfedce00730db6"
Sep 12 00:59:44.999971 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:44.999936    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"]
Sep 12 00:59:45.000460 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.000141    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6" podUID="57c84d1e-44c3-4942-a621-63fe6c439c0d" containerName="payload-processing" containerID="cri-o://1c379e0e4f671e011d1ea4d23d10a22d5e509a58e35e6781fb0d1f5a211b8407" gracePeriod=30
Sep 12 00:59:45.009174 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.009146    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"]
Sep 12 00:59:45.009526 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.009495    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l" podUID="71d398c5-af9c-4f61-adbd-f5f0d945e00f" containerName="payload-pre-processing" containerID="cri-o://7f5284b8285959107914b74cf29f47c6dc2e7c2bd57c1db0ebf6a1e0e47d5194" gracePeriod=30
Sep 12 00:59:45.188999 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.188964    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b"]
Sep 12 00:59:45.191649 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.191622    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-088c83-cmh9b"]
Sep 12 00:59:45.286975 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.286949    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"
Sep 12 00:59:45.289714 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.289691    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"
Sep 12 00:59:45.359994 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.359956    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d398c5-af9c-4f61-adbd-f5f0d945e00f-plugins-config-volume\") pod \"71d398c5-af9c-4f61-adbd-f5f0d945e00f\" (UID: \"71d398c5-af9c-4f61-adbd-f5f0d945e00f\") "
Sep 12 00:59:45.360192 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.360171    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85phc\" (UniqueName: \"kubernetes.io/projected/71d398c5-af9c-4f61-adbd-f5f0d945e00f-kube-api-access-85phc\") pod \"71d398c5-af9c-4f61-adbd-f5f0d945e00f\" (UID: \"71d398c5-af9c-4f61-adbd-f5f0d945e00f\") "
Sep 12 00:59:45.360269 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.360207    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c84d1e-44c3-4942-a621-63fe6c439c0d-plugins-config-volume\") pod \"57c84d1e-44c3-4942-a621-63fe6c439c0d\" (UID: \"57c84d1e-44c3-4942-a621-63fe6c439c0d\") "
Sep 12 00:59:45.360329 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.360249    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qz5p\" (UniqueName: \"kubernetes.io/projected/57c84d1e-44c3-4942-a621-63fe6c439c0d-kube-api-access-5qz5p\") pod \"57c84d1e-44c3-4942-a621-63fe6c439c0d\" (UID: \"57c84d1e-44c3-4942-a621-63fe6c439c0d\") "
Sep 12 00:59:45.360390 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.360370    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d398c5-af9c-4f61-adbd-f5f0d945e00f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71d398c5-af9c-4f61-adbd-f5f0d945e00f" (UID: "71d398c5-af9c-4f61-adbd-f5f0d945e00f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:59:45.360691 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.360659    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c84d1e-44c3-4942-a621-63fe6c439c0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57c84d1e-44c3-4942-a621-63fe6c439c0d" (UID: "57c84d1e-44c3-4942-a621-63fe6c439c0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:59:45.360953 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.360932    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57c84d1e-44c3-4942-a621-63fe6c439c0d-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:45.361048 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.360958    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d398c5-af9c-4f61-adbd-f5f0d945e00f-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:45.362481 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.362448    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c84d1e-44c3-4942-a621-63fe6c439c0d-kube-api-access-5qz5p" (OuterVolumeSpecName: "kube-api-access-5qz5p") pod "57c84d1e-44c3-4942-a621-63fe6c439c0d" (UID: "57c84d1e-44c3-4942-a621-63fe6c439c0d"). InnerVolumeSpecName "kube-api-access-5qz5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:59:45.362591 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.362514    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d398c5-af9c-4f61-adbd-f5f0d945e00f-kube-api-access-85phc" (OuterVolumeSpecName: "kube-api-access-85phc") pod "71d398c5-af9c-4f61-adbd-f5f0d945e00f" (UID: "71d398c5-af9c-4f61-adbd-f5f0d945e00f"). InnerVolumeSpecName "kube-api-access-85phc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:59:45.462225 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.462183    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85phc\" (UniqueName: \"kubernetes.io/projected/71d398c5-af9c-4f61-adbd-f5f0d945e00f-kube-api-access-85phc\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:45.462225 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.462219    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qz5p\" (UniqueName: \"kubernetes.io/projected/57c84d1e-44c3-4942-a621-63fe6c439c0d-kube-api-access-5qz5p\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:45.890845 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.890807    2567 generic.go:356] "Generic (PLEG): container finished" podID="57c84d1e-44c3-4942-a621-63fe6c439c0d" containerID="1c379e0e4f671e011d1ea4d23d10a22d5e509a58e35e6781fb0d1f5a211b8407" exitCode=0
Sep 12 00:59:45.891052 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.890851    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"
Sep 12 00:59:45.891052 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.890889    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6" event={"ID":"57c84d1e-44c3-4942-a621-63fe6c439c0d","Type":"ContainerDied","Data":"1c379e0e4f671e011d1ea4d23d10a22d5e509a58e35e6781fb0d1f5a211b8407"}
Sep 12 00:59:45.891052 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.890945    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6" event={"ID":"57c84d1e-44c3-4942-a621-63fe6c439c0d","Type":"ContainerDied","Data":"8b3844f737bab0a13c579058562c9f44ceb3a6fc99c81d2e9f14a89dfa74d79d"}
Sep 12 00:59:45.891052 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.890977    2567 scope.go:117] "RemoveContainer" containerID="1c379e0e4f671e011d1ea4d23d10a22d5e509a58e35e6781fb0d1f5a211b8407"
Sep 12 00:59:45.892503 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.892478    2567 generic.go:356] "Generic (PLEG): container finished" podID="71d398c5-af9c-4f61-adbd-f5f0d945e00f" containerID="7f5284b8285959107914b74cf29f47c6dc2e7c2bd57c1db0ebf6a1e0e47d5194" exitCode=0
Sep 12 00:59:45.892673 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.892511    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l" event={"ID":"71d398c5-af9c-4f61-adbd-f5f0d945e00f","Type":"ContainerDied","Data":"7f5284b8285959107914b74cf29f47c6dc2e7c2bd57c1db0ebf6a1e0e47d5194"}
Sep 12 00:59:45.892673 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.892541    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"
Sep 12 00:59:45.892673 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.892564    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l" event={"ID":"71d398c5-af9c-4f61-adbd-f5f0d945e00f","Type":"ContainerDied","Data":"ded2ad2b1ab1cbdb36d1f0be71836d29bb306b0cc9ae5431597ea8e3d939994d"}
Sep 12 00:59:45.899942 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.899926    2567 scope.go:117] "RemoveContainer" containerID="1c379e0e4f671e011d1ea4d23d10a22d5e509a58e35e6781fb0d1f5a211b8407"
Sep 12 00:59:45.900238 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:59:45.900217    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c379e0e4f671e011d1ea4d23d10a22d5e509a58e35e6781fb0d1f5a211b8407\": container with ID starting with 1c379e0e4f671e011d1ea4d23d10a22d5e509a58e35e6781fb0d1f5a211b8407 not found: ID does not exist" containerID="1c379e0e4f671e011d1ea4d23d10a22d5e509a58e35e6781fb0d1f5a211b8407"
Sep 12 00:59:45.900337 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.900246    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c379e0e4f671e011d1ea4d23d10a22d5e509a58e35e6781fb0d1f5a211b8407"} err="failed to get container status \"1c379e0e4f671e011d1ea4d23d10a22d5e509a58e35e6781fb0d1f5a211b8407\": rpc error: code = NotFound desc = could not find container \"1c379e0e4f671e011d1ea4d23d10a22d5e509a58e35e6781fb0d1f5a211b8407\": container with ID starting with 1c379e0e4f671e011d1ea4d23d10a22d5e509a58e35e6781fb0d1f5a211b8407 not found: ID does not exist"
Sep 12 00:59:45.900337 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.900266    2567 scope.go:117] "RemoveContainer" containerID="7f5284b8285959107914b74cf29f47c6dc2e7c2bd57c1db0ebf6a1e0e47d5194"
Sep 12 00:59:45.908752 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.908680    2567 scope.go:117] "RemoveContainer" containerID="7f5284b8285959107914b74cf29f47c6dc2e7c2bd57c1db0ebf6a1e0e47d5194"
Sep 12 00:59:45.908949 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:59:45.908931    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5284b8285959107914b74cf29f47c6dc2e7c2bd57c1db0ebf6a1e0e47d5194\": container with ID starting with 7f5284b8285959107914b74cf29f47c6dc2e7c2bd57c1db0ebf6a1e0e47d5194 not found: ID does not exist" containerID="7f5284b8285959107914b74cf29f47c6dc2e7c2bd57c1db0ebf6a1e0e47d5194"
Sep 12 00:59:45.909009 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.908955    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5284b8285959107914b74cf29f47c6dc2e7c2bd57c1db0ebf6a1e0e47d5194"} err="failed to get container status \"7f5284b8285959107914b74cf29f47c6dc2e7c2bd57c1db0ebf6a1e0e47d5194\": rpc error: code = NotFound desc = could not find container \"7f5284b8285959107914b74cf29f47c6dc2e7c2bd57c1db0ebf6a1e0e47d5194\": container with ID starting with 7f5284b8285959107914b74cf29f47c6dc2e7c2bd57c1db0ebf6a1e0e47d5194 not found: ID does not exist"
Sep 12 00:59:45.916865 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.916844    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"]
Sep 12 00:59:45.926302 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.926277    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-088c83-9d4dbc6f7-rb8n6"]
Sep 12 00:59:45.941478 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.941454    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"]
Sep 12 00:59:45.945782 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:45.945753    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-088c83-68bc58bff8-8j22l"]
Sep 12 00:59:46.274026 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:46.273988    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="403cdf9e-c1dc-4827-8b3b-a6d970713cee" path="/var/lib/kubelet/pods/403cdf9e-c1dc-4827-8b3b-a6d970713cee/volumes"
Sep 12 00:59:46.274430 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:46.274357    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57c84d1e-44c3-4942-a621-63fe6c439c0d" path="/var/lib/kubelet/pods/57c84d1e-44c3-4942-a621-63fe6c439c0d/volumes"
Sep 12 00:59:46.274700 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:46.274686    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71d398c5-af9c-4f61-adbd-f5f0d945e00f" path="/var/lib/kubelet/pods/71d398c5-af9c-4f61-adbd-f5f0d945e00f/volumes"
Sep 12 00:59:47.330005 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:47.329955    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq"]
Sep 12 00:59:47.331980 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:47.331956    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4d485e-ds4lq"]
Sep 12 00:59:48.274334 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:48.274301    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e6be17-8ded-4a74-9d4b-e1cb46017aa6" path="/var/lib/kubelet/pods/d9e6be17-8ded-4a74-9d4b-e1cb46017aa6/volumes"
Sep 12 00:59:50.296000 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.295958    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc" podUID="f794a452-52ef-435e-86dc-8f6a39f7ba22" containerName="main" containerID="cri-o://f41dd5913b8a50af40a3996b351ea3ae6016b23c016e943ee8c6b4e4bba05653" gracePeriod=45
Sep 12 00:59:50.305958 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.305922    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm" podUID="d312b5d9-e69e-4c04-8c12-0003d3dd69e7" containerName="main" containerID="cri-o://10d8f7efdf2630412051250a5c73a18a36266d0a498bf4dc543ee6ec9abedfa2" gracePeriod=45
Sep 12 00:59:50.319077 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.319038    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t" podUID="2509fb91-4691-47f0-8985-7aa084a93cda" containerName="main" containerID="cri-o://62275a0c5676b8caee857c232533eac8731c2f1cb7b3546030c6ac14821d3a92" gracePeriod=45
Sep 12 00:59:50.581301 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.581262    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:59:50.608899 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.608864    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:59:50.624853 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.624824    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:59:50.717436 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717405    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-home\") pod \"2509fb91-4691-47f0-8985-7aa084a93cda\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") "
Sep 12 00:59:50.717638 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717446    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-dshm\") pod \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") "
Sep 12 00:59:50.717638 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717484    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-tmp-dir\") pod \"f794a452-52ef-435e-86dc-8f6a39f7ba22\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") "
Sep 12 00:59:50.717638 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717515    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-model-cache\") pod \"2509fb91-4691-47f0-8985-7aa084a93cda\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") "
Sep 12 00:59:50.717638 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717538    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-model-cache\") pod \"f794a452-52ef-435e-86dc-8f6a39f7ba22\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") "
Sep 12 00:59:50.717638 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717606    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-tmp-dir\") pod \"2509fb91-4691-47f0-8985-7aa084a93cda\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") "
Sep 12 00:59:50.717927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717657    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-tmp-dir\") pod \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") "
Sep 12 00:59:50.717927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717698    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkplv\" (UniqueName: \"kubernetes.io/projected/2509fb91-4691-47f0-8985-7aa084a93cda-kube-api-access-xkplv\") pod \"2509fb91-4691-47f0-8985-7aa084a93cda\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") "
Sep 12 00:59:50.717927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717729    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t7gf\" (UniqueName: \"kubernetes.io/projected/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-kube-api-access-2t7gf\") pod \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") "
Sep 12 00:59:50.717927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717752    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-model-cache\") pod \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") "
Sep 12 00:59:50.717927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717773    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-home\") pod \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") "
Sep 12 00:59:50.717927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717775    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-model-cache" (OuterVolumeSpecName: "model-cache") pod "2509fb91-4691-47f0-8985-7aa084a93cda" (UID: "2509fb91-4691-47f0-8985-7aa084a93cda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:50.717927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717787    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-home" (OuterVolumeSpecName: "home") pod "2509fb91-4691-47f0-8985-7aa084a93cda" (UID: "2509fb91-4691-47f0-8985-7aa084a93cda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:50.717927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717795    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-dshm\") pod \"f794a452-52ef-435e-86dc-8f6a39f7ba22\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") "
Sep 12 00:59:50.717927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717809    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f794a452-52ef-435e-86dc-8f6a39f7ba22" (UID: "f794a452-52ef-435e-86dc-8f6a39f7ba22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:50.717927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717818    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-model-cache" (OuterVolumeSpecName: "model-cache") pod "f794a452-52ef-435e-86dc-8f6a39f7ba22" (UID: "f794a452-52ef-435e-86dc-8f6a39f7ba22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:50.717927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717868    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2509fb91-4691-47f0-8985-7aa084a93cda" (UID: "2509fb91-4691-47f0-8985-7aa084a93cda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:50.717927 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717892    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p27qr\" (UniqueName: \"kubernetes.io/projected/f794a452-52ef-435e-86dc-8f6a39f7ba22-kube-api-access-p27qr\") pod \"f794a452-52ef-435e-86dc-8f6a39f7ba22\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") "
Sep 12 00:59:50.718554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717961    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-tls-certs\") pod \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\" (UID: \"d312b5d9-e69e-4c04-8c12-0003d3dd69e7\") "
Sep 12 00:59:50.718554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.717987    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-dshm\") pod \"2509fb91-4691-47f0-8985-7aa084a93cda\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") "
Sep 12 00:59:50.718554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718013    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f794a452-52ef-435e-86dc-8f6a39f7ba22-tls-certs\") pod \"f794a452-52ef-435e-86dc-8f6a39f7ba22\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") "
Sep 12 00:59:50.718554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718030    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "d312b5d9-e69e-4c04-8c12-0003d3dd69e7" (UID: "d312b5d9-e69e-4c04-8c12-0003d3dd69e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:50.718554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718037    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-home\") pod \"f794a452-52ef-435e-86dc-8f6a39f7ba22\" (UID: \"f794a452-52ef-435e-86dc-8f6a39f7ba22\") "
Sep 12 00:59:50.718554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718047    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d312b5d9-e69e-4c04-8c12-0003d3dd69e7" (UID: "d312b5d9-e69e-4c04-8c12-0003d3dd69e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:50.718554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718118    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2509fb91-4691-47f0-8985-7aa084a93cda-tls-certs\") pod \"2509fb91-4691-47f0-8985-7aa084a93cda\" (UID: \"2509fb91-4691-47f0-8985-7aa084a93cda\") "
Sep 12 00:59:50.718554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718199    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-home" (OuterVolumeSpecName: "home") pod "d312b5d9-e69e-4c04-8c12-0003d3dd69e7" (UID: "d312b5d9-e69e-4c04-8c12-0003d3dd69e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:50.718554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718225    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-home" (OuterVolumeSpecName: "home") pod "f794a452-52ef-435e-86dc-8f6a39f7ba22" (UID: "f794a452-52ef-435e-86dc-8f6a39f7ba22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:50.718554 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718538    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-home\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.719146 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718560    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-home\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.719146 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718592    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-tmp-dir\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.719146 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718611    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-model-cache\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.719146 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718625    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-model-cache\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.719146 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718638    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-tmp-dir\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.719146 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718652    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-tmp-dir\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.719146 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718665    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-model-cache\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.719146 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.718678    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-home\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.720530 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.720496    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-dshm" (OuterVolumeSpecName: "dshm") pod "f794a452-52ef-435e-86dc-8f6a39f7ba22" (UID: "f794a452-52ef-435e-86dc-8f6a39f7ba22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:50.720530 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.720503    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-dshm" (OuterVolumeSpecName: "dshm") pod "d312b5d9-e69e-4c04-8c12-0003d3dd69e7" (UID: "d312b5d9-e69e-4c04-8c12-0003d3dd69e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:50.720760 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.720635    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-kube-api-access-2t7gf" (OuterVolumeSpecName: "kube-api-access-2t7gf") pod "d312b5d9-e69e-4c04-8c12-0003d3dd69e7" (UID: "d312b5d9-e69e-4c04-8c12-0003d3dd69e7"). InnerVolumeSpecName "kube-api-access-2t7gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:59:50.720760 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.720669    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2509fb91-4691-47f0-8985-7aa084a93cda-kube-api-access-xkplv" (OuterVolumeSpecName: "kube-api-access-xkplv") pod "2509fb91-4691-47f0-8985-7aa084a93cda" (UID: "2509fb91-4691-47f0-8985-7aa084a93cda"). InnerVolumeSpecName "kube-api-access-xkplv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:59:50.720958 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.720933    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-dshm" (OuterVolumeSpecName: "dshm") pod "2509fb91-4691-47f0-8985-7aa084a93cda" (UID: "2509fb91-4691-47f0-8985-7aa084a93cda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:59:50.721083 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.720950    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2509fb91-4691-47f0-8985-7aa084a93cda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2509fb91-4691-47f0-8985-7aa084a93cda" (UID: "2509fb91-4691-47f0-8985-7aa084a93cda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:59:50.721595 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.721543    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f794a452-52ef-435e-86dc-8f6a39f7ba22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f794a452-52ef-435e-86dc-8f6a39f7ba22" (UID: "f794a452-52ef-435e-86dc-8f6a39f7ba22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:59:50.721727 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.721708    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d312b5d9-e69e-4c04-8c12-0003d3dd69e7" (UID: "d312b5d9-e69e-4c04-8c12-0003d3dd69e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:59:50.722094 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.722076    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f794a452-52ef-435e-86dc-8f6a39f7ba22-kube-api-access-p27qr" (OuterVolumeSpecName: "kube-api-access-p27qr") pod "f794a452-52ef-435e-86dc-8f6a39f7ba22" (UID: "f794a452-52ef-435e-86dc-8f6a39f7ba22"). InnerVolumeSpecName "kube-api-access-p27qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:59:50.820033 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.819923    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-dshm\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.820033 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.819964    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkplv\" (UniqueName: \"kubernetes.io/projected/2509fb91-4691-47f0-8985-7aa084a93cda-kube-api-access-xkplv\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.820033 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.819982    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t7gf\" (UniqueName: \"kubernetes.io/projected/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-kube-api-access-2t7gf\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.820033 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.819994    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f794a452-52ef-435e-86dc-8f6a39f7ba22-dshm\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.820033 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.820007    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p27qr\" (UniqueName: \"kubernetes.io/projected/f794a452-52ef-435e-86dc-8f6a39f7ba22-kube-api-access-p27qr\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.820033 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.820018    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d312b5d9-e69e-4c04-8c12-0003d3dd69e7-tls-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.820033 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.820029    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2509fb91-4691-47f0-8985-7aa084a93cda-dshm\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.820033 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.820040    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f794a452-52ef-435e-86dc-8f6a39f7ba22-tls-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.820402 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.820051    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2509fb91-4691-47f0-8985-7aa084a93cda-tls-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:50.921309 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.921274    2567 generic.go:356] "Generic (PLEG): container finished" podID="d312b5d9-e69e-4c04-8c12-0003d3dd69e7" containerID="10d8f7efdf2630412051250a5c73a18a36266d0a498bf4dc543ee6ec9abedfa2" exitCode=0
Sep 12 00:59:50.921493 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.921320    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"
Sep 12 00:59:50.921493 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.921365    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm" event={"ID":"d312b5d9-e69e-4c04-8c12-0003d3dd69e7","Type":"ContainerDied","Data":"10d8f7efdf2630412051250a5c73a18a36266d0a498bf4dc543ee6ec9abedfa2"}
Sep 12 00:59:50.921493 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.921417    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm" event={"ID":"d312b5d9-e69e-4c04-8c12-0003d3dd69e7","Type":"ContainerDied","Data":"a6717fe771f7fa0ca8bf09c28a1575f1a9df0e812f4f6fc190c983e07416c61b"}
Sep 12 00:59:50.921493 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.921439    2567 scope.go:117] "RemoveContainer" containerID="10d8f7efdf2630412051250a5c73a18a36266d0a498bf4dc543ee6ec9abedfa2"
Sep 12 00:59:50.922989 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.922964    2567 generic.go:356] "Generic (PLEG): container finished" podID="f794a452-52ef-435e-86dc-8f6a39f7ba22" containerID="f41dd5913b8a50af40a3996b351ea3ae6016b23c016e943ee8c6b4e4bba05653" exitCode=0
Sep 12 00:59:50.923112 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.923022    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"
Sep 12 00:59:50.923112 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.923047    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc" event={"ID":"f794a452-52ef-435e-86dc-8f6a39f7ba22","Type":"ContainerDied","Data":"f41dd5913b8a50af40a3996b351ea3ae6016b23c016e943ee8c6b4e4bba05653"}
Sep 12 00:59:50.923112 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.923089    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc" event={"ID":"f794a452-52ef-435e-86dc-8f6a39f7ba22","Type":"ContainerDied","Data":"4725db0eca0bb79ea3bc6c72a0f10f96fe1f11078d23050cb56ff245ccaad645"}
Sep 12 00:59:50.924842 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.924817    2567 generic.go:356] "Generic (PLEG): container finished" podID="2509fb91-4691-47f0-8985-7aa084a93cda" containerID="62275a0c5676b8caee857c232533eac8731c2f1cb7b3546030c6ac14821d3a92" exitCode=0
Sep 12 00:59:50.924955 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.924860    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"
Sep 12 00:59:50.924955 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.924860    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t" event={"ID":"2509fb91-4691-47f0-8985-7aa084a93cda","Type":"ContainerDied","Data":"62275a0c5676b8caee857c232533eac8731c2f1cb7b3546030c6ac14821d3a92"}
Sep 12 00:59:50.925034 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.924954    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t" event={"ID":"2509fb91-4691-47f0-8985-7aa084a93cda","Type":"ContainerDied","Data":"ff849e88075cbd9fb674b98feebf983d2b05403cd1a12d6bb30c614ad90cfbaa"}
Sep 12 00:59:50.932024 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.931997    2567 scope.go:117] "RemoveContainer" containerID="10d8f7efdf2630412051250a5c73a18a36266d0a498bf4dc543ee6ec9abedfa2"
Sep 12 00:59:50.932375 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:59:50.932353    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d8f7efdf2630412051250a5c73a18a36266d0a498bf4dc543ee6ec9abedfa2\": container with ID starting with 10d8f7efdf2630412051250a5c73a18a36266d0a498bf4dc543ee6ec9abedfa2 not found: ID does not exist" containerID="10d8f7efdf2630412051250a5c73a18a36266d0a498bf4dc543ee6ec9abedfa2"
Sep 12 00:59:50.932471 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.932383    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d8f7efdf2630412051250a5c73a18a36266d0a498bf4dc543ee6ec9abedfa2"} err="failed to get container status \"10d8f7efdf2630412051250a5c73a18a36266d0a498bf4dc543ee6ec9abedfa2\": rpc error: code = NotFound desc = could not find container \"10d8f7efdf2630412051250a5c73a18a36266d0a498bf4dc543ee6ec9abedfa2\": container with ID starting with 10d8f7efdf2630412051250a5c73a18a36266d0a498bf4dc543ee6ec9abedfa2 not found: ID does not exist"
Sep 12 00:59:50.932471 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.932400    2567 scope.go:117] "RemoveContainer" containerID="f41dd5913b8a50af40a3996b351ea3ae6016b23c016e943ee8c6b4e4bba05653"
Sep 12 00:59:50.941037 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.941020    2567 scope.go:117] "RemoveContainer" containerID="f41dd5913b8a50af40a3996b351ea3ae6016b23c016e943ee8c6b4e4bba05653"
Sep 12 00:59:50.941276 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:59:50.941260    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41dd5913b8a50af40a3996b351ea3ae6016b23c016e943ee8c6b4e4bba05653\": container with ID starting with f41dd5913b8a50af40a3996b351ea3ae6016b23c016e943ee8c6b4e4bba05653 not found: ID does not exist" containerID="f41dd5913b8a50af40a3996b351ea3ae6016b23c016e943ee8c6b4e4bba05653"
Sep 12 00:59:50.941329 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.941281    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41dd5913b8a50af40a3996b351ea3ae6016b23c016e943ee8c6b4e4bba05653"} err="failed to get container status \"f41dd5913b8a50af40a3996b351ea3ae6016b23c016e943ee8c6b4e4bba05653\": rpc error: code = NotFound desc = could not find container \"f41dd5913b8a50af40a3996b351ea3ae6016b23c016e943ee8c6b4e4bba05653\": container with ID starting with f41dd5913b8a50af40a3996b351ea3ae6016b23c016e943ee8c6b4e4bba05653 not found: ID does not exist"
Sep 12 00:59:50.941329 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.941296    2567 scope.go:117] "RemoveContainer" containerID="62275a0c5676b8caee857c232533eac8731c2f1cb7b3546030c6ac14821d3a92"
Sep 12 00:59:50.948120 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.948097    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"]
Sep 12 00:59:50.949972 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.949951    2567 scope.go:117] "RemoveContainer" containerID="62275a0c5676b8caee857c232533eac8731c2f1cb7b3546030c6ac14821d3a92"
Sep 12 00:59:50.950072 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.949986    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/premium-w0-4d485e-kserve-79479c44cd-g75nm"]
Sep 12 00:59:50.950285 ip-10-0-130-132 kubenswrapper[2567]: E0912 00:59:50.950264    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62275a0c5676b8caee857c232533eac8731c2f1cb7b3546030c6ac14821d3a92\": container with ID starting with 62275a0c5676b8caee857c232533eac8731c2f1cb7b3546030c6ac14821d3a92 not found: ID does not exist" containerID="62275a0c5676b8caee857c232533eac8731c2f1cb7b3546030c6ac14821d3a92"
Sep 12 00:59:50.950359 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.950291    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62275a0c5676b8caee857c232533eac8731c2f1cb7b3546030c6ac14821d3a92"} err="failed to get container status \"62275a0c5676b8caee857c232533eac8731c2f1cb7b3546030c6ac14821d3a92\": rpc error: code = NotFound desc = could not find container \"62275a0c5676b8caee857c232533eac8731c2f1cb7b3546030c6ac14821d3a92\": container with ID starting with 62275a0c5676b8caee857c232533eac8731c2f1cb7b3546030c6ac14821d3a92 not found: ID does not exist"
Sep 12 00:59:50.959694 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.959667    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"]
Sep 12 00:59:50.965237 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.965212    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/embedding-w0-4d485e-kserve-5bbccc5558-vnnlc"]
Sep 12 00:59:50.975947 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.975918    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"]
Sep 12 00:59:50.979283 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:50.979257    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4d485e/simulator-w0-4d485e-kserve-69d8cdc45f-gq86t"]
Sep 12 00:59:52.274231 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.274194    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2509fb91-4691-47f0-8985-7aa084a93cda" path="/var/lib/kubelet/pods/2509fb91-4691-47f0-8985-7aa084a93cda/volumes"
Sep 12 00:59:52.274650 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.274632    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d312b5d9-e69e-4c04-8c12-0003d3dd69e7" path="/var/lib/kubelet/pods/d312b5d9-e69e-4c04-8c12-0003d3dd69e7/volumes"
Sep 12 00:59:52.275006 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.274986    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f794a452-52ef-435e-86dc-8f6a39f7ba22" path="/var/lib/kubelet/pods/f794a452-52ef-435e-86dc-8f6a39f7ba22/volumes"
Sep 12 00:59:52.558367 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558252    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6"]
Sep 12 00:59:52.558754 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558734    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d312b5d9-e69e-4c04-8c12-0003d3dd69e7" containerName="main"
Sep 12 00:59:52.558861 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558756    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d312b5d9-e69e-4c04-8c12-0003d3dd69e7" containerName="main"
Sep 12 00:59:52.558861 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558767    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2509fb91-4691-47f0-8985-7aa084a93cda" containerName="main"
Sep 12 00:59:52.558861 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558774    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2509fb91-4691-47f0-8985-7aa084a93cda" containerName="main"
Sep 12 00:59:52.558861 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558783    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57c84d1e-44c3-4942-a621-63fe6c439c0d" containerName="payload-processing"
Sep 12 00:59:52.558861 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558791    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="57c84d1e-44c3-4942-a621-63fe6c439c0d" containerName="payload-processing"
Sep 12 00:59:52.558861 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558812    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d398c5-af9c-4f61-adbd-f5f0d945e00f" containerName="payload-pre-processing"
Sep 12 00:59:52.558861 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558822    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d398c5-af9c-4f61-adbd-f5f0d945e00f" containerName="payload-pre-processing"
Sep 12 00:59:52.558861 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558837    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="403cdf9e-c1dc-4827-8b3b-a6d970713cee" containerName="revoke-keys"
Sep 12 00:59:52.558861 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558845    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="403cdf9e-c1dc-4827-8b3b-a6d970713cee" containerName="revoke-keys"
Sep 12 00:59:52.559294 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558865    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f794a452-52ef-435e-86dc-8f6a39f7ba22" containerName="main"
Sep 12 00:59:52.559294 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558873    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f794a452-52ef-435e-86dc-8f6a39f7ba22" containerName="main"
Sep 12 00:59:52.559294 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.558988    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="71d398c5-af9c-4f61-adbd-f5f0d945e00f" containerName="payload-pre-processing"
Sep 12 00:59:52.559294 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.559008    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="403cdf9e-c1dc-4827-8b3b-a6d970713cee" containerName="revoke-keys"
Sep 12 00:59:52.559294 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.559018    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d312b5d9-e69e-4c04-8c12-0003d3dd69e7" containerName="main"
Sep 12 00:59:52.559294 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.559028    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="57c84d1e-44c3-4942-a621-63fe6c439c0d" containerName="payload-processing"
Sep 12 00:59:52.559294 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.559038    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2509fb91-4691-47f0-8985-7aa084a93cda" containerName="main"
Sep 12 00:59:52.559294 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.559046    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f794a452-52ef-435e-86dc-8f6a39f7ba22" containerName="main"
Sep 12 00:59:52.563621 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.563596    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6"
Sep 12 00:59:52.566501 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.566319    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:59:52.567049 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.567027    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 00:59:52.568747 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.568709    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6"]
Sep 12 00:59:52.639062 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.639015    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7235f1e1-4529-4620-9f3c-a619d7bae9c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6\" (UID: \"7235f1e1-4529-4620-9f3c-a619d7bae9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6"
Sep 12 00:59:52.740182 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.740144    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7235f1e1-4529-4620-9f3c-a619d7bae9c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6\" (UID: \"7235f1e1-4529-4620-9f3c-a619d7bae9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6"
Sep 12 00:59:52.740822 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.740803    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7235f1e1-4529-4620-9f3c-a619d7bae9c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6\" (UID: \"7235f1e1-4529-4620-9f3c-a619d7bae9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6"
Sep 12 00:59:52.876491 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:52.876410    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6"
Sep 12 00:59:53.153986 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:53.153951    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6"]
Sep 12 00:59:53.156477 ip-10-0-130-132 kubenswrapper[2567]: W0912 00:59:53.156445    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7235f1e1_4529_4620_9f3c_a619d7bae9c2.slice/crio-db36adf140282d4a31b169d34bf5071ac63c5f401d64f948460bb83b615977ed WatchSource:0}: Error finding container db36adf140282d4a31b169d34bf5071ac63c5f401d64f948460bb83b615977ed: Status 404 returned error can't find the container with id db36adf140282d4a31b169d34bf5071ac63c5f401d64f948460bb83b615977ed
Sep 12 00:59:53.977245 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:53.977210    2567 generic.go:356] "Generic (PLEG): container finished" podID="7235f1e1-4529-4620-9f3c-a619d7bae9c2" containerID="462bfac48f6bf5a56dca3fa710c6f1b09badf36e63aff2ebdd11ce71dfe55e68" exitCode=0
Sep 12 00:59:53.977690 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:53.977295    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6" event={"ID":"7235f1e1-4529-4620-9f3c-a619d7bae9c2","Type":"ContainerDied","Data":"462bfac48f6bf5a56dca3fa710c6f1b09badf36e63aff2ebdd11ce71dfe55e68"}
Sep 12 00:59:53.977690 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:53.977357    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6" event={"ID":"7235f1e1-4529-4620-9f3c-a619d7bae9c2","Type":"ContainerStarted","Data":"db36adf140282d4a31b169d34bf5071ac63c5f401d64f948460bb83b615977ed"}
Sep 12 00:59:55.116448 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:55.116415    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6"
Sep 12 00:59:55.279509 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:55.279416    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7235f1e1-4529-4620-9f3c-a619d7bae9c2-maas-api-service-ca\") pod \"7235f1e1-4529-4620-9f3c-a619d7bae9c2\" (UID: \"7235f1e1-4529-4620-9f3c-a619d7bae9c2\") "
Sep 12 00:59:55.279821 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:55.279794    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7235f1e1-4529-4620-9f3c-a619d7bae9c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7235f1e1-4529-4620-9f3c-a619d7bae9c2" (UID: "7235f1e1-4529-4620-9f3c-a619d7bae9c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:59:55.381020 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:55.380974    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7235f1e1-4529-4620-9f3c-a619d7bae9c2-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 00:59:55.989890 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:55.989860    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6"
Sep 12 00:59:55.990064 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:55.989860    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6" event={"ID":"7235f1e1-4529-4620-9f3c-a619d7bae9c2","Type":"ContainerDied","Data":"db36adf140282d4a31b169d34bf5071ac63c5f401d64f948460bb83b615977ed"}
Sep 12 00:59:55.990064 ip-10-0-130-132 kubenswrapper[2567]: I0912 00:59:55.989986    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db36adf140282d4a31b169d34bf5071ac63c5f401d64f948460bb83b615977ed"
Sep 12 01:00:00.141100 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.141056    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj"]
Sep 12 01:00:00.141633 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.141607    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7235f1e1-4529-4620-9f3c-a619d7bae9c2" containerName="revoke-keys"
Sep 12 01:00:00.141633 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.141628    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7235f1e1-4529-4620-9f3c-a619d7bae9c2" containerName="revoke-keys"
Sep 12 01:00:00.141781 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.141767    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7235f1e1-4529-4620-9f3c-a619d7bae9c2" containerName="revoke-keys"
Sep 12 01:00:00.144737 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.144715    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks"]
Sep 12 01:00:00.144857 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.144826    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj"
Sep 12 01:00:00.147376 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.147354    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 01:00:00.147737 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.147719    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819580-x5rmz"]
Sep 12 01:00:00.147842 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.147824    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks"
Sep 12 01:00:00.151041 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.150895    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-x5rmz"
Sep 12 01:00:00.153007 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.152981    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj"]
Sep 12 01:00:00.156127 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.156104    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks"]
Sep 12 01:00:00.157418 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.157392    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj"
Sep 12 01:00:00.167360 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.167332    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks"
Sep 12 01:00:00.171172 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.171144    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819580-x5rmz"]
Sep 12 01:00:00.204964 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.204936    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-x5rmz"
Sep 12 01:00:00.439861 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.439815    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj"]
Sep 12 01:00:00.442321 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:00:00.442292    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d03319_a8e3_45c4_98c9_76f00bee2402.slice/crio-b8e361b1b5e2b10879a016cb79fc0a04cf036b1df86f80acf88369af927a2b9d WatchSource:0}: Error finding container b8e361b1b5e2b10879a016cb79fc0a04cf036b1df86f80acf88369af927a2b9d: Status 404 returned error can't find the container with id b8e361b1b5e2b10879a016cb79fc0a04cf036b1df86f80acf88369af927a2b9d
Sep 12 01:00:00.463303 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.463270    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks"]
Sep 12 01:00:00.466556 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:00:00.466517    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6666a48d_595f_4b08_bf45_7ed70c9f0a05.slice/crio-9fb173347c4b854f01eff6e9931733c0e052b2c09533f65b8e1ab09e38e452e5 WatchSource:0}: Error finding container 9fb173347c4b854f01eff6e9931733c0e052b2c09533f65b8e1ab09e38e452e5: Status 404 returned error can't find the container with id 9fb173347c4b854f01eff6e9931733c0e052b2c09533f65b8e1ab09e38e452e5
Sep 12 01:00:00.499597 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:00.499551    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819580-x5rmz"]
Sep 12 01:00:00.502162 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:00:00.502136    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15625d32_7a10_4b8c_80b3_b344a80855cb.slice/crio-e9e3c31ce9e73f4587aa60285a989807219fd1add4a45ac577024fdc3f13b726 WatchSource:0}: Error finding container e9e3c31ce9e73f4587aa60285a989807219fd1add4a45ac577024fdc3f13b726: Status 404 returned error can't find the container with id e9e3c31ce9e73f4587aa60285a989807219fd1add4a45ac577024fdc3f13b726
Sep 12 01:00:01.018281 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:01.018173    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-x5rmz" event={"ID":"15625d32-7a10-4b8c-80b3-b344a80855cb","Type":"ContainerStarted","Data":"e6605874df2ca0cb32d39097566c9d8ffae1aafb58307516e8c4ba97d240bc81"}
Sep 12 01:00:01.018281 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:01.018238    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-x5rmz" event={"ID":"15625d32-7a10-4b8c-80b3-b344a80855cb","Type":"ContainerStarted","Data":"e9e3c31ce9e73f4587aa60285a989807219fd1add4a45ac577024fdc3f13b726"}
Sep 12 01:00:01.019810 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:01.019783    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks" event={"ID":"6666a48d-595f-4b08-bf45-7ed70c9f0a05","Type":"ContainerStarted","Data":"e27755ed9f9bd3a135cc15166e79cc87d652ca7a1bf266b53eea18c9f929d739"}
Sep 12 01:00:01.019953 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:01.019831    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks" event={"ID":"6666a48d-595f-4b08-bf45-7ed70c9f0a05","Type":"ContainerStarted","Data":"9fb173347c4b854f01eff6e9931733c0e052b2c09533f65b8e1ab09e38e452e5"}
Sep 12 01:00:01.021281 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:01.021256    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj" event={"ID":"d6d03319-a8e3-45c4-98c9-76f00bee2402","Type":"ContainerStarted","Data":"eb56aacd7e69d8c87482f23e2115750303310066fa72154dc6cd9c733285eca1"}
Sep 12 01:00:01.021418 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:01.021298    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj" event={"ID":"d6d03319-a8e3-45c4-98c9-76f00bee2402","Type":"ContainerStarted","Data":"b8e361b1b5e2b10879a016cb79fc0a04cf036b1df86f80acf88369af927a2b9d"}
Sep 12 01:00:01.033806 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:01.033758    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-x5rmz" podStartSLOduration=1.033743095 podStartE2EDuration="1.033743095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:00:01.032423169 +0000 UTC m=+1833.236423993" watchObservedRunningTime="2026-09-12 01:00:01.033743095 +0000 UTC m=+1833.237743916"
Sep 12 01:00:01.052730 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:01.052674    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj" podStartSLOduration=1.052656744 podStartE2EDuration="1.052656744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:00:01.050533621 +0000 UTC m=+1833.254534443" watchObservedRunningTime="2026-09-12 01:00:01.052656744 +0000 UTC m=+1833.256657562"
Sep 12 01:00:01.066674 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:01.066612    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks" podStartSLOduration=1.066597693 podStartE2EDuration="1.066597693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:00:01.065614196 +0000 UTC m=+1833.269615020" watchObservedRunningTime="2026-09-12 01:00:01.066597693 +0000 UTC m=+1833.270598509"
Sep 12 01:00:02.027587 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:02.027532    2567 generic.go:356] "Generic (PLEG): container finished" podID="15625d32-7a10-4b8c-80b3-b344a80855cb" containerID="e6605874df2ca0cb32d39097566c9d8ffae1aafb58307516e8c4ba97d240bc81" exitCode=0
Sep 12 01:00:02.028013 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:02.027628    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-x5rmz" event={"ID":"15625d32-7a10-4b8c-80b3-b344a80855cb","Type":"ContainerDied","Data":"e6605874df2ca0cb32d39097566c9d8ffae1aafb58307516e8c4ba97d240bc81"}
Sep 12 01:00:02.029237 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:02.029215    2567 generic.go:356] "Generic (PLEG): container finished" podID="6666a48d-595f-4b08-bf45-7ed70c9f0a05" containerID="e27755ed9f9bd3a135cc15166e79cc87d652ca7a1bf266b53eea18c9f929d739" exitCode=0
Sep 12 01:00:02.029356 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:02.029295    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks" event={"ID":"6666a48d-595f-4b08-bf45-7ed70c9f0a05","Type":"ContainerDied","Data":"e27755ed9f9bd3a135cc15166e79cc87d652ca7a1bf266b53eea18c9f929d739"}
Sep 12 01:00:02.030732 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:02.030710    2567 generic.go:356] "Generic (PLEG): container finished" podID="d6d03319-a8e3-45c4-98c9-76f00bee2402" containerID="eb56aacd7e69d8c87482f23e2115750303310066fa72154dc6cd9c733285eca1" exitCode=0
Sep 12 01:00:02.030851 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:02.030768    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj" event={"ID":"d6d03319-a8e3-45c4-98c9-76f00bee2402","Type":"ContainerDied","Data":"eb56aacd7e69d8c87482f23e2115750303310066fa72154dc6cd9c733285eca1"}
Sep 12 01:00:03.174773 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.174741    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks"
Sep 12 01:00:03.232771 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.232744    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj"
Sep 12 01:00:03.236496 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.236476    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-x5rmz"
Sep 12 01:00:03.351054 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.350953    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"]
Sep 12 01:00:03.351243 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.351196    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb" podUID="6b583281-a094-4abe-9a47-da7dea65f629" containerName="payload-processing" containerID="cri-o://495cbc9fadf7f638a55c0aa08458db3db804bec0ce90bb8a27688aa91bca687f" gracePeriod=30
Sep 12 01:00:03.360092 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.360064    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"]
Sep 12 01:00:03.360371 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.360329    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql" podUID="1d9321a3-4b8e-4f87-b165-4cb4000ec575" containerName="payload-pre-processing" containerID="cri-o://77e2cc695703a10814386d0c00529fe8c0c5cac5580b8c55a30cb8156dd27cdb" gracePeriod=30
Sep 12 01:00:03.551032 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.550991    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6"]
Sep 12 01:00:03.558536 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.558502    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c3ca75-j2qm6"]
Sep 12 01:00:03.617754 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.617717    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"
Sep 12 01:00:03.633117 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.633088    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"
Sep 12 01:00:03.663824 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.663795    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr6lh\" (UniqueName: \"kubernetes.io/projected/1d9321a3-4b8e-4f87-b165-4cb4000ec575-kube-api-access-vr6lh\") pod \"1d9321a3-4b8e-4f87-b165-4cb4000ec575\" (UID: \"1d9321a3-4b8e-4f87-b165-4cb4000ec575\") "
Sep 12 01:00:03.664010 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.663990    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b583281-a094-4abe-9a47-da7dea65f629-plugins-config-volume\") pod \"6b583281-a094-4abe-9a47-da7dea65f629\" (UID: \"6b583281-a094-4abe-9a47-da7dea65f629\") "
Sep 12 01:00:03.664063 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.664050    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d9321a3-4b8e-4f87-b165-4cb4000ec575-plugins-config-volume\") pod \"1d9321a3-4b8e-4f87-b165-4cb4000ec575\" (UID: \"1d9321a3-4b8e-4f87-b165-4cb4000ec575\") "
Sep 12 01:00:03.664197 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.664160    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzmrn\" (UniqueName: \"kubernetes.io/projected/6b583281-a094-4abe-9a47-da7dea65f629-kube-api-access-hzmrn\") pod \"6b583281-a094-4abe-9a47-da7dea65f629\" (UID: \"6b583281-a094-4abe-9a47-da7dea65f629\") "
Sep 12 01:00:03.664413 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.664390    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b583281-a094-4abe-9a47-da7dea65f629-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b583281-a094-4abe-9a47-da7dea65f629" (UID: "6b583281-a094-4abe-9a47-da7dea65f629"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:00:03.664500 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.664409    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9321a3-4b8e-4f87-b165-4cb4000ec575-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d9321a3-4b8e-4f87-b165-4cb4000ec575" (UID: "1d9321a3-4b8e-4f87-b165-4cb4000ec575"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:00:03.664593 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.664533    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d9321a3-4b8e-4f87-b165-4cb4000ec575-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:03.664593 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.664549    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b583281-a094-4abe-9a47-da7dea65f629-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:03.665959 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.665938    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9321a3-4b8e-4f87-b165-4cb4000ec575-kube-api-access-vr6lh" (OuterVolumeSpecName: "kube-api-access-vr6lh") pod "1d9321a3-4b8e-4f87-b165-4cb4000ec575" (UID: "1d9321a3-4b8e-4f87-b165-4cb4000ec575"). InnerVolumeSpecName "kube-api-access-vr6lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:00:03.666197 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.666177    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b583281-a094-4abe-9a47-da7dea65f629-kube-api-access-hzmrn" (OuterVolumeSpecName: "kube-api-access-hzmrn") pod "6b583281-a094-4abe-9a47-da7dea65f629" (UID: "6b583281-a094-4abe-9a47-da7dea65f629"). InnerVolumeSpecName "kube-api-access-hzmrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:00:03.765843 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.765795    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzmrn\" (UniqueName: \"kubernetes.io/projected/6b583281-a094-4abe-9a47-da7dea65f629-kube-api-access-hzmrn\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:03.765843 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:03.765834    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr6lh\" (UniqueName: \"kubernetes.io/projected/1d9321a3-4b8e-4f87-b165-4cb4000ec575-kube-api-access-vr6lh\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:04.043586 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.043532    2567 generic.go:356] "Generic (PLEG): container finished" podID="6b583281-a094-4abe-9a47-da7dea65f629" containerID="495cbc9fadf7f638a55c0aa08458db3db804bec0ce90bb8a27688aa91bca687f" exitCode=0
Sep 12 01:00:04.043777 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.043595    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"
Sep 12 01:00:04.043777 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.043605    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb" event={"ID":"6b583281-a094-4abe-9a47-da7dea65f629","Type":"ContainerDied","Data":"495cbc9fadf7f638a55c0aa08458db3db804bec0ce90bb8a27688aa91bca687f"}
Sep 12 01:00:04.043777 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.043679    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb" event={"ID":"6b583281-a094-4abe-9a47-da7dea65f629","Type":"ContainerDied","Data":"856e9c020085762d505e6c84646b5a6e38c7bef659740d354fd580b405bf1eef"}
Sep 12 01:00:04.043777 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.043713    2567 scope.go:117] "RemoveContainer" containerID="495cbc9fadf7f638a55c0aa08458db3db804bec0ce90bb8a27688aa91bca687f"
Sep 12 01:00:04.045176 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.045147    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj"
Sep 12 01:00:04.045254 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.045167    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj" event={"ID":"d6d03319-a8e3-45c4-98c9-76f00bee2402","Type":"ContainerDied","Data":"b8e361b1b5e2b10879a016cb79fc0a04cf036b1df86f80acf88369af927a2b9d"}
Sep 12 01:00:04.045254 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.045226    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8e361b1b5e2b10879a016cb79fc0a04cf036b1df86f80acf88369af927a2b9d"
Sep 12 01:00:04.046688 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.046661    2567 generic.go:356] "Generic (PLEG): container finished" podID="1d9321a3-4b8e-4f87-b165-4cb4000ec575" containerID="77e2cc695703a10814386d0c00529fe8c0c5cac5580b8c55a30cb8156dd27cdb" exitCode=0
Sep 12 01:00:04.046836 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.046691    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql" event={"ID":"1d9321a3-4b8e-4f87-b165-4cb4000ec575","Type":"ContainerDied","Data":"77e2cc695703a10814386d0c00529fe8c0c5cac5580b8c55a30cb8156dd27cdb"}
Sep 12 01:00:04.046836 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.046707    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"
Sep 12 01:00:04.046836 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.046732    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql" event={"ID":"1d9321a3-4b8e-4f87-b165-4cb4000ec575","Type":"ContainerDied","Data":"3be40cc400b97e6ac598227f8d6963f217bad5c7f3bcbad007dc183b1666e478"}
Sep 12 01:00:04.048795 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.048278    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-x5rmz"
Sep 12 01:00:04.048795 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.048308    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-x5rmz" event={"ID":"15625d32-7a10-4b8c-80b3-b344a80855cb","Type":"ContainerDied","Data":"e9e3c31ce9e73f4587aa60285a989807219fd1add4a45ac577024fdc3f13b726"}
Sep 12 01:00:04.048795 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.048356    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e3c31ce9e73f4587aa60285a989807219fd1add4a45ac577024fdc3f13b726"
Sep 12 01:00:04.052837 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.049971    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks" event={"ID":"6666a48d-595f-4b08-bf45-7ed70c9f0a05","Type":"ContainerDied","Data":"9fb173347c4b854f01eff6e9931733c0e052b2c09533f65b8e1ab09e38e452e5"}
Sep 12 01:00:04.052837 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.049992    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks"
Sep 12 01:00:04.052837 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.050014    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fb173347c4b854f01eff6e9931733c0e052b2c09533f65b8e1ab09e38e452e5"
Sep 12 01:00:04.055711 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.055666    2567 scope.go:117] "RemoveContainer" containerID="495cbc9fadf7f638a55c0aa08458db3db804bec0ce90bb8a27688aa91bca687f"
Sep 12 01:00:04.056024 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:00:04.055998    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495cbc9fadf7f638a55c0aa08458db3db804bec0ce90bb8a27688aa91bca687f\": container with ID starting with 495cbc9fadf7f638a55c0aa08458db3db804bec0ce90bb8a27688aa91bca687f not found: ID does not exist" containerID="495cbc9fadf7f638a55c0aa08458db3db804bec0ce90bb8a27688aa91bca687f"
Sep 12 01:00:04.056092 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.056031    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495cbc9fadf7f638a55c0aa08458db3db804bec0ce90bb8a27688aa91bca687f"} err="failed to get container status \"495cbc9fadf7f638a55c0aa08458db3db804bec0ce90bb8a27688aa91bca687f\": rpc error: code = NotFound desc = could not find container \"495cbc9fadf7f638a55c0aa08458db3db804bec0ce90bb8a27688aa91bca687f\": container with ID starting with 495cbc9fadf7f638a55c0aa08458db3db804bec0ce90bb8a27688aa91bca687f not found: ID does not exist"
Sep 12 01:00:04.056092 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.056054    2567 scope.go:117] "RemoveContainer" containerID="77e2cc695703a10814386d0c00529fe8c0c5cac5580b8c55a30cb8156dd27cdb"
Sep 12 01:00:04.073843 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.073821    2567 scope.go:117] "RemoveContainer" containerID="77e2cc695703a10814386d0c00529fe8c0c5cac5580b8c55a30cb8156dd27cdb"
Sep 12 01:00:04.074410 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:00:04.074353    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e2cc695703a10814386d0c00529fe8c0c5cac5580b8c55a30cb8156dd27cdb\": container with ID starting with 77e2cc695703a10814386d0c00529fe8c0c5cac5580b8c55a30cb8156dd27cdb not found: ID does not exist" containerID="77e2cc695703a10814386d0c00529fe8c0c5cac5580b8c55a30cb8156dd27cdb"
Sep 12 01:00:04.074674 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.074638    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e2cc695703a10814386d0c00529fe8c0c5cac5580b8c55a30cb8156dd27cdb"} err="failed to get container status \"77e2cc695703a10814386d0c00529fe8c0c5cac5580b8c55a30cb8156dd27cdb\": rpc error: code = NotFound desc = could not find container \"77e2cc695703a10814386d0c00529fe8c0c5cac5580b8c55a30cb8156dd27cdb\": container with ID starting with 77e2cc695703a10814386d0c00529fe8c0c5cac5580b8c55a30cb8156dd27cdb not found: ID does not exist"
Sep 12 01:00:04.077002 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.076978    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"]
Sep 12 01:00:04.080442 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.080420    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c3ca75-797c7b55d7-nfhkb"]
Sep 12 01:00:04.100787 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.100744    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"]
Sep 12 01:00:04.105392 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.105369    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c3ca75-5446f78574-8twql"]
Sep 12 01:00:04.273631 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.273600    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d9321a3-4b8e-4f87-b165-4cb4000ec575" path="/var/lib/kubelet/pods/1d9321a3-4b8e-4f87-b165-4cb4000ec575/volumes"
Sep 12 01:00:04.273993 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.273969    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b583281-a094-4abe-9a47-da7dea65f629" path="/var/lib/kubelet/pods/6b583281-a094-4abe-9a47-da7dea65f629/volumes"
Sep 12 01:00:04.274277 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:04.274261    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7235f1e1-4529-4620-9f3c-a619d7bae9c2" path="/var/lib/kubelet/pods/7235f1e1-4529-4620-9f3c-a619d7bae9c2/volumes"
Sep 12 01:00:23.391155 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391121    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz"]
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391454    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9321a3-4b8e-4f87-b165-4cb4000ec575" containerName="payload-pre-processing"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391465    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9321a3-4b8e-4f87-b165-4cb4000ec575" containerName="payload-pre-processing"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391477    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15625d32-7a10-4b8c-80b3-b344a80855cb" containerName="cleanup"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391482    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="15625d32-7a10-4b8c-80b3-b344a80855cb" containerName="cleanup"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391494    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6d03319-a8e3-45c4-98c9-76f00bee2402" containerName="cleanup"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391501    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6d03319-a8e3-45c4-98c9-76f00bee2402" containerName="cleanup"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391513    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6666a48d-595f-4b08-bf45-7ed70c9f0a05" containerName="cleanup"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391519    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6666a48d-595f-4b08-bf45-7ed70c9f0a05" containerName="cleanup"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391526    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b583281-a094-4abe-9a47-da7dea65f629" containerName="payload-processing"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391531    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b583281-a094-4abe-9a47-da7dea65f629" containerName="payload-processing"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391595    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9321a3-4b8e-4f87-b165-4cb4000ec575" containerName="payload-pre-processing"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391610    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b583281-a094-4abe-9a47-da7dea65f629" containerName="payload-processing"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391617    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6666a48d-595f-4b08-bf45-7ed70c9f0a05" containerName="cleanup"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391627    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6d03319-a8e3-45c4-98c9-76f00bee2402" containerName="cleanup"
Sep 12 01:00:23.391698 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.391637    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="15625d32-7a10-4b8c-80b3-b344a80855cb" containerName="cleanup"
Sep 12 01:00:23.396018 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.395999    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz"
Sep 12 01:00:23.398874 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.398708    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:00:23.399695 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.399674    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 01:00:23.401389 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.401361    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz"]
Sep 12 01:00:23.460822 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.460785    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43265a00-f4b7-4e6c-bf13-99bca2d4c199-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz\" (UID: \"43265a00-f4b7-4e6c-bf13-99bca2d4c199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz"
Sep 12 01:00:23.561525 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.561466    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43265a00-f4b7-4e6c-bf13-99bca2d4c199-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz\" (UID: \"43265a00-f4b7-4e6c-bf13-99bca2d4c199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz"
Sep 12 01:00:23.562164 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.562143    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43265a00-f4b7-4e6c-bf13-99bca2d4c199-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz\" (UID: \"43265a00-f4b7-4e6c-bf13-99bca2d4c199\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz"
Sep 12 01:00:23.708870 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.708775    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz"
Sep 12 01:00:23.981957 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:23.981928    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz"]
Sep 12 01:00:23.984070 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:00:23.984041    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43265a00_f4b7_4e6c_bf13_99bca2d4c199.slice/crio-161bdcc3421a9647c84c79b437b432245887c7256b816b081a7723b11e27d3e3 WatchSource:0}: Error finding container 161bdcc3421a9647c84c79b437b432245887c7256b816b081a7723b11e27d3e3: Status 404 returned error can't find the container with id 161bdcc3421a9647c84c79b437b432245887c7256b816b081a7723b11e27d3e3
Sep 12 01:00:24.158464 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:24.158434    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz" event={"ID":"43265a00-f4b7-4e6c-bf13-99bca2d4c199","Type":"ContainerStarted","Data":"161bdcc3421a9647c84c79b437b432245887c7256b816b081a7723b11e27d3e3"}
Sep 12 01:00:25.165319 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:25.165284    2567 generic.go:356] "Generic (PLEG): container finished" podID="43265a00-f4b7-4e6c-bf13-99bca2d4c199" containerID="2dd70412cd7a9a8e90e22fc33f7179c8808c209c2a7b466f6a077a303a017289" exitCode=0
Sep 12 01:00:25.165751 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:25.165371    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz" event={"ID":"43265a00-f4b7-4e6c-bf13-99bca2d4c199","Type":"ContainerDied","Data":"2dd70412cd7a9a8e90e22fc33f7179c8808c209c2a7b466f6a077a303a017289"}
Sep 12 01:00:26.302058 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:26.302026    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz"
Sep 12 01:00:26.389997 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:26.389961    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43265a00-f4b7-4e6c-bf13-99bca2d4c199-maas-api-service-ca\") pod \"43265a00-f4b7-4e6c-bf13-99bca2d4c199\" (UID: \"43265a00-f4b7-4e6c-bf13-99bca2d4c199\") "
Sep 12 01:00:26.390421 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:26.390383    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43265a00-f4b7-4e6c-bf13-99bca2d4c199-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43265a00-f4b7-4e6c-bf13-99bca2d4c199" (UID: "43265a00-f4b7-4e6c-bf13-99bca2d4c199"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:00:26.490921 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:26.490881    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43265a00-f4b7-4e6c-bf13-99bca2d4c199-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:27.177772 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:27.177739    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz"
Sep 12 01:00:27.177945 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:27.177770    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz" event={"ID":"43265a00-f4b7-4e6c-bf13-99bca2d4c199","Type":"ContainerDied","Data":"161bdcc3421a9647c84c79b437b432245887c7256b816b081a7723b11e27d3e3"}
Sep 12 01:00:27.177945 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:27.177820    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="161bdcc3421a9647c84c79b437b432245887c7256b816b081a7723b11e27d3e3"
Sep 12 01:00:28.265218 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:28.265182    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"]
Sep 12 01:00:28.274878 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:28.274850    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"]
Sep 12 01:00:28.283222 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:28.283190    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"]
Sep 12 01:00:38.972535 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:38.972494    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj"]
Sep 12 01:00:38.975247 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:38.975214    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-876808-29819580-4mrvj"]
Sep 12 01:00:39.169978 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:39.169944    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz"]
Sep 12 01:00:39.175408 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:39.175376    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-876808-rnqsz"]
Sep 12 01:00:40.274033 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:40.273993    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43265a00-f4b7-4e6c-bf13-99bca2d4c199" path="/var/lib/kubelet/pods/43265a00-f4b7-4e6c-bf13-99bca2d4c199/volumes"
Sep 12 01:00:40.274439 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:40.274299    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6d03319-a8e3-45c4-98c9-76f00bee2402" path="/var/lib/kubelet/pods/d6d03319-a8e3-45c4-98c9-76f00bee2402/volumes"
Sep 12 01:00:43.285650 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.285566    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7" podUID="827f9d20-74d3-4601-a68a-927b164ff00f" containerName="main" containerID="cri-o://4c2a54e26fd47091629632cb5923b54fb06165159b309b11d1659feac9f40c4f" gracePeriod=45
Sep 12 01:00:43.294012 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.293929    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69" podUID="ed476580-cb27-4a37-9ba4-f117f62232b6" containerName="main" containerID="cri-o://de5743abfdb8b741497a3e27c9edc858c504cada77e71cc136809867e7ccad4e" gracePeriod=45
Sep 12 01:00:43.305021 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.304981    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg" podUID="5f441720-a3c0-4875-9313-26f87334f860" containerName="main" containerID="cri-o://152319cf398337aafb839a7d6f3f7b6e66279a475096b80c460ce18e96e781e5" gracePeriod=45
Sep 12 01:00:43.592974 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.592890    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 01:00:43.596887 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.596853    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 01:00:43.605107 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.605079    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 01:00:43.657304 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657270    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m8lq\" (UniqueName: \"kubernetes.io/projected/ed476580-cb27-4a37-9ba4-f117f62232b6-kube-api-access-2m8lq\") pod \"ed476580-cb27-4a37-9ba4-f117f62232b6\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") "
Sep 12 01:00:43.657540 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657319    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-home\") pod \"ed476580-cb27-4a37-9ba4-f117f62232b6\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") "
Sep 12 01:00:43.657540 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657348    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-dshm\") pod \"ed476580-cb27-4a37-9ba4-f117f62232b6\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") "
Sep 12 01:00:43.657540 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657383    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-dshm\") pod \"5f441720-a3c0-4875-9313-26f87334f860\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") "
Sep 12 01:00:43.657540 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657409    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed476580-cb27-4a37-9ba4-f117f62232b6-tls-certs\") pod \"ed476580-cb27-4a37-9ba4-f117f62232b6\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") "
Sep 12 01:00:43.657540 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657435    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-tmp-dir\") pod \"ed476580-cb27-4a37-9ba4-f117f62232b6\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") "
Sep 12 01:00:43.657540 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657463    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-tmp-dir\") pod \"5f441720-a3c0-4875-9313-26f87334f860\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") "
Sep 12 01:00:43.657540 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657493    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hr5f\" (UniqueName: \"kubernetes.io/projected/827f9d20-74d3-4601-a68a-927b164ff00f-kube-api-access-5hr5f\") pod \"827f9d20-74d3-4601-a68a-927b164ff00f\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") "
Sep 12 01:00:43.657540 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657520    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdmlm\" (UniqueName: \"kubernetes.io/projected/5f441720-a3c0-4875-9313-26f87334f860-kube-api-access-qdmlm\") pod \"5f441720-a3c0-4875-9313-26f87334f860\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") "
Sep 12 01:00:43.658006 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657635    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f441720-a3c0-4875-9313-26f87334f860-tls-certs\") pod \"5f441720-a3c0-4875-9313-26f87334f860\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") "
Sep 12 01:00:43.658006 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657649    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-home" (OuterVolumeSpecName: "home") pod "ed476580-cb27-4a37-9ba4-f117f62232b6" (UID: "ed476580-cb27-4a37-9ba4-f117f62232b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:43.658006 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657672    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-dshm\") pod \"827f9d20-74d3-4601-a68a-927b164ff00f\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") "
Sep 12 01:00:43.658006 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657704    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-home\") pod \"827f9d20-74d3-4601-a68a-927b164ff00f\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") "
Sep 12 01:00:43.658006 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657751    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-model-cache\") pod \"827f9d20-74d3-4601-a68a-927b164ff00f\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") "
Sep 12 01:00:43.658006 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657802    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827f9d20-74d3-4601-a68a-927b164ff00f-tls-certs\") pod \"827f9d20-74d3-4601-a68a-927b164ff00f\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") "
Sep 12 01:00:43.658006 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657831    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-home\") pod \"5f441720-a3c0-4875-9313-26f87334f860\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") "
Sep 12 01:00:43.658006 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657860    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-tmp-dir\") pod \"827f9d20-74d3-4601-a68a-927b164ff00f\" (UID: \"827f9d20-74d3-4601-a68a-927b164ff00f\") "
Sep 12 01:00:43.658006 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657891    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-model-cache\") pod \"ed476580-cb27-4a37-9ba4-f117f62232b6\" (UID: \"ed476580-cb27-4a37-9ba4-f117f62232b6\") "
Sep 12 01:00:43.658006 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657940    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-model-cache\") pod \"5f441720-a3c0-4875-9313-26f87334f860\" (UID: \"5f441720-a3c0-4875-9313-26f87334f860\") "
Sep 12 01:00:43.658494 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.658316    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-home\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.660009 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657831    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f441720-a3c0-4875-9313-26f87334f860" (UID: "5f441720-a3c0-4875-9313-26f87334f860"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:43.660009 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.657980    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed476580-cb27-4a37-9ba4-f117f62232b6" (UID: "ed476580-cb27-4a37-9ba4-f117f62232b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:43.660009 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.658185    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-model-cache" (OuterVolumeSpecName: "model-cache") pod "827f9d20-74d3-4601-a68a-927b164ff00f" (UID: "827f9d20-74d3-4601-a68a-927b164ff00f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:43.660009 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.658528    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f441720-a3c0-4875-9313-26f87334f860" (UID: "5f441720-a3c0-4875-9313-26f87334f860"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:43.660330 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.660007    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-home" (OuterVolumeSpecName: "home") pod "827f9d20-74d3-4601-a68a-927b164ff00f" (UID: "827f9d20-74d3-4601-a68a-927b164ff00f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:43.660330 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.660172    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed476580-cb27-4a37-9ba4-f117f62232b6" (UID: "ed476580-cb27-4a37-9ba4-f117f62232b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:43.660330 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.660281    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "827f9d20-74d3-4601-a68a-927b164ff00f" (UID: "827f9d20-74d3-4601-a68a-927b164ff00f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:43.660606 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.660558    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-home" (OuterVolumeSpecName: "home") pod "5f441720-a3c0-4875-9313-26f87334f860" (UID: "5f441720-a3c0-4875-9313-26f87334f860"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:43.660912 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.660843    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed476580-cb27-4a37-9ba4-f117f62232b6-kube-api-access-2m8lq" (OuterVolumeSpecName: "kube-api-access-2m8lq") pod "ed476580-cb27-4a37-9ba4-f117f62232b6" (UID: "ed476580-cb27-4a37-9ba4-f117f62232b6"). InnerVolumeSpecName "kube-api-access-2m8lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:00:43.661161 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.661135    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-dshm" (OuterVolumeSpecName: "dshm") pod "ed476580-cb27-4a37-9ba4-f117f62232b6" (UID: "ed476580-cb27-4a37-9ba4-f117f62232b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:43.661399 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.661344    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/827f9d20-74d3-4601-a68a-927b164ff00f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "827f9d20-74d3-4601-a68a-927b164ff00f" (UID: "827f9d20-74d3-4601-a68a-927b164ff00f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:00:43.661399 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.661349    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827f9d20-74d3-4601-a68a-927b164ff00f-kube-api-access-5hr5f" (OuterVolumeSpecName: "kube-api-access-5hr5f") pod "827f9d20-74d3-4601-a68a-927b164ff00f" (UID: "827f9d20-74d3-4601-a68a-927b164ff00f"). InnerVolumeSpecName "kube-api-access-5hr5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:00:43.661549 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.661405    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-dshm" (OuterVolumeSpecName: "dshm") pod "5f441720-a3c0-4875-9313-26f87334f860" (UID: "5f441720-a3c0-4875-9313-26f87334f860"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:43.661836 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.661810    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed476580-cb27-4a37-9ba4-f117f62232b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed476580-cb27-4a37-9ba4-f117f62232b6" (UID: "ed476580-cb27-4a37-9ba4-f117f62232b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:00:43.662304 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.662286    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-dshm" (OuterVolumeSpecName: "dshm") pod "827f9d20-74d3-4601-a68a-927b164ff00f" (UID: "827f9d20-74d3-4601-a68a-927b164ff00f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:43.662971 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.662941    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f441720-a3c0-4875-9313-26f87334f860-kube-api-access-qdmlm" (OuterVolumeSpecName: "kube-api-access-qdmlm") pod "5f441720-a3c0-4875-9313-26f87334f860" (UID: "5f441720-a3c0-4875-9313-26f87334f860"). InnerVolumeSpecName "kube-api-access-qdmlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:00:43.663280 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.663252    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f441720-a3c0-4875-9313-26f87334f860-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f441720-a3c0-4875-9313-26f87334f860" (UID: "5f441720-a3c0-4875-9313-26f87334f860"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:00:43.759033 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.758983    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-dshm\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759033 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759029    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed476580-cb27-4a37-9ba4-f117f62232b6-tls-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759033 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759040    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-tmp-dir\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759047    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-tmp-dir\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759061    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hr5f\" (UniqueName: \"kubernetes.io/projected/827f9d20-74d3-4601-a68a-927b164ff00f-kube-api-access-5hr5f\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759077    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdmlm\" (UniqueName: \"kubernetes.io/projected/5f441720-a3c0-4875-9313-26f87334f860-kube-api-access-qdmlm\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759086    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f441720-a3c0-4875-9313-26f87334f860-tls-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759094    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-dshm\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759101    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-home\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759108    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-model-cache\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759116    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827f9d20-74d3-4601-a68a-927b164ff00f-tls-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759123    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-home\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759131    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827f9d20-74d3-4601-a68a-927b164ff00f-tmp-dir\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759138    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-model-cache\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759144    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f441720-a3c0-4875-9313-26f87334f860-model-cache\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759152    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m8lq\" (UniqueName: \"kubernetes.io/projected/ed476580-cb27-4a37-9ba4-f117f62232b6-kube-api-access-2m8lq\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:43.759290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:43.759160    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed476580-cb27-4a37-9ba4-f117f62232b6-dshm\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:44.272187 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.272145    2567 generic.go:356] "Generic (PLEG): container finished" podID="ed476580-cb27-4a37-9ba4-f117f62232b6" containerID="de5743abfdb8b741497a3e27c9edc858c504cada77e71cc136809867e7ccad4e" exitCode=0
Sep 12 01:00:44.272414 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.272246    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"
Sep 12 01:00:44.273187 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.273154    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69" event={"ID":"ed476580-cb27-4a37-9ba4-f117f62232b6","Type":"ContainerDied","Data":"de5743abfdb8b741497a3e27c9edc858c504cada77e71cc136809867e7ccad4e"}
Sep 12 01:00:44.273338 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.273204    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69" event={"ID":"ed476580-cb27-4a37-9ba4-f117f62232b6","Type":"ContainerDied","Data":"6c4468089c1faecad0a395eb4a443edd019168df42eb236e1feb0490c9d7f1e5"}
Sep 12 01:00:44.273338 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.273228    2567 scope.go:117] "RemoveContainer" containerID="de5743abfdb8b741497a3e27c9edc858c504cada77e71cc136809867e7ccad4e"
Sep 12 01:00:44.273897 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.273872    2567 generic.go:356] "Generic (PLEG): container finished" podID="827f9d20-74d3-4601-a68a-927b164ff00f" containerID="4c2a54e26fd47091629632cb5923b54fb06165159b309b11d1659feac9f40c4f" exitCode=0
Sep 12 01:00:44.273995 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.273931    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"
Sep 12 01:00:44.273995 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.273964    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7" event={"ID":"827f9d20-74d3-4601-a68a-927b164ff00f","Type":"ContainerDied","Data":"4c2a54e26fd47091629632cb5923b54fb06165159b309b11d1659feac9f40c4f"}
Sep 12 01:00:44.274114 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.274005    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7" event={"ID":"827f9d20-74d3-4601-a68a-927b164ff00f","Type":"ContainerDied","Data":"54283699241c2af69124cb46dd5f72a690d6e335c96c63ae75c45356b4549f3c"}
Sep 12 01:00:44.276762 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.276737    2567 generic.go:356] "Generic (PLEG): container finished" podID="5f441720-a3c0-4875-9313-26f87334f860" containerID="152319cf398337aafb839a7d6f3f7b6e66279a475096b80c460ce18e96e781e5" exitCode=0
Sep 12 01:00:44.276885 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.276826    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg" event={"ID":"5f441720-a3c0-4875-9313-26f87334f860","Type":"ContainerDied","Data":"152319cf398337aafb839a7d6f3f7b6e66279a475096b80c460ce18e96e781e5"}
Sep 12 01:00:44.276885 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.276854    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg" event={"ID":"5f441720-a3c0-4875-9313-26f87334f860","Type":"ContainerDied","Data":"1c0a965d95c8148fcaef17b09df2ea8aa7007ef5a13c84c7ad78f4246bdfabca"}
Sep 12 01:00:44.277004 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.276908    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"
Sep 12 01:00:44.288949 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.288752    2567 scope.go:117] "RemoveContainer" containerID="de5743abfdb8b741497a3e27c9edc858c504cada77e71cc136809867e7ccad4e"
Sep 12 01:00:44.289242 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:00:44.289133    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5743abfdb8b741497a3e27c9edc858c504cada77e71cc136809867e7ccad4e\": container with ID starting with de5743abfdb8b741497a3e27c9edc858c504cada77e71cc136809867e7ccad4e not found: ID does not exist" containerID="de5743abfdb8b741497a3e27c9edc858c504cada77e71cc136809867e7ccad4e"
Sep 12 01:00:44.289242 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.289170    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5743abfdb8b741497a3e27c9edc858c504cada77e71cc136809867e7ccad4e"} err="failed to get container status \"de5743abfdb8b741497a3e27c9edc858c504cada77e71cc136809867e7ccad4e\": rpc error: code = NotFound desc = could not find container \"de5743abfdb8b741497a3e27c9edc858c504cada77e71cc136809867e7ccad4e\": container with ID starting with de5743abfdb8b741497a3e27c9edc858c504cada77e71cc136809867e7ccad4e not found: ID does not exist"
Sep 12 01:00:44.289242 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.289194    2567 scope.go:117] "RemoveContainer" containerID="4c2a54e26fd47091629632cb5923b54fb06165159b309b11d1659feac9f40c4f"
Sep 12 01:00:44.307636 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.307604    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"]
Sep 12 01:00:44.312118 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.312093    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-876808/premium-w4-876808-kserve-78fddb56d9-l5w69"]
Sep 12 01:00:44.315527 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.315506    2567 scope.go:117] "RemoveContainer" containerID="4c2a54e26fd47091629632cb5923b54fb06165159b309b11d1659feac9f40c4f"
Sep 12 01:00:44.315904 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:00:44.315879    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2a54e26fd47091629632cb5923b54fb06165159b309b11d1659feac9f40c4f\": container with ID starting with 4c2a54e26fd47091629632cb5923b54fb06165159b309b11d1659feac9f40c4f not found: ID does not exist" containerID="4c2a54e26fd47091629632cb5923b54fb06165159b309b11d1659feac9f40c4f"
Sep 12 01:00:44.315983 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.315911    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2a54e26fd47091629632cb5923b54fb06165159b309b11d1659feac9f40c4f"} err="failed to get container status \"4c2a54e26fd47091629632cb5923b54fb06165159b309b11d1659feac9f40c4f\": rpc error: code = NotFound desc = could not find container \"4c2a54e26fd47091629632cb5923b54fb06165159b309b11d1659feac9f40c4f\": container with ID starting with 4c2a54e26fd47091629632cb5923b54fb06165159b309b11d1659feac9f40c4f not found: ID does not exist"
Sep 12 01:00:44.315983 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.315931    2567 scope.go:117] "RemoveContainer" containerID="152319cf398337aafb839a7d6f3f7b6e66279a475096b80c460ce18e96e781e5"
Sep 12 01:00:44.324397 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.324365    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"]
Sep 12 01:00:44.326094 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.326070    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-876808/distinct-w4-876808-kserve-85cdc976d4-5mfz7"]
Sep 12 01:00:44.326167 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.326151    2567 scope.go:117] "RemoveContainer" containerID="152319cf398337aafb839a7d6f3f7b6e66279a475096b80c460ce18e96e781e5"
Sep 12 01:00:44.326566 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:00:44.326544    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152319cf398337aafb839a7d6f3f7b6e66279a475096b80c460ce18e96e781e5\": container with ID starting with 152319cf398337aafb839a7d6f3f7b6e66279a475096b80c460ce18e96e781e5 not found: ID does not exist" containerID="152319cf398337aafb839a7d6f3f7b6e66279a475096b80c460ce18e96e781e5"
Sep 12 01:00:44.326651 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.326594    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152319cf398337aafb839a7d6f3f7b6e66279a475096b80c460ce18e96e781e5"} err="failed to get container status \"152319cf398337aafb839a7d6f3f7b6e66279a475096b80c460ce18e96e781e5\": rpc error: code = NotFound desc = could not find container \"152319cf398337aafb839a7d6f3f7b6e66279a475096b80c460ce18e96e781e5\": container with ID starting with 152319cf398337aafb839a7d6f3f7b6e66279a475096b80c460ce18e96e781e5 not found: ID does not exist"
Sep 12 01:00:44.335696 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.335662    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"]
Sep 12 01:00:44.337336 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:44.337313    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-876808/simulator-w4-876808-kserve-57cdb897cb-d29mg"]
Sep 12 01:00:45.906918 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:45.906884    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"]
Sep 12 01:00:45.907418 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:45.907179    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy" containerID="cri-o://6560a0a0d0c6c0d1a90b823fe5daee035765c2d94feda30340ca16dbdee0da03" gracePeriod=30
Sep 12 01:00:46.273110 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:46.273069    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f441720-a3c0-4875-9313-26f87334f860" path="/var/lib/kubelet/pods/5f441720-a3c0-4875-9313-26f87334f860/volumes"
Sep 12 01:00:46.273530 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:46.273516    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="827f9d20-74d3-4601-a68a-927b164ff00f" path="/var/lib/kubelet/pods/827f9d20-74d3-4601-a68a-927b164ff00f/volumes"
Sep 12 01:00:46.273954 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:46.273938    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed476580-cb27-4a37-9ba4-f117f62232b6" path="/var/lib/kubelet/pods/ed476580-cb27-4a37-9ba4-f117f62232b6/volumes"
Sep 12 01:00:49.065659 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:49.065617    2567 patch_prober.go:28] interesting pod/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 01:00:49.066096 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:49.065716    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 01:00:51.159853 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.159820    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 01:00:51.228055 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228017    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cm84\" (UniqueName: \"kubernetes.io/projected/3ed5365d-b02a-4167-b878-2e78a963ad37-kube-api-access-8cm84\") pod \"3ed5365d-b02a-4167-b878-2e78a963ad37\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") "
Sep 12 01:00:51.228255 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228069    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-credential-socket\") pod \"3ed5365d-b02a-4167-b878-2e78a963ad37\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") "
Sep 12 01:00:51.228255 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228122    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-workload-certs\") pod \"3ed5365d-b02a-4167-b878-2e78a963ad37\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") "
Sep 12 01:00:51.228255 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228167    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-data\") pod \"3ed5365d-b02a-4167-b878-2e78a963ad37\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") "
Sep 12 01:00:51.228255 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228195    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-podinfo\") pod \"3ed5365d-b02a-4167-b878-2e78a963ad37\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") "
Sep 12 01:00:51.228255 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228222    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed5365d-b02a-4167-b878-2e78a963ad37-istiod-ca-cert\") pod \"3ed5365d-b02a-4167-b878-2e78a963ad37\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") "
Sep 12 01:00:51.228511 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228265    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-envoy\") pod \"3ed5365d-b02a-4167-b878-2e78a963ad37\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") "
Sep 12 01:00:51.228511 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228297    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-token\") pod \"3ed5365d-b02a-4167-b878-2e78a963ad37\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") "
Sep 12 01:00:51.228511 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228330    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-workload-socket\") pod \"3ed5365d-b02a-4167-b878-2e78a963ad37\" (UID: \"3ed5365d-b02a-4167-b878-2e78a963ad37\") "
Sep 12 01:00:51.228511 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228414    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ed5365d-b02a-4167-b878-2e78a963ad37" (UID: "3ed5365d-b02a-4167-b878-2e78a963ad37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:51.228511 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228465    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ed5365d-b02a-4167-b878-2e78a963ad37" (UID: "3ed5365d-b02a-4167-b878-2e78a963ad37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:51.228800 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228501    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ed5365d-b02a-4167-b878-2e78a963ad37" (UID: "3ed5365d-b02a-4167-b878-2e78a963ad37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:51.228800 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228722    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ed5365d-b02a-4167-b878-2e78a963ad37" (UID: "3ed5365d-b02a-4167-b878-2e78a963ad37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:51.228800 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228719    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed5365d-b02a-4167-b878-2e78a963ad37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ed5365d-b02a-4167-b878-2e78a963ad37" (UID: "3ed5365d-b02a-4167-b878-2e78a963ad37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:00:51.228800 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228754    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-workload-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:51.228800 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228776    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-data\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:51.228800 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.228790    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-credential-socket\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:51.230370 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.230340    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed5365d-b02a-4167-b878-2e78a963ad37-kube-api-access-8cm84" (OuterVolumeSpecName: "kube-api-access-8cm84") pod "3ed5365d-b02a-4167-b878-2e78a963ad37" (UID: "3ed5365d-b02a-4167-b878-2e78a963ad37"). InnerVolumeSpecName "kube-api-access-8cm84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:00:51.230492 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.230382    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ed5365d-b02a-4167-b878-2e78a963ad37" (UID: "3ed5365d-b02a-4167-b878-2e78a963ad37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:00:51.230492 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.230409    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ed5365d-b02a-4167-b878-2e78a963ad37" (UID: "3ed5365d-b02a-4167-b878-2e78a963ad37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 01:00:51.230492 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.230486    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ed5365d-b02a-4167-b878-2e78a963ad37" (UID: "3ed5365d-b02a-4167-b878-2e78a963ad37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:00:51.325443 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.325403    2567 generic.go:356] "Generic (PLEG): container finished" podID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerID="6560a0a0d0c6c0d1a90b823fe5daee035765c2d94feda30340ca16dbdee0da03" exitCode=0
Sep 12 01:00:51.325622 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.325455    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"
Sep 12 01:00:51.325622 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.325524    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" event={"ID":"3ed5365d-b02a-4167-b878-2e78a963ad37","Type":"ContainerDied","Data":"6560a0a0d0c6c0d1a90b823fe5daee035765c2d94feda30340ca16dbdee0da03"}
Sep 12 01:00:51.325622 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.325599    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7" event={"ID":"3ed5365d-b02a-4167-b878-2e78a963ad37","Type":"ContainerDied","Data":"918b4d00648108986f7ff9b8669bcad5515251839d350aa0fcb9770e5bfc684e"}
Sep 12 01:00:51.325622 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.325622    2567 scope.go:117] "RemoveContainer" containerID="6560a0a0d0c6c0d1a90b823fe5daee035765c2d94feda30340ca16dbdee0da03"
Sep 12 01:00:51.329370 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.329350    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-podinfo\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:51.329370 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.329370    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed5365d-b02a-4167-b878-2e78a963ad37-istiod-ca-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:51.329520 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.329386    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-envoy\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:51.329520 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.329397    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed5365d-b02a-4167-b878-2e78a963ad37-istio-token\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:51.329520 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.329408    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed5365d-b02a-4167-b878-2e78a963ad37-workload-socket\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:51.329520 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.329421    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cm84\" (UniqueName: \"kubernetes.io/projected/3ed5365d-b02a-4167-b878-2e78a963ad37-kube-api-access-8cm84\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:00:51.335215 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.335197    2567 scope.go:117] "RemoveContainer" containerID="5d3bcb637c77ff77e3901528a54f47d624c368f63e231a11947b2daeacebc672"
Sep 12 01:00:51.343649 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.343623    2567 scope.go:117] "RemoveContainer" containerID="6560a0a0d0c6c0d1a90b823fe5daee035765c2d94feda30340ca16dbdee0da03"
Sep 12 01:00:51.343980 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:00:51.343958    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6560a0a0d0c6c0d1a90b823fe5daee035765c2d94feda30340ca16dbdee0da03\": container with ID starting with 6560a0a0d0c6c0d1a90b823fe5daee035765c2d94feda30340ca16dbdee0da03 not found: ID does not exist" containerID="6560a0a0d0c6c0d1a90b823fe5daee035765c2d94feda30340ca16dbdee0da03"
Sep 12 01:00:51.344061 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.343994    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6560a0a0d0c6c0d1a90b823fe5daee035765c2d94feda30340ca16dbdee0da03"} err="failed to get container status \"6560a0a0d0c6c0d1a90b823fe5daee035765c2d94feda30340ca16dbdee0da03\": rpc error: code = NotFound desc = could not find container \"6560a0a0d0c6c0d1a90b823fe5daee035765c2d94feda30340ca16dbdee0da03\": container with ID starting with 6560a0a0d0c6c0d1a90b823fe5daee035765c2d94feda30340ca16dbdee0da03 not found: ID does not exist"
Sep 12 01:00:51.344061 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.344023    2567 scope.go:117] "RemoveContainer" containerID="5d3bcb637c77ff77e3901528a54f47d624c368f63e231a11947b2daeacebc672"
Sep 12 01:00:51.344304 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:00:51.344286    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3bcb637c77ff77e3901528a54f47d624c368f63e231a11947b2daeacebc672\": container with ID starting with 5d3bcb637c77ff77e3901528a54f47d624c368f63e231a11947b2daeacebc672 not found: ID does not exist" containerID="5d3bcb637c77ff77e3901528a54f47d624c368f63e231a11947b2daeacebc672"
Sep 12 01:00:51.344344 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.344310    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3bcb637c77ff77e3901528a54f47d624c368f63e231a11947b2daeacebc672"} err="failed to get container status \"5d3bcb637c77ff77e3901528a54f47d624c368f63e231a11947b2daeacebc672\": rpc error: code = NotFound desc = could not find container \"5d3bcb637c77ff77e3901528a54f47d624c368f63e231a11947b2daeacebc672\": container with ID starting with 5d3bcb637c77ff77e3901528a54f47d624c368f63e231a11947b2daeacebc672 not found: ID does not exist"
Sep 12 01:00:51.348859 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.348834    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"]
Sep 12 01:00:51.354558 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:51.354533    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-876808-openshift-default-b6fbd76c4-f22b7"]
Sep 12 01:00:52.273274 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:52.273236    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" path="/var/lib/kubelet/pods/3ed5365d-b02a-4167-b878-2e78a963ad37/volumes"
Sep 12 01:00:57.104048 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104006    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"]
Sep 12 01:00:57.104545 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104498    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43265a00-f4b7-4e6c-bf13-99bca2d4c199" containerName="revoke-keys"
Sep 12 01:00:57.104545 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104516    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="43265a00-f4b7-4e6c-bf13-99bca2d4c199" containerName="revoke-keys"
Sep 12 01:00:57.104545 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104531    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827f9d20-74d3-4601-a68a-927b164ff00f" containerName="main"
Sep 12 01:00:57.104545 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104540    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="827f9d20-74d3-4601-a68a-927b164ff00f" containerName="main"
Sep 12 01:00:57.104702 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104559    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy"
Sep 12 01:00:57.104702 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104593    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy"
Sep 12 01:00:57.104702 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104606    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed476580-cb27-4a37-9ba4-f117f62232b6" containerName="main"
Sep 12 01:00:57.104702 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104613    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed476580-cb27-4a37-9ba4-f117f62232b6" containerName="main"
Sep 12 01:00:57.104702 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104625    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy"
Sep 12 01:00:57.104702 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104634    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy"
Sep 12 01:00:57.104702 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104643    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f441720-a3c0-4875-9313-26f87334f860" containerName="main"
Sep 12 01:00:57.104702 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104652    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f441720-a3c0-4875-9313-26f87334f860" containerName="main"
Sep 12 01:00:57.104942 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104736    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="43265a00-f4b7-4e6c-bf13-99bca2d4c199" containerName="revoke-keys"
Sep 12 01:00:57.104942 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104746    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy"
Sep 12 01:00:57.104942 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104756    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed476580-cb27-4a37-9ba4-f117f62232b6" containerName="main"
Sep 12 01:00:57.104942 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104764    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f441720-a3c0-4875-9313-26f87334f860" containerName="main"
Sep 12 01:00:57.104942 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104775    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed5365d-b02a-4167-b878-2e78a963ad37" containerName="istio-proxy"
Sep 12 01:00:57.104942 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.104784    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="827f9d20-74d3-4601-a68a-927b164ff00f" containerName="main"
Sep 12 01:00:57.109469 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.109439    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.109879 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.109849    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"]
Sep 12 01:00:57.112178 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.112149    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9511bd\"/\"kube-root-ca.crt\""
Sep 12 01:00:57.112580 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.112551    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9511bd\"/\"default-dockercfg-scc8f\""
Sep 12 01:00:57.113500 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.113460    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9511bd\"/\"sub-test-model-9511bd-kserve-self-signed-certs\""
Sep 12 01:00:57.113679 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.113617    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9511bd\"/\"openshift-service-ca.crt\""
Sep 12 01:00:57.185173 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.185138    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m487h\" (UniqueName: \"kubernetes.io/projected/d85741f5-95fc-4ddf-a575-aba7381316fc-kube-api-access-m487h\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.185173 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.185179    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d85741f5-95fc-4ddf-a575-aba7381316fc-tls-certs\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.185401 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.185201    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-model-cache\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.185401 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.185223    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-home\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.185401 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.185312    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-dshm\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.185401 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.185348    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-tmp-dir\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.286332 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.286295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m487h\" (UniqueName: \"kubernetes.io/projected/d85741f5-95fc-4ddf-a575-aba7381316fc-kube-api-access-m487h\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.286535 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.286340    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d85741f5-95fc-4ddf-a575-aba7381316fc-tls-certs\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.286535 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.286360    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-model-cache\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.286535 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.286383    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-home\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.286535 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.286407    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-dshm\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.286535 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.286424    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-tmp-dir\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.286856 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.286804    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-model-cache\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.286913 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.286854    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-tmp-dir\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.286913 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.286855    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-home\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.288727 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.288701    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-dshm\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.288896 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.288872    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d85741f5-95fc-4ddf-a575-aba7381316fc-tls-certs\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.294342 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.294315    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m487h\" (UniqueName: \"kubernetes.io/projected/d85741f5-95fc-4ddf-a575-aba7381316fc-kube-api-access-m487h\") pod \"sub-test-model-9511bd-kserve-7b654bc979-fr9dh\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.419366 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.419265    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:00:57.692592 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:57.692499    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"]
Sep 12 01:00:57.696367 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:00:57.696336    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd85741f5_95fc_4ddf_a575_aba7381316fc.slice/crio-3a70c7d6097a4cf3e8a902d84b46eb87b1867865bb18e76c16f2e8eb35c8174c WatchSource:0}: Error finding container 3a70c7d6097a4cf3e8a902d84b46eb87b1867865bb18e76c16f2e8eb35c8174c: Status 404 returned error can't find the container with id 3a70c7d6097a4cf3e8a902d84b46eb87b1867865bb18e76c16f2e8eb35c8174c
Sep 12 01:00:58.362090 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:58.362050    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh" event={"ID":"d85741f5-95fc-4ddf-a575-aba7381316fc","Type":"ContainerStarted","Data":"0b4545e29af9110cac6fed157cece26e77657dcce35bf7fe5c01e4a58452ba3e"}
Sep 12 01:00:58.362090 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:58.362095    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh" event={"ID":"d85741f5-95fc-4ddf-a575-aba7381316fc","Type":"ContainerStarted","Data":"3a70c7d6097a4cf3e8a902d84b46eb87b1867865bb18e76c16f2e8eb35c8174c"}
Sep 12 01:00:58.380276 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:00:58.380223    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh" podStartSLOduration=1.380207743 podStartE2EDuration="1.380207743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:00:58.378668836 +0000 UTC m=+1890.582669659" watchObservedRunningTime="2026-09-12 01:00:58.380207743 +0000 UTC m=+1890.584208564"
Sep 12 01:01:07.419781 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:01:07.419725    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:01:07.420207 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:01:07.419799    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:01:07.432531 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:01:07.432496    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:01:08.427516 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:01:08.427484    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:02:04.409738 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:04.409703    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7679-da2488"]
Sep 12 01:02:04.412733 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:04.412714    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-da2488"
Sep 12 01:02:04.415523 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:04.415498    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tqwft\""
Sep 12 01:02:04.419530 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:04.419504    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-da2488"]
Sep 12 01:02:04.479529 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:04.479488    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7kgw\" (UniqueName: \"kubernetes.io/projected/cbfd674c-c6e7-443e-9f03-54ee24793c52-kube-api-access-r7kgw\") pod \"test-curl-7679-da2488\" (UID: \"cbfd674c-c6e7-443e-9f03-54ee24793c52\") " pod="opendatahub/test-curl-7679-da2488"
Sep 12 01:02:04.580709 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:04.580670    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7kgw\" (UniqueName: \"kubernetes.io/projected/cbfd674c-c6e7-443e-9f03-54ee24793c52-kube-api-access-r7kgw\") pod \"test-curl-7679-da2488\" (UID: \"cbfd674c-c6e7-443e-9f03-54ee24793c52\") " pod="opendatahub/test-curl-7679-da2488"
Sep 12 01:02:04.590510 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:04.590475    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7kgw\" (UniqueName: \"kubernetes.io/projected/cbfd674c-c6e7-443e-9f03-54ee24793c52-kube-api-access-r7kgw\") pod \"test-curl-7679-da2488\" (UID: \"cbfd674c-c6e7-443e-9f03-54ee24793c52\") " pod="opendatahub/test-curl-7679-da2488"
Sep 12 01:02:04.723739 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:04.723703    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-da2488"
Sep 12 01:02:04.993044 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:04.993015    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-da2488"]
Sep 12 01:02:04.995342 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:02:04.995313    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbfd674c_c6e7_443e_9f03_54ee24793c52.slice/crio-f11f092a9e1f733ee792d8b5844360316dfecce39adc865b1d5265a978449443 WatchSource:0}: Error finding container f11f092a9e1f733ee792d8b5844360316dfecce39adc865b1d5265a978449443: Status 404 returned error can't find the container with id f11f092a9e1f733ee792d8b5844360316dfecce39adc865b1d5265a978449443
Sep 12 01:02:05.700950 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:05.700914    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-da2488" event={"ID":"cbfd674c-c6e7-443e-9f03-54ee24793c52","Type":"ContainerStarted","Data":"7a29d9a5bcedde0aff555472ea52ee22d047d74e41e7a0e56ccf6d6ca421bfa3"}
Sep 12 01:02:05.701383 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:05.700962    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-da2488" event={"ID":"cbfd674c-c6e7-443e-9f03-54ee24793c52","Type":"ContainerStarted","Data":"f11f092a9e1f733ee792d8b5844360316dfecce39adc865b1d5265a978449443"}
Sep 12 01:02:05.716126 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:05.716068    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7679-da2488" podStartSLOduration=1.716049914 podStartE2EDuration="1.716049914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:02:05.715160471 +0000 UTC m=+1957.919161294" watchObservedRunningTime="2026-09-12 01:02:05.716049914 +0000 UTC m=+1957.920050735"
Sep 12 01:02:06.295232 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:06.295200    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7679-da2488"]
Sep 12 01:02:06.298889 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:06.298860    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7679-da2488"]
Sep 12 01:02:06.413117 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:06.413036    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7679-9c9726"]
Sep 12 01:02:06.413371 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:06.413357    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbfd674c-c6e7-443e-9f03-54ee24793c52" containerName="test-curl-7679-da2488"
Sep 12 01:02:06.413416 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:06.413372    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbfd674c-c6e7-443e-9f03-54ee24793c52" containerName="test-curl-7679-da2488"
Sep 12 01:02:06.413452 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:06.413439    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbfd674c-c6e7-443e-9f03-54ee24793c52" containerName="test-curl-7679-da2488"
Sep 12 01:02:06.416422 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:06.416400    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-9c9726"
Sep 12 01:02:06.423030 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:06.423003    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-9c9726"]
Sep 12 01:02:06.506534 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:06.506493    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8m8r\" (UniqueName: \"kubernetes.io/projected/7d6abd08-31cc-43c9-862f-1b68ed32abae-kube-api-access-b8m8r\") pod \"test-curl-7679-9c9726\" (UID: \"7d6abd08-31cc-43c9-862f-1b68ed32abae\") " pod="opendatahub/test-curl-7679-9c9726"
Sep 12 01:02:06.607812 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:06.607774    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8m8r\" (UniqueName: \"kubernetes.io/projected/7d6abd08-31cc-43c9-862f-1b68ed32abae-kube-api-access-b8m8r\") pod \"test-curl-7679-9c9726\" (UID: \"7d6abd08-31cc-43c9-862f-1b68ed32abae\") " pod="opendatahub/test-curl-7679-9c9726"
Sep 12 01:02:06.615881 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:06.615858    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8m8r\" (UniqueName: \"kubernetes.io/projected/7d6abd08-31cc-43c9-862f-1b68ed32abae-kube-api-access-b8m8r\") pod \"test-curl-7679-9c9726\" (UID: \"7d6abd08-31cc-43c9-862f-1b68ed32abae\") " pod="opendatahub/test-curl-7679-9c9726"
Sep 12 01:02:06.706298 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:02:06.706218    2567 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7679-da2488" container="test-curl-7679-da2488"
Sep 12 01:02:06.727446 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:06.727419    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-9c9726"
Sep 12 01:02:06.997060 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:06.996977    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7679-9c9726"]
Sep 12 01:02:06.999500 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:02:06.999472    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d6abd08_31cc_43c9_862f_1b68ed32abae.slice/crio-f5f084f035836e35cd2c02da101d97ff33390bedf5db157600ce9307efd43818 WatchSource:0}: Error finding container f5f084f035836e35cd2c02da101d97ff33390bedf5db157600ce9307efd43818: Status 404 returned error can't find the container with id f5f084f035836e35cd2c02da101d97ff33390bedf5db157600ce9307efd43818
Sep 12 01:02:07.712768 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:07.712718    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-9c9726" event={"ID":"7d6abd08-31cc-43c9-862f-1b68ed32abae","Type":"ContainerStarted","Data":"0483689c1b91d946f932127f2a7bbaf9c8533df59420cfdbd422b112851b2475"}
Sep 12 01:02:07.712768 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:07.712755    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7679-da2488" podUID="cbfd674c-c6e7-443e-9f03-54ee24793c52" containerName="test-curl-7679-da2488" containerID="cri-o://7a29d9a5bcedde0aff555472ea52ee22d047d74e41e7a0e56ccf6d6ca421bfa3" gracePeriod=2
Sep 12 01:02:07.712768 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:07.712772    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7679-9c9726" event={"ID":"7d6abd08-31cc-43c9-862f-1b68ed32abae","Type":"ContainerStarted","Data":"f5f084f035836e35cd2c02da101d97ff33390bedf5db157600ce9307efd43818"}
Sep 12 01:02:07.715113 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:02:07.715084    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7679-da2488\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="cbfd674c-c6e7-443e-9f03-54ee24793c52" pod="opendatahub/test-curl-7679-da2488"
Sep 12 01:02:07.728955 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:07.728911    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7679-9c9726" podStartSLOduration=1.72889637 podStartE2EDuration="1.72889637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:02:07.728056333 +0000 UTC m=+1959.932057156" watchObservedRunningTime="2026-09-12 01:02:07.72889637 +0000 UTC m=+1959.932897191"
Sep 12 01:02:07.730063 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:02:07.730034    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7679-da2488\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="cbfd674c-c6e7-443e-9f03-54ee24793c52" pod="opendatahub/test-curl-7679-da2488"
Sep 12 01:02:08.271098 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:02:08.271057    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7679-da2488\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="cbfd674c-c6e7-443e-9f03-54ee24793c52" pod="opendatahub/test-curl-7679-da2488"
Sep 12 01:02:08.286996 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:08.286956    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7679-9c9726"]
Sep 12 01:02:08.291471 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:08.291440    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7679-9c9726"]
Sep 12 01:02:08.717934 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:02:08.717899    2567 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7679-9c9726" container="test-curl-7679-9c9726"
Sep 12 01:02:09.723884 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:09.723848    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7679-9c9726" podUID="7d6abd08-31cc-43c9-862f-1b68ed32abae" containerName="test-curl-7679-9c9726" containerID="cri-o://0483689c1b91d946f932127f2a7bbaf9c8533df59420cfdbd422b112851b2475" gracePeriod=2
Sep 12 01:02:09.726337 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:02:09.726309    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7679-9c9726\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="7d6abd08-31cc-43c9-862f-1b68ed32abae" pod="opendatahub/test-curl-7679-9c9726"
Sep 12 01:02:09.898741 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:09.898711    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-da2488"
Sep 12 01:02:09.900923 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:02:09.900896    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7679-9c9726\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="7d6abd08-31cc-43c9-862f-1b68ed32abae" pod="opendatahub/test-curl-7679-9c9726"
Sep 12 01:02:10.040613 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:10.040555    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7kgw\" (UniqueName: \"kubernetes.io/projected/cbfd674c-c6e7-443e-9f03-54ee24793c52-kube-api-access-r7kgw\") pod \"cbfd674c-c6e7-443e-9f03-54ee24793c52\" (UID: \"cbfd674c-c6e7-443e-9f03-54ee24793c52\") "
Sep 12 01:02:10.042675 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:10.042641    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfd674c-c6e7-443e-9f03-54ee24793c52-kube-api-access-r7kgw" (OuterVolumeSpecName: "kube-api-access-r7kgw") pod "cbfd674c-c6e7-443e-9f03-54ee24793c52" (UID: "cbfd674c-c6e7-443e-9f03-54ee24793c52"). InnerVolumeSpecName "kube-api-access-r7kgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:02:10.142155 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:10.142122    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7kgw\" (UniqueName: \"kubernetes.io/projected/cbfd674c-c6e7-443e-9f03-54ee24793c52-kube-api-access-r7kgw\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:02:10.226609 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:10.226099    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"]
Sep 12 01:02:10.229453 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:02:10.229410    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7679-9c9726\" is forbidden: User \"system:node:ip-10-0-130-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-132.ec2.internal' and this object" podUID="7d6abd08-31cc-43c9-862f-1b68ed32abae" pod="opendatahub/test-curl-7679-9c9726"
Sep 12 01:02:10.273509 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:10.273475    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbfd674c-c6e7-443e-9f03-54ee24793c52" path="/var/lib/kubelet/pods/cbfd674c-c6e7-443e-9f03-54ee24793c52/volumes"
Sep 12 01:02:10.729545 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:10.729512    2567 generic.go:356] "Generic (PLEG): container finished" podID="cbfd674c-c6e7-443e-9f03-54ee24793c52" containerID="7a29d9a5bcedde0aff555472ea52ee22d047d74e41e7a0e56ccf6d6ca421bfa3" exitCode=137
Sep 12 01:02:10.729982 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:10.729589    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-da2488"
Sep 12 01:02:10.729982 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:10.729611    2567 scope.go:117] "RemoveContainer" containerID="7a29d9a5bcedde0aff555472ea52ee22d047d74e41e7a0e56ccf6d6ca421bfa3"
Sep 12 01:02:10.738074 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:10.738054    2567 scope.go:117] "RemoveContainer" containerID="7a29d9a5bcedde0aff555472ea52ee22d047d74e41e7a0e56ccf6d6ca421bfa3"
Sep 12 01:02:10.738346 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:02:10.738325    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a29d9a5bcedde0aff555472ea52ee22d047d74e41e7a0e56ccf6d6ca421bfa3\": container with ID starting with 7a29d9a5bcedde0aff555472ea52ee22d047d74e41e7a0e56ccf6d6ca421bfa3 not found: ID does not exist" containerID="7a29d9a5bcedde0aff555472ea52ee22d047d74e41e7a0e56ccf6d6ca421bfa3"
Sep 12 01:02:10.738391 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:10.738357    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a29d9a5bcedde0aff555472ea52ee22d047d74e41e7a0e56ccf6d6ca421bfa3"} err="failed to get container status \"7a29d9a5bcedde0aff555472ea52ee22d047d74e41e7a0e56ccf6d6ca421bfa3\": rpc error: code = NotFound desc = could not find container \"7a29d9a5bcedde0aff555472ea52ee22d047d74e41e7a0e56ccf6d6ca421bfa3\": container with ID starting with 7a29d9a5bcedde0aff555472ea52ee22d047d74e41e7a0e56ccf6d6ca421bfa3 not found: ID does not exist"
Sep 12 01:02:11.890161 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:11.890134    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-9c9726"
Sep 12 01:02:12.061667 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:12.061553    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8m8r\" (UniqueName: \"kubernetes.io/projected/7d6abd08-31cc-43c9-862f-1b68ed32abae-kube-api-access-b8m8r\") pod \"7d6abd08-31cc-43c9-862f-1b68ed32abae\" (UID: \"7d6abd08-31cc-43c9-862f-1b68ed32abae\") "
Sep 12 01:02:12.063672 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:12.063646    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6abd08-31cc-43c9-862f-1b68ed32abae-kube-api-access-b8m8r" (OuterVolumeSpecName: "kube-api-access-b8m8r") pod "7d6abd08-31cc-43c9-862f-1b68ed32abae" (UID: "7d6abd08-31cc-43c9-862f-1b68ed32abae"). InnerVolumeSpecName "kube-api-access-b8m8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:02:12.162294 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:12.162251    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8m8r\" (UniqueName: \"kubernetes.io/projected/7d6abd08-31cc-43c9-862f-1b68ed32abae-kube-api-access-b8m8r\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:02:12.273178 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:12.273144    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d6abd08-31cc-43c9-862f-1b68ed32abae" path="/var/lib/kubelet/pods/7d6abd08-31cc-43c9-862f-1b68ed32abae/volumes"
Sep 12 01:02:12.741682 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:12.741643    2567 generic.go:356] "Generic (PLEG): container finished" podID="7d6abd08-31cc-43c9-862f-1b68ed32abae" containerID="0483689c1b91d946f932127f2a7bbaf9c8533df59420cfdbd422b112851b2475" exitCode=137
Sep 12 01:02:12.741909 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:12.741689    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7679-9c9726"
Sep 12 01:02:12.741909 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:12.741713    2567 scope.go:117] "RemoveContainer" containerID="0483689c1b91d946f932127f2a7bbaf9c8533df59420cfdbd422b112851b2475"
Sep 12 01:02:12.750169 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:12.750146    2567 scope.go:117] "RemoveContainer" containerID="0483689c1b91d946f932127f2a7bbaf9c8533df59420cfdbd422b112851b2475"
Sep 12 01:02:12.750494 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:02:12.750471    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0483689c1b91d946f932127f2a7bbaf9c8533df59420cfdbd422b112851b2475\": container with ID starting with 0483689c1b91d946f932127f2a7bbaf9c8533df59420cfdbd422b112851b2475 not found: ID does not exist" containerID="0483689c1b91d946f932127f2a7bbaf9c8533df59420cfdbd422b112851b2475"
Sep 12 01:02:12.750594 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:12.750503    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0483689c1b91d946f932127f2a7bbaf9c8533df59420cfdbd422b112851b2475"} err="failed to get container status \"0483689c1b91d946f932127f2a7bbaf9c8533df59420cfdbd422b112851b2475\": rpc error: code = NotFound desc = could not find container \"0483689c1b91d946f932127f2a7bbaf9c8533df59420cfdbd422b112851b2475\": container with ID starting with 0483689c1b91d946f932127f2a7bbaf9c8533df59420cfdbd422b112851b2475 not found: ID does not exist"
Sep 12 01:02:19.318836 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.318792    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"]
Sep 12 01:02:19.319255 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.319154    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d6abd08-31cc-43c9-862f-1b68ed32abae" containerName="test-curl-7679-9c9726"
Sep 12 01:02:19.319255 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.319166    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d6abd08-31cc-43c9-862f-1b68ed32abae" containerName="test-curl-7679-9c9726"
Sep 12 01:02:19.319255 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.319230    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d6abd08-31cc-43c9-862f-1b68ed32abae" containerName="test-curl-7679-9c9726"
Sep 12 01:02:19.323621 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.323599    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.326261 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.326242    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9511bd\"/\"rate-test-model-9511bd-kserve-self-signed-certs\""
Sep 12 01:02:19.326740 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.326718    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgbhk\" (UniqueName: \"kubernetes.io/projected/08c0461f-5fbd-4ccd-9f19-700464a9b629-kube-api-access-wgbhk\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.326788 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.326759    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-model-cache\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.326903 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.326879    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-tmp-dir\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.326956 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.326941    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-home\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.327015 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.327000    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c0461f-5fbd-4ccd-9f19-700464a9b629-tls-certs\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.327066 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.327054    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-dshm\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.331464 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.331437    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"]
Sep 12 01:02:19.427604 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.427526    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-model-cache\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.427813 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.427623    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-tmp-dir\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.427813 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.427667    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-home\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.427813 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.427695    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c0461f-5fbd-4ccd-9f19-700464a9b629-tls-certs\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.427813 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.427732    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-dshm\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.427813 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.427797    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbhk\" (UniqueName: \"kubernetes.io/projected/08c0461f-5fbd-4ccd-9f19-700464a9b629-kube-api-access-wgbhk\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.428080 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.428049    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-model-cache\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.428137 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.428081    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-tmp-dir\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.428137 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.428095    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-home\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.430057 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.430026    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-dshm\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.430350 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.430330    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c0461f-5fbd-4ccd-9f19-700464a9b629-tls-certs\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.435696 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.435669    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgbhk\" (UniqueName: \"kubernetes.io/projected/08c0461f-5fbd-4ccd-9f19-700464a9b629-kube-api-access-wgbhk\") pod \"rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") " pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.634979 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.634856    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:19.916023 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:19.915943    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"]
Sep 12 01:02:19.919778 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:02:19.919746    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08c0461f_5fbd_4ccd_9f19_700464a9b629.slice/crio-1ab733478047ecdddcd5276988d4c3843a11f7b0e57e06e914c70ec4a55ac77e WatchSource:0}: Error finding container 1ab733478047ecdddcd5276988d4c3843a11f7b0e57e06e914c70ec4a55ac77e: Status 404 returned error can't find the container with id 1ab733478047ecdddcd5276988d4c3843a11f7b0e57e06e914c70ec4a55ac77e
Sep 12 01:02:20.785666 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:20.785557    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs" event={"ID":"08c0461f-5fbd-4ccd-9f19-700464a9b629","Type":"ContainerStarted","Data":"2e330d309ce043c8878a9593942a8173b1be1ddc57a736873f66d6b28ddbf584"}
Sep 12 01:02:20.785666 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:20.785616    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs" event={"ID":"08c0461f-5fbd-4ccd-9f19-700464a9b629","Type":"ContainerStarted","Data":"1ab733478047ecdddcd5276988d4c3843a11f7b0e57e06e914c70ec4a55ac77e"}
Sep 12 01:02:25.249071 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.249009    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh" podUID="d85741f5-95fc-4ddf-a575-aba7381316fc" containerName="main" containerID="cri-o://0b4545e29af9110cac6fed157cece26e77657dcce35bf7fe5c01e4a58452ba3e" gracePeriod=45
Sep 12 01:02:25.496444 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.496412    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:02:25.515323 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.515204    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs" podStartSLOduration=6.515182868 podStartE2EDuration="6.515182868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:02:20.805865764 +0000 UTC m=+1973.009866586" watchObservedRunningTime="2026-09-12 01:02:25.515182868 +0000 UTC m=+1977.719183701"
Sep 12 01:02:25.586119 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.586074    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-dshm\") pod \"d85741f5-95fc-4ddf-a575-aba7381316fc\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") "
Sep 12 01:02:25.586303 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.586155    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d85741f5-95fc-4ddf-a575-aba7381316fc-tls-certs\") pod \"d85741f5-95fc-4ddf-a575-aba7381316fc\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") "
Sep 12 01:02:25.586303 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.586204    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m487h\" (UniqueName: \"kubernetes.io/projected/d85741f5-95fc-4ddf-a575-aba7381316fc-kube-api-access-m487h\") pod \"d85741f5-95fc-4ddf-a575-aba7381316fc\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") "
Sep 12 01:02:25.586303 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.586244    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-model-cache\") pod \"d85741f5-95fc-4ddf-a575-aba7381316fc\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") "
Sep 12 01:02:25.586303 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.586277    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-home\") pod \"d85741f5-95fc-4ddf-a575-aba7381316fc\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") "
Sep 12 01:02:25.586526 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.586305    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-tmp-dir\") pod \"d85741f5-95fc-4ddf-a575-aba7381316fc\" (UID: \"d85741f5-95fc-4ddf-a575-aba7381316fc\") "
Sep 12 01:02:25.586619 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.586559    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-home" (OuterVolumeSpecName: "home") pod "d85741f5-95fc-4ddf-a575-aba7381316fc" (UID: "d85741f5-95fc-4ddf-a575-aba7381316fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:02:25.586679 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.586608    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "d85741f5-95fc-4ddf-a575-aba7381316fc" (UID: "d85741f5-95fc-4ddf-a575-aba7381316fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:02:25.586728 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.586680    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d85741f5-95fc-4ddf-a575-aba7381316fc" (UID: "d85741f5-95fc-4ddf-a575-aba7381316fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:02:25.588501 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.588464    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-dshm" (OuterVolumeSpecName: "dshm") pod "d85741f5-95fc-4ddf-a575-aba7381316fc" (UID: "d85741f5-95fc-4ddf-a575-aba7381316fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:02:25.588501 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.588473    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85741f5-95fc-4ddf-a575-aba7381316fc-kube-api-access-m487h" (OuterVolumeSpecName: "kube-api-access-m487h") pod "d85741f5-95fc-4ddf-a575-aba7381316fc" (UID: "d85741f5-95fc-4ddf-a575-aba7381316fc"). InnerVolumeSpecName "kube-api-access-m487h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:02:25.588765 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.588538    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85741f5-95fc-4ddf-a575-aba7381316fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d85741f5-95fc-4ddf-a575-aba7381316fc" (UID: "d85741f5-95fc-4ddf-a575-aba7381316fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:02:25.687797 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.687752    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m487h\" (UniqueName: \"kubernetes.io/projected/d85741f5-95fc-4ddf-a575-aba7381316fc-kube-api-access-m487h\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:02:25.687797 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.687789    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-model-cache\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:02:25.687797 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.687802    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-home\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:02:25.688050 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.687812    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-tmp-dir\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:02:25.688050 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.687823    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d85741f5-95fc-4ddf-a575-aba7381316fc-dshm\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:02:25.688050 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.687834    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d85741f5-95fc-4ddf-a575-aba7381316fc-tls-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:02:25.812907 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.812813    2567 generic.go:356] "Generic (PLEG): container finished" podID="d85741f5-95fc-4ddf-a575-aba7381316fc" containerID="0b4545e29af9110cac6fed157cece26e77657dcce35bf7fe5c01e4a58452ba3e" exitCode=0
Sep 12 01:02:25.812907 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.812861    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"
Sep 12 01:02:25.812907 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.812894    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh" event={"ID":"d85741f5-95fc-4ddf-a575-aba7381316fc","Type":"ContainerDied","Data":"0b4545e29af9110cac6fed157cece26e77657dcce35bf7fe5c01e4a58452ba3e"}
Sep 12 01:02:25.813198 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.812944    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh" event={"ID":"d85741f5-95fc-4ddf-a575-aba7381316fc","Type":"ContainerDied","Data":"3a70c7d6097a4cf3e8a902d84b46eb87b1867865bb18e76c16f2e8eb35c8174c"}
Sep 12 01:02:25.813198 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.812966    2567 scope.go:117] "RemoveContainer" containerID="0b4545e29af9110cac6fed157cece26e77657dcce35bf7fe5c01e4a58452ba3e"
Sep 12 01:02:25.822377 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.822354    2567 scope.go:117] "RemoveContainer" containerID="0b4545e29af9110cac6fed157cece26e77657dcce35bf7fe5c01e4a58452ba3e"
Sep 12 01:02:25.822716 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:02:25.822698    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4545e29af9110cac6fed157cece26e77657dcce35bf7fe5c01e4a58452ba3e\": container with ID starting with 0b4545e29af9110cac6fed157cece26e77657dcce35bf7fe5c01e4a58452ba3e not found: ID does not exist" containerID="0b4545e29af9110cac6fed157cece26e77657dcce35bf7fe5c01e4a58452ba3e"
Sep 12 01:02:25.822772 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.822725    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4545e29af9110cac6fed157cece26e77657dcce35bf7fe5c01e4a58452ba3e"} err="failed to get container status \"0b4545e29af9110cac6fed157cece26e77657dcce35bf7fe5c01e4a58452ba3e\": rpc error: code = NotFound desc = could not find container \"0b4545e29af9110cac6fed157cece26e77657dcce35bf7fe5c01e4a58452ba3e\": container with ID starting with 0b4545e29af9110cac6fed157cece26e77657dcce35bf7fe5c01e4a58452ba3e not found: ID does not exist"
Sep 12 01:02:25.833204 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.833173    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"]
Sep 12 01:02:25.836958 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:25.836927    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9511bd/sub-test-model-9511bd-kserve-7b654bc979-fr9dh"]
Sep 12 01:02:26.273809 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:26.273777    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d85741f5-95fc-4ddf-a575-aba7381316fc" path="/var/lib/kubelet/pods/d85741f5-95fc-4ddf-a575-aba7381316fc/volumes"
Sep 12 01:02:29.050733 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:29.050698    2567 scope.go:117] "RemoveContainer" containerID="d146b7608cb601c3a89e011186c50e7702f500a2bbafb290e5e52cd81c07384e"
Sep 12 01:02:29.635148 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:29.635110    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:29.635338 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:29.635197    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:29.647859 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:29.647827    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:02:29.844924 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:02:29.844892    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:03:14.707444 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:14.707410    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"]
Sep 12 01:03:21.658909 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.658868    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"]
Sep 12 01:03:21.659354 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.659237    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d85741f5-95fc-4ddf-a575-aba7381316fc" containerName="main"
Sep 12 01:03:21.659354 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.659249    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d85741f5-95fc-4ddf-a575-aba7381316fc" containerName="main"
Sep 12 01:03:21.659354 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.659307    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d85741f5-95fc-4ddf-a575-aba7381316fc" containerName="main"
Sep 12 01:03:21.662551 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.662516    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"
Sep 12 01:03:21.665704 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.665676    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9a2657-dockercfg-g5ghz\""
Sep 12 01:03:21.665825 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.665711    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9a2657\""
Sep 12 01:03:21.672457 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.672428    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"]
Sep 12 01:03:21.803376 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.803340    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ce9b50-1da2-4df5-96a0-8ba13cf38088-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4\" (UID: \"51ce9b50-1da2-4df5-96a0-8ba13cf38088\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"
Sep 12 01:03:21.803556 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.803384    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvs68\" (UniqueName: \"kubernetes.io/projected/51ce9b50-1da2-4df5-96a0-8ba13cf38088-kube-api-access-cvs68\") pod \"payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4\" (UID: \"51ce9b50-1da2-4df5-96a0-8ba13cf38088\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"
Sep 12 01:03:21.904661 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.904615    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ce9b50-1da2-4df5-96a0-8ba13cf38088-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4\" (UID: \"51ce9b50-1da2-4df5-96a0-8ba13cf38088\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"
Sep 12 01:03:21.904661 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.904666    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvs68\" (UniqueName: \"kubernetes.io/projected/51ce9b50-1da2-4df5-96a0-8ba13cf38088-kube-api-access-cvs68\") pod \"payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4\" (UID: \"51ce9b50-1da2-4df5-96a0-8ba13cf38088\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"
Sep 12 01:03:21.905343 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.905322    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ce9b50-1da2-4df5-96a0-8ba13cf38088-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4\" (UID: \"51ce9b50-1da2-4df5-96a0-8ba13cf38088\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"
Sep 12 01:03:21.914809 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.914747    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvs68\" (UniqueName: \"kubernetes.io/projected/51ce9b50-1da2-4df5-96a0-8ba13cf38088-kube-api-access-cvs68\") pod \"payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4\" (UID: \"51ce9b50-1da2-4df5-96a0-8ba13cf38088\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"
Sep 12 01:03:21.974479 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:21.974440    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"
Sep 12 01:03:22.248229 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:22.248200    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"]
Sep 12 01:03:22.250823 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:03:22.250789    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ce9b50_1da2_4df5_96a0_8ba13cf38088.slice/crio-fb2380777c407c8bfd866c413a558f0a53d956b305cd24e70ac56867fca149f1 WatchSource:0}: Error finding container fb2380777c407c8bfd866c413a558f0a53d956b305cd24e70ac56867fca149f1: Status 404 returned error can't find the container with id fb2380777c407c8bfd866c413a558f0a53d956b305cd24e70ac56867fca149f1
Sep 12 01:03:23.104761 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:23.104716    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4" event={"ID":"51ce9b50-1da2-4df5-96a0-8ba13cf38088","Type":"ContainerStarted","Data":"f7cd981ffd437e62393e580576509574383f1cebc9ec4dc0664c0eac83937033"}
Sep 12 01:03:23.105198 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:23.104786    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4" event={"ID":"51ce9b50-1da2-4df5-96a0-8ba13cf38088","Type":"ContainerStarted","Data":"fb2380777c407c8bfd866c413a558f0a53d956b305cd24e70ac56867fca149f1"}
Sep 12 01:03:23.105198 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:23.104905    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"
Sep 12 01:03:23.121919 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:23.121821    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4" podStartSLOduration=2.12180046 podStartE2EDuration="2.12180046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:03:23.121084012 +0000 UTC m=+2035.325084834" watchObservedRunningTime="2026-09-12 01:03:23.12180046 +0000 UTC m=+2035.325801283"
Sep 12 01:03:29.726639 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:29.726557    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs" podUID="08c0461f-5fbd-4ccd-9f19-700464a9b629" containerName="main" containerID="cri-o://2e330d309ce043c8878a9593942a8173b1be1ddc57a736873f66d6b28ddbf584" gracePeriod=45
Sep 12 01:03:29.834410 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:29.834364    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs" podUID="08c0461f-5fbd-4ccd-9f19-700464a9b629" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 12 01:03:29.984106 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:29.984034    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:03:30.084330 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.084285    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c0461f-5fbd-4ccd-9f19-700464a9b629-tls-certs\") pod \"08c0461f-5fbd-4ccd-9f19-700464a9b629\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") "
Sep 12 01:03:30.084510 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.084342    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgbhk\" (UniqueName: \"kubernetes.io/projected/08c0461f-5fbd-4ccd-9f19-700464a9b629-kube-api-access-wgbhk\") pod \"08c0461f-5fbd-4ccd-9f19-700464a9b629\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") "
Sep 12 01:03:30.084510 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.084370    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-model-cache\") pod \"08c0461f-5fbd-4ccd-9f19-700464a9b629\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") "
Sep 12 01:03:30.084510 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.084409    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-home\") pod \"08c0461f-5fbd-4ccd-9f19-700464a9b629\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") "
Sep 12 01:03:30.084510 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.084438    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-dshm\") pod \"08c0461f-5fbd-4ccd-9f19-700464a9b629\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") "
Sep 12 01:03:30.084510 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.084487    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-tmp-dir\") pod \"08c0461f-5fbd-4ccd-9f19-700464a9b629\" (UID: \"08c0461f-5fbd-4ccd-9f19-700464a9b629\") "
Sep 12 01:03:30.084821 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.084740    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-home" (OuterVolumeSpecName: "home") pod "08c0461f-5fbd-4ccd-9f19-700464a9b629" (UID: "08c0461f-5fbd-4ccd-9f19-700464a9b629"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:03:30.084821 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.084751    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-model-cache" (OuterVolumeSpecName: "model-cache") pod "08c0461f-5fbd-4ccd-9f19-700464a9b629" (UID: "08c0461f-5fbd-4ccd-9f19-700464a9b629"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:03:30.084933 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.084829    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08c0461f-5fbd-4ccd-9f19-700464a9b629" (UID: "08c0461f-5fbd-4ccd-9f19-700464a9b629"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:03:30.086539 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.086513    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c0461f-5fbd-4ccd-9f19-700464a9b629-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08c0461f-5fbd-4ccd-9f19-700464a9b629" (UID: "08c0461f-5fbd-4ccd-9f19-700464a9b629"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:03:30.086635 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.086535    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c0461f-5fbd-4ccd-9f19-700464a9b629-kube-api-access-wgbhk" (OuterVolumeSpecName: "kube-api-access-wgbhk") pod "08c0461f-5fbd-4ccd-9f19-700464a9b629" (UID: "08c0461f-5fbd-4ccd-9f19-700464a9b629"). InnerVolumeSpecName "kube-api-access-wgbhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:03:30.086635 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.086603    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-dshm" (OuterVolumeSpecName: "dshm") pod "08c0461f-5fbd-4ccd-9f19-700464a9b629" (UID: "08c0461f-5fbd-4ccd-9f19-700464a9b629"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:03:30.141063 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.141028    2567 generic.go:356] "Generic (PLEG): container finished" podID="08c0461f-5fbd-4ccd-9f19-700464a9b629" containerID="2e330d309ce043c8878a9593942a8173b1be1ddc57a736873f66d6b28ddbf584" exitCode=0
Sep 12 01:03:30.141240 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.141077    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"
Sep 12 01:03:30.141240 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.141084    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs" event={"ID":"08c0461f-5fbd-4ccd-9f19-700464a9b629","Type":"ContainerDied","Data":"2e330d309ce043c8878a9593942a8173b1be1ddc57a736873f66d6b28ddbf584"}
Sep 12 01:03:30.141240 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.141118    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs" event={"ID":"08c0461f-5fbd-4ccd-9f19-700464a9b629","Type":"ContainerDied","Data":"1ab733478047ecdddcd5276988d4c3843a11f7b0e57e06e914c70ec4a55ac77e"}
Sep 12 01:03:30.141240 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.141135    2567 scope.go:117] "RemoveContainer" containerID="2e330d309ce043c8878a9593942a8173b1be1ddc57a736873f66d6b28ddbf584"
Sep 12 01:03:30.150795 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.150772    2567 scope.go:117] "RemoveContainer" containerID="2e330d309ce043c8878a9593942a8173b1be1ddc57a736873f66d6b28ddbf584"
Sep 12 01:03:30.151136 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:03:30.151115    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e330d309ce043c8878a9593942a8173b1be1ddc57a736873f66d6b28ddbf584\": container with ID starting with 2e330d309ce043c8878a9593942a8173b1be1ddc57a736873f66d6b28ddbf584 not found: ID does not exist" containerID="2e330d309ce043c8878a9593942a8173b1be1ddc57a736873f66d6b28ddbf584"
Sep 12 01:03:30.151202 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.151152    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e330d309ce043c8878a9593942a8173b1be1ddc57a736873f66d6b28ddbf584"} err="failed to get container status \"2e330d309ce043c8878a9593942a8173b1be1ddc57a736873f66d6b28ddbf584\": rpc error: code = NotFound desc = could not find container \"2e330d309ce043c8878a9593942a8173b1be1ddc57a736873f66d6b28ddbf584\": container with ID starting with 2e330d309ce043c8878a9593942a8173b1be1ddc57a736873f66d6b28ddbf584 not found: ID does not exist"
Sep 12 01:03:30.162781 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.162742    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"]
Sep 12 01:03:30.164306 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.164282    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9511bd/rate-test-model-9511bd-kserve-797dc7fc6c-bzcvs"]
Sep 12 01:03:30.185697 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.185666    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c0461f-5fbd-4ccd-9f19-700464a9b629-tls-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:03:30.185697 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.185694    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgbhk\" (UniqueName: \"kubernetes.io/projected/08c0461f-5fbd-4ccd-9f19-700464a9b629-kube-api-access-wgbhk\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:03:30.185870 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.185704    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-model-cache\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:03:30.185870 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.185714    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-home\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:03:30.185870 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.185721    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-dshm\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:03:30.185870 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.185731    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c0461f-5fbd-4ccd-9f19-700464a9b629-tmp-dir\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:03:30.273931 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:30.273840    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08c0461f-5fbd-4ccd-9f19-700464a9b629" path="/var/lib/kubelet/pods/08c0461f-5fbd-4ccd-9f19-700464a9b629/volumes"
Sep 12 01:03:34.111132 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:34.111098    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"
Sep 12 01:03:45.198078 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.198043    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"]
Sep 12 01:03:45.198479 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.198391    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c0461f-5fbd-4ccd-9f19-700464a9b629" containerName="main"
Sep 12 01:03:45.198479 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.198404    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c0461f-5fbd-4ccd-9f19-700464a9b629" containerName="main"
Sep 12 01:03:45.198479 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.198462    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c0461f-5fbd-4ccd-9f19-700464a9b629" containerName="main"
Sep 12 01:03:45.201361 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.201336    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.205103 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.204990    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5b711c-openshift-default-dockercfg-mrdcz\""
Sep 12 01:03:45.219238 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.219206    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"]
Sep 12 01:03:45.228042 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.228006    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-envoy\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.228206 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.228059    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfkrf\" (UniqueName: \"kubernetes.io/projected/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-kube-api-access-lfkrf\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.228206 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.228132    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-token\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.228336 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.228202    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-data\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.228336 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.228261    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-credential-socket\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.228336 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.228310    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istiod-ca-cert\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.228479 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.228345    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-podinfo\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.228479 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.228388    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-workload-certs\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.228479 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.228451    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-workload-socket\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.329440 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.329395    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istiod-ca-cert\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.329440 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.329437    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-podinfo\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.329750 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.329464    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-workload-certs\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.329750 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.329652    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-workload-socket\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.329863 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.329746    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-envoy\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.329863 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.329802    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfkrf\" (UniqueName: \"kubernetes.io/projected/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-kube-api-access-lfkrf\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.330341 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.329866    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-workload-certs\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.330341 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.330054    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-token\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.330341 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.330101    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-data\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.330341 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.330162    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-credential-socket\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.330640 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.330468    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-data\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.330640 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.330493    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-credential-socket\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.330640 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.330602    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-workload-socket\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.330912 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.330891    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istiod-ca-cert\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.333229 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.333196    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-envoy\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.333778 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.333755    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-podinfo\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.340713 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.340685    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfkrf\" (UniqueName: \"kubernetes.io/projected/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-kube-api-access-lfkrf\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.341301 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.341282    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-token\") pod \"e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") " pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.512162 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.512123    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:45.795216 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.795188    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"]
Sep 12 01:03:45.797448 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:03:45.797417    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf058bf56_36ef_49d4_a180_6a47e3d4bdd8.slice/crio-f897c09e275f95fd8813e93215e65c3fa38f0c958ff698f0b0cdba964e8c4bcf WatchSource:0}: Error finding container f897c09e275f95fd8813e93215e65c3fa38f0c958ff698f0b0cdba964e8c4bcf: Status 404 returned error can't find the container with id f897c09e275f95fd8813e93215e65c3fa38f0c958ff698f0b0cdba964e8c4bcf
Sep 12 01:03:45.799975 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.799936    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:03:45.800078 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.800022    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:03:45.800078 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:45.800065    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:03:46.226356 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:46.226312    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4" event={"ID":"f058bf56-36ef-49d4-a180-6a47e3d4bdd8","Type":"ContainerStarted","Data":"cdcee6b539698c9c2ae8556500b18f910f08f7649e61e0fe8ae20429e8b798d1"}
Sep 12 01:03:46.226803 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:46.226387    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4" event={"ID":"f058bf56-36ef-49d4-a180-6a47e3d4bdd8","Type":"ContainerStarted","Data":"f897c09e275f95fd8813e93215e65c3fa38f0c958ff698f0b0cdba964e8c4bcf"}
Sep 12 01:03:46.250888 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:46.250822    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4" podStartSLOduration=1.250800686 podStartE2EDuration="1.250800686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:03:46.249503958 +0000 UTC m=+2058.453504780" watchObservedRunningTime="2026-09-12 01:03:46.250800686 +0000 UTC m=+2058.454801508"
Sep 12 01:03:46.513098 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:46.512991    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:46.518124 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:46.518093    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:47.231011 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:47.230983    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:47.231947 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:47.231926    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:03:52.192066 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:52.191969    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"]
Sep 12 01:03:52.195548 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:52.195521    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"
Sep 12 01:03:52.203267 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:52.203241    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5b711c\""
Sep 12 01:03:52.227543 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:52.227496    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"]
Sep 12 01:03:52.298594 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:52.298523    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwwhg\" (UniqueName: \"kubernetes.io/projected/8f9b2b42-e02e-41b5-8937-731d81469b5f-kube-api-access-dwwhg\") pod \"payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g\" (UID: \"8f9b2b42-e02e-41b5-8937-731d81469b5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"
Sep 12 01:03:52.298769 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:52.298616    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9b2b42-e02e-41b5-8937-731d81469b5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g\" (UID: \"8f9b2b42-e02e-41b5-8937-731d81469b5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"
Sep 12 01:03:52.399209 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:52.399159    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwwhg\" (UniqueName: \"kubernetes.io/projected/8f9b2b42-e02e-41b5-8937-731d81469b5f-kube-api-access-dwwhg\") pod \"payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g\" (UID: \"8f9b2b42-e02e-41b5-8937-731d81469b5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"
Sep 12 01:03:52.399406 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:52.399214    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9b2b42-e02e-41b5-8937-731d81469b5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g\" (UID: \"8f9b2b42-e02e-41b5-8937-731d81469b5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"
Sep 12 01:03:52.400024 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:52.400002    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9b2b42-e02e-41b5-8937-731d81469b5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g\" (UID: \"8f9b2b42-e02e-41b5-8937-731d81469b5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"
Sep 12 01:03:52.408943 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:52.408907    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwwhg\" (UniqueName: \"kubernetes.io/projected/8f9b2b42-e02e-41b5-8937-731d81469b5f-kube-api-access-dwwhg\") pod \"payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g\" (UID: \"8f9b2b42-e02e-41b5-8937-731d81469b5f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"
Sep 12 01:03:52.507149 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:52.507110    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"
Sep 12 01:03:52.782253 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:52.782226    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"]
Sep 12 01:03:52.784387 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:03:52.784359    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9b2b42_e02e_41b5_8937_731d81469b5f.slice/crio-35084818ab603a92eb74ef5da7e5c2c2cb51b84756b2d34da083bc3fe52f81ea WatchSource:0}: Error finding container 35084818ab603a92eb74ef5da7e5c2c2cb51b84756b2d34da083bc3fe52f81ea: Status 404 returned error can't find the container with id 35084818ab603a92eb74ef5da7e5c2c2cb51b84756b2d34da083bc3fe52f81ea
Sep 12 01:03:53.264665 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:53.264627    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g" event={"ID":"8f9b2b42-e02e-41b5-8937-731d81469b5f","Type":"ContainerStarted","Data":"ad022d21945d998b886c9c89abf2ea99653c93080d67fa08813d3624180d155c"}
Sep 12 01:03:53.265169 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:53.264679    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g" event={"ID":"8f9b2b42-e02e-41b5-8937-731d81469b5f","Type":"ContainerStarted","Data":"35084818ab603a92eb74ef5da7e5c2c2cb51b84756b2d34da083bc3fe52f81ea"}
Sep 12 01:03:53.265169 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:53.264756    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"
Sep 12 01:03:53.286159 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:03:53.286105    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g" podStartSLOduration=1.286092527 podStartE2EDuration="1.286092527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:03:53.285099353 +0000 UTC m=+2065.489100175" watchObservedRunningTime="2026-09-12 01:03:53.286092527 +0000 UTC m=+2065.490093349"
Sep 12 01:04:04.280908 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:04.280868    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"
Sep 12 01:04:06.335644 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:06.335535    2567 generic.go:356] "Generic (PLEG): container finished" podID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerID="cdcee6b539698c9c2ae8556500b18f910f08f7649e61e0fe8ae20429e8b798d1" exitCode=0
Sep 12 01:04:06.335644 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:06.335610    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4" event={"ID":"f058bf56-36ef-49d4-a180-6a47e3d4bdd8","Type":"ContainerDied","Data":"cdcee6b539698c9c2ae8556500b18f910f08f7649e61e0fe8ae20429e8b798d1"}
Sep 12 01:04:06.336105 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:06.336026    2567 scope.go:117] "RemoveContainer" containerID="cdcee6b539698c9c2ae8556500b18f910f08f7649e61e0fe8ae20429e8b798d1"
Sep 12 01:04:06.337089 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:06.337061    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:04:06.337151 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:06.337134    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:04:06.337192 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:06.337173    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:04:07.347328 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:07.347268    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4" event={"ID":"f058bf56-36ef-49d4-a180-6a47e3d4bdd8","Type":"ContainerStarted","Data":"875bf81c1435c9e8e88423bc3fbcbf09d0220b79bf5863eb94acf4fed1234001"}
Sep 12 01:04:07.512939 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:07.512899    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:04:07.513216 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:07.513188    2567 patch_prober.go:28] interesting pod/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 12 01:04:07.513270 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:07.513247    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4" podUID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 12 01:04:08.513516 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:08.513476    2567 patch_prober.go:28] interesting pod/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 12 01:04:08.513959 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:08.513544    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4" podUID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 12 01:04:09.512868 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:09.512828    2567 patch_prober.go:28] interesting pod/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 12 01:04:09.513068 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:09.512887    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4" podUID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 12 01:04:10.513382 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:10.513339    2567 patch_prober.go:28] interesting pod/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 12 01:04:10.513821 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:10.513401    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4" podUID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 12 01:04:11.513338 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:11.513295    2567 patch_prober.go:28] interesting pod/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 12 01:04:11.513536 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:11.513359    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4" podUID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 12 01:04:12.517113 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:12.517074    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:04:12.517601 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:12.517358    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:04:12.517964 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:12.517945    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:04:36.023489 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.023451    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"]
Sep 12 01:04:36.027212 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.027187    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.029916 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.029889    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-9a2657\"/\"kube-root-ca.crt\""
Sep 12 01:04:36.030082 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.029929    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-9a2657\"/\"openshift-service-ca.crt\""
Sep 12 01:04:36.030949 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.030928    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-9a2657\"/\"ipp-route-9a2657-kserve-self-signed-certs\""
Sep 12 01:04:36.031085 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.030950    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-9a2657\"/\"default-dockercfg-r4bkq\""
Sep 12 01:04:36.036510 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.036482    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"]
Sep 12 01:04:36.110259 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.110220    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-home\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.110478 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.110272    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-tmp-dir\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.110478 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.110379    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-tls-certs\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.110478 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.110438    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-model-cache\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.110657 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.110489    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-dshm\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.110657 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.110564    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx642\" (UniqueName: \"kubernetes.io/projected/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-kube-api-access-rx642\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.211669 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.211617    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-tls-certs\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.211869 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.211689    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-model-cache\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.211869 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.211713    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-dshm\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.211869 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.211752    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx642\" (UniqueName: \"kubernetes.io/projected/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-kube-api-access-rx642\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.211869 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.211777    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-home\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.211869 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.211811    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-tmp-dir\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.212201 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.212179    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-model-cache\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.212267 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.212241    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-tmp-dir\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.212335 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.212266    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-home\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.214080 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.214058    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-dshm\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.214255 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.214236    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-tls-certs\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.221619 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.221556    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx642\" (UniqueName: \"kubernetes.io/projected/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-kube-api-access-rx642\") pod \"ipp-route-9a2657-kserve-585b74c-2d8hw\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") " pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.337516 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.337417    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:36.612849 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:36.612818    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"]
Sep 12 01:04:36.615062 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:04:36.615023    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fd139db_ba36_4baa_ae0f_d8df7ceea65a.slice/crio-5605491bb3f1817e71ebcdb758d07d1b971b7e96b3c78f1fee187a9964ce983c WatchSource:0}: Error finding container 5605491bb3f1817e71ebcdb758d07d1b971b7e96b3c78f1fee187a9964ce983c: Status 404 returned error can't find the container with id 5605491bb3f1817e71ebcdb758d07d1b971b7e96b3c78f1fee187a9964ce983c
Sep 12 01:04:37.505341 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:37.505303    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw" event={"ID":"0fd139db-ba36-4baa-ae0f-d8df7ceea65a","Type":"ContainerStarted","Data":"e60f0b91310cf59154901800858dbf6922f722f110853f27792a591b3b594443"}
Sep 12 01:04:37.505341 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:37.505348    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw" event={"ID":"0fd139db-ba36-4baa-ae0f-d8df7ceea65a","Type":"ContainerStarted","Data":"5605491bb3f1817e71ebcdb758d07d1b971b7e96b3c78f1fee187a9964ce983c"}
Sep 12 01:04:37.529208 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:37.529147    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw" podStartSLOduration=1.5291288060000001 podStartE2EDuration="1.529128806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:04:37.527030568 +0000 UTC m=+2109.731031397" watchObservedRunningTime="2026-09-12 01:04:37.529128806 +0000 UTC m=+2109.733129628"
Sep 12 01:04:46.338621 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:46.338564    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:46.339166 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:46.338642    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:46.351353 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:46.351324    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:04:46.563827 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:04:46.563796    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:05:10.214514 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.214478    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"]
Sep 12 01:05:10.217654 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.217635    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.218237 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.218211    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q85fz\" (UniqueName: \"kubernetes.io/projected/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-kube-api-access-q85fz\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.218328 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.218269    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-token\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.218328 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.218307    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-podinfo\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.218399 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.218341    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-envoy\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.218399 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.218393    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-workload-certs\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.218477 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.218450    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-data\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.218528 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.218492    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-credential-socket\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.218528 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.218513    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.218626 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.218598    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-workload-socket\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.220333 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.220315    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-783804-openshift-default-dockercfg-7sxpc\""
Sep 12 01:05:10.230506 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.230479    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"]
Sep 12 01:05:10.319362 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.319316    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-workload-socket\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.319608 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.319396    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q85fz\" (UniqueName: \"kubernetes.io/projected/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-kube-api-access-q85fz\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.319608 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.319439    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-token\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.319608 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.319464    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-podinfo\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.319608 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.319498    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-envoy\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.319608 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.319549    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-workload-certs\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.319907 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.319613    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-data\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.319907 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.319654    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-credential-socket\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.319907 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.319682    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.320071 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.319972    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-data\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.320071 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.320017    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-workload-certs\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.320290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.320243    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-credential-socket\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.320437 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.320419    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istiod-ca-cert\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.320658 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.320627    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-workload-socket\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.323259 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.323069    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-envoy\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.323408 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.323285    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-podinfo\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.329095 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.329063    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-token\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.329242 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.329199    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q85fz\" (UniqueName: \"kubernetes.io/projected/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-kube-api-access-q85fz\") pod \"e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") " pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.528597 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.528536    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:10.810687 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:05:10.810655    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5b8f09_284b_4c99_ac99_ed7c674ab28a.slice/crio-c6f82f99e96ab9dee4872633122a9dd8691bcc95acb81b8d7ee47853fea444fd WatchSource:0}: Error finding container c6f82f99e96ab9dee4872633122a9dd8691bcc95acb81b8d7ee47853fea444fd: Status 404 returned error can't find the container with id c6f82f99e96ab9dee4872633122a9dd8691bcc95acb81b8d7ee47853fea444fd
Sep 12 01:05:10.811846 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.811823    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"]
Sep 12 01:05:10.813153 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.813121    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:05:10.813226 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.813186    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:05:10.813226 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:10.813218    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:05:11.679474 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:11.679439    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" event={"ID":"9e5b8f09-284b-4c99-ac99-ed7c674ab28a","Type":"ContainerStarted","Data":"d1ad42b9eae54926b60eb8cd67b096900a462d18f9ff3c9d35a67b82fa0fda31"}
Sep 12 01:05:11.679964 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:11.679494    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" event={"ID":"9e5b8f09-284b-4c99-ac99-ed7c674ab28a","Type":"ContainerStarted","Data":"c6f82f99e96ab9dee4872633122a9dd8691bcc95acb81b8d7ee47853fea444fd"}
Sep 12 01:05:11.701768 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:11.701717    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" podStartSLOduration=1.7016976179999999 podStartE2EDuration="1.701697618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:05:11.69876495 +0000 UTC m=+2143.902765785" watchObservedRunningTime="2026-09-12 01:05:11.701697618 +0000 UTC m=+2143.905698437"
Sep 12 01:05:12.528837 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:12.528797    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:12.534123 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:12.534092    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:12.685142 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:12.685100    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:12.686124 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:12.686100    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:17.539164 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:17.539129    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"]
Sep 12 01:05:17.542556 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:17.542537    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"
Sep 12 01:05:17.545074 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:17.545042    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-783804\""
Sep 12 01:05:17.545201 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:17.545114    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-783804-dockercfg-6ktzc\""
Sep 12 01:05:17.554359 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:17.554333    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"]
Sep 12 01:05:17.583864 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:17.583824    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e92f20-2e25-462f-9fc9-dd91dd6c59f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc\" (UID: \"49e92f20-2e25-462f-9fc9-dd91dd6c59f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"
Sep 12 01:05:17.584040 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:17.583951    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm2n4\" (UniqueName: \"kubernetes.io/projected/49e92f20-2e25-462f-9fc9-dd91dd6c59f2-kube-api-access-dm2n4\") pod \"payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc\" (UID: \"49e92f20-2e25-462f-9fc9-dd91dd6c59f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"
Sep 12 01:05:17.685558 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:17.685514    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2n4\" (UniqueName: \"kubernetes.io/projected/49e92f20-2e25-462f-9fc9-dd91dd6c59f2-kube-api-access-dm2n4\") pod \"payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc\" (UID: \"49e92f20-2e25-462f-9fc9-dd91dd6c59f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"
Sep 12 01:05:17.685780 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:17.685662    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e92f20-2e25-462f-9fc9-dd91dd6c59f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc\" (UID: \"49e92f20-2e25-462f-9fc9-dd91dd6c59f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"
Sep 12 01:05:17.686380 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:17.686358    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e92f20-2e25-462f-9fc9-dd91dd6c59f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc\" (UID: \"49e92f20-2e25-462f-9fc9-dd91dd6c59f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"
Sep 12 01:05:17.695090 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:17.695056    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2n4\" (UniqueName: \"kubernetes.io/projected/49e92f20-2e25-462f-9fc9-dd91dd6c59f2-kube-api-access-dm2n4\") pod \"payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc\" (UID: \"49e92f20-2e25-462f-9fc9-dd91dd6c59f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"
Sep 12 01:05:17.854410 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:17.854293    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"
Sep 12 01:05:18.128052 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:18.128026    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"]
Sep 12 01:05:18.129970 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:05:18.129942    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e92f20_2e25_462f_9fc9_dd91dd6c59f2.slice/crio-f085f2399435a2bf43f543499a1e59c27d9c132aa9b1ba413516c4978554ff6b WatchSource:0}: Error finding container f085f2399435a2bf43f543499a1e59c27d9c132aa9b1ba413516c4978554ff6b: Status 404 returned error can't find the container with id f085f2399435a2bf43f543499a1e59c27d9c132aa9b1ba413516c4978554ff6b
Sep 12 01:05:18.717765 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:18.717726    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc" event={"ID":"49e92f20-2e25-462f-9fc9-dd91dd6c59f2","Type":"ContainerStarted","Data":"9c13ec57fa9549826ca853ac733b6b73ca1b836c56a585a292b64d04aa8da7b7"}
Sep 12 01:05:18.718251 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:18.717786    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc" event={"ID":"49e92f20-2e25-462f-9fc9-dd91dd6c59f2","Type":"ContainerStarted","Data":"f085f2399435a2bf43f543499a1e59c27d9c132aa9b1ba413516c4978554ff6b"}
Sep 12 01:05:18.718251 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:18.717912    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"
Sep 12 01:05:18.739446 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:18.739393    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc" podStartSLOduration=1.739374278 podStartE2EDuration="1.739374278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:05:18.737277416 +0000 UTC m=+2150.941278237" watchObservedRunningTime="2026-09-12 01:05:18.739374278 +0000 UTC m=+2150.943375100"
Sep 12 01:05:29.100019 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:29.099976    2567 scope.go:117] "RemoveContainer" containerID="37160fbd8d09544f8381b151ea9020b37ef916ddb94e1fd4d545a93548befba2"
Sep 12 01:05:29.108804 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:29.108774    2567 scope.go:117] "RemoveContainer" containerID="363e7cacc49c6f81d5704d23d2ff2bc1a7554b094482092644732d2cc08cca31"
Sep 12 01:05:29.725763 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:29.725728    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"
Sep 12 01:05:31.797288 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:31.797252    2567 generic.go:356] "Generic (PLEG): container finished" podID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerID="d1ad42b9eae54926b60eb8cd67b096900a462d18f9ff3c9d35a67b82fa0fda31" exitCode=0
Sep 12 01:05:31.797774 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:31.797324    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" event={"ID":"9e5b8f09-284b-4c99-ac99-ed7c674ab28a","Type":"ContainerDied","Data":"d1ad42b9eae54926b60eb8cd67b096900a462d18f9ff3c9d35a67b82fa0fda31"}
Sep 12 01:05:31.797861 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:31.797842    2567 scope.go:117] "RemoveContainer" containerID="d1ad42b9eae54926b60eb8cd67b096900a462d18f9ff3c9d35a67b82fa0fda31"
Sep 12 01:05:31.801948 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:31.801906    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:05:31.802099 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:31.802004    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:05:31.802099 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:31.802051    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:05:32.804453 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:32.804415    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" event={"ID":"9e5b8f09-284b-4c99-ac99-ed7c674ab28a","Type":"ContainerStarted","Data":"828e03956f3e93776d825ea0cb67cba6f32d131bb355ffae562ce055ee4eb52c"}
Sep 12 01:05:33.529160 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:33.529112    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:33.529393 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:33.529366    2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 12 01:05:33.529473 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:33.529424    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 12 01:05:34.529421 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:34.529385    2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 12 01:05:34.529792 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:34.529445    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 12 01:05:35.529178 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:35.529138    2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 12 01:05:35.529353 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:35.529210    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 12 01:05:36.529113 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:36.529073    2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 12 01:05:36.529542 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:36.529137    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 12 01:05:37.529782 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:37.529745    2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 12 01:05:37.530184 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:37.529808    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 12 01:05:38.533264 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:38.533233    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:38.533678 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:38.533493    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:38.534302 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:38.534284    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:39.258660 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:39.258625    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk"]
Sep 12 01:05:39.261027 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:39.261003    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk"
Sep 12 01:05:39.263863 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:39.263838    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:05:39.264006 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:39.263934    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 01:05:39.271291 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:39.271258    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk"]
Sep 12 01:05:39.413317 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:39.413277    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654ba166-3867-4788-a4e5-795068aa9391-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk\" (UID: \"654ba166-3867-4788-a4e5-795068aa9391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk"
Sep 12 01:05:39.514090 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:39.513982    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654ba166-3867-4788-a4e5-795068aa9391-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk\" (UID: \"654ba166-3867-4788-a4e5-795068aa9391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk"
Sep 12 01:05:39.514635 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:39.514614    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654ba166-3867-4788-a4e5-795068aa9391-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk\" (UID: \"654ba166-3867-4788-a4e5-795068aa9391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk"
Sep 12 01:05:39.573141 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:39.573107    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk"
Sep 12 01:05:39.846336 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:39.846313    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk"]
Sep 12 01:05:39.848901 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:05:39.848874    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654ba166_3867_4788_a4e5_795068aa9391.slice/crio-0bec3bbb2a78932b33c9f3594802df63674927cfa38e7f06cf49fc77f9884323 WatchSource:0}: Error finding container 0bec3bbb2a78932b33c9f3594802df63674927cfa38e7f06cf49fc77f9884323: Status 404 returned error can't find the container with id 0bec3bbb2a78932b33c9f3594802df63674927cfa38e7f06cf49fc77f9884323
Sep 12 01:05:40.847276 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:40.847160    2567 generic.go:356] "Generic (PLEG): container finished" podID="654ba166-3867-4788-a4e5-795068aa9391" containerID="1371fdbb4e075c13b731b36e504f1a67a88428d28919a6d1dfd245e37906bc87" exitCode=0
Sep 12 01:05:40.847276 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:40.847225    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk" event={"ID":"654ba166-3867-4788-a4e5-795068aa9391","Type":"ContainerDied","Data":"1371fdbb4e075c13b731b36e504f1a67a88428d28919a6d1dfd245e37906bc87"}
Sep 12 01:05:40.847890 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:40.847280    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk" event={"ID":"654ba166-3867-4788-a4e5-795068aa9391","Type":"ContainerStarted","Data":"0bec3bbb2a78932b33c9f3594802df63674927cfa38e7f06cf49fc77f9884323"}
Sep 12 01:05:41.985676 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:41.985651    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk"
Sep 12 01:05:42.137149 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:42.137044    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654ba166-3867-4788-a4e5-795068aa9391-maas-api-service-ca\") pod \"654ba166-3867-4788-a4e5-795068aa9391\" (UID: \"654ba166-3867-4788-a4e5-795068aa9391\") "
Sep 12 01:05:42.137411 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:42.137386    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654ba166-3867-4788-a4e5-795068aa9391-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "654ba166-3867-4788-a4e5-795068aa9391" (UID: "654ba166-3867-4788-a4e5-795068aa9391"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:05:42.238672 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:42.238634    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654ba166-3867-4788-a4e5-795068aa9391-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:05:42.858757 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:42.858726    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk"
Sep 12 01:05:42.858922 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:42.858759    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk" event={"ID":"654ba166-3867-4788-a4e5-795068aa9391","Type":"ContainerDied","Data":"0bec3bbb2a78932b33c9f3594802df63674927cfa38e7f06cf49fc77f9884323"}
Sep 12 01:05:42.858922 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:42.858805    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bec3bbb2a78932b33c9f3594802df63674927cfa38e7f06cf49fc77f9884323"
Sep 12 01:05:45.708320 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:45.708275    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"]
Sep 12 01:05:45.708843 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:45.708788    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc" podUID="49e92f20-2e25-462f-9fc9-dd91dd6c59f2" containerName="payload-pre-processing" containerID="cri-o://9c13ec57fa9549826ca853ac733b6b73ca1b836c56a585a292b64d04aa8da7b7" gracePeriod=30
Sep 12 01:05:45.878362 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:45.878327    2567 generic.go:356] "Generic (PLEG): container finished" podID="49e92f20-2e25-462f-9fc9-dd91dd6c59f2" containerID="9c13ec57fa9549826ca853ac733b6b73ca1b836c56a585a292b64d04aa8da7b7" exitCode=0
Sep 12 01:05:45.878540 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:45.878375    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc" event={"ID":"49e92f20-2e25-462f-9fc9-dd91dd6c59f2","Type":"ContainerDied","Data":"9c13ec57fa9549826ca853ac733b6b73ca1b836c56a585a292b64d04aa8da7b7"}
Sep 12 01:05:45.881750 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:45.881708    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk"]
Sep 12 01:05:45.884302 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:45.884278    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-783804-k8brk"]
Sep 12 01:05:45.974610 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:45.974561    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"
Sep 12 01:05:46.075677 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:46.075617    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm2n4\" (UniqueName: \"kubernetes.io/projected/49e92f20-2e25-462f-9fc9-dd91dd6c59f2-kube-api-access-dm2n4\") pod \"49e92f20-2e25-462f-9fc9-dd91dd6c59f2\" (UID: \"49e92f20-2e25-462f-9fc9-dd91dd6c59f2\") "
Sep 12 01:05:46.075891 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:46.075760    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e92f20-2e25-462f-9fc9-dd91dd6c59f2-plugins-config-volume\") pod \"49e92f20-2e25-462f-9fc9-dd91dd6c59f2\" (UID: \"49e92f20-2e25-462f-9fc9-dd91dd6c59f2\") "
Sep 12 01:05:46.076271 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:46.076237    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e92f20-2e25-462f-9fc9-dd91dd6c59f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49e92f20-2e25-462f-9fc9-dd91dd6c59f2" (UID: "49e92f20-2e25-462f-9fc9-dd91dd6c59f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:05:46.077920 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:46.077893    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e92f20-2e25-462f-9fc9-dd91dd6c59f2-kube-api-access-dm2n4" (OuterVolumeSpecName: "kube-api-access-dm2n4") pod "49e92f20-2e25-462f-9fc9-dd91dd6c59f2" (UID: "49e92f20-2e25-462f-9fc9-dd91dd6c59f2"). InnerVolumeSpecName "kube-api-access-dm2n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:05:46.176906 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:46.176850    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm2n4\" (UniqueName: \"kubernetes.io/projected/49e92f20-2e25-462f-9fc9-dd91dd6c59f2-kube-api-access-dm2n4\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:05:46.176906 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:46.176899    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e92f20-2e25-462f-9fc9-dd91dd6c59f2-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:05:46.274078 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:46.274039    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="654ba166-3867-4788-a4e5-795068aa9391" path="/var/lib/kubelet/pods/654ba166-3867-4788-a4e5-795068aa9391/volumes"
Sep 12 01:05:46.884591 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:46.884538    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"
Sep 12 01:05:46.885074 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:46.884541    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc" event={"ID":"49e92f20-2e25-462f-9fc9-dd91dd6c59f2","Type":"ContainerDied","Data":"f085f2399435a2bf43f543499a1e59c27d9c132aa9b1ba413516c4978554ff6b"}
Sep 12 01:05:46.885074 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:46.884697    2567 scope.go:117] "RemoveContainer" containerID="9c13ec57fa9549826ca853ac733b6b73ca1b836c56a585a292b64d04aa8da7b7"
Sep 12 01:05:46.901221 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:46.901186    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"]
Sep 12 01:05:46.904862 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:46.904831    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-783804-746986ddc8-krjgc"]
Sep 12 01:05:48.274145 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:48.274109    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49e92f20-2e25-462f-9fc9-dd91dd6c59f2" path="/var/lib/kubelet/pods/49e92f20-2e25-462f-9fc9-dd91dd6c59f2/volumes"
Sep 12 01:05:52.609840 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:52.609801    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"]
Sep 12 01:05:52.610333 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:52.610100    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerName="istio-proxy" containerID="cri-o://828e03956f3e93776d825ea0cb67cba6f32d131bb355ffae562ce055ee4eb52c" gracePeriod=30
Sep 12 01:05:53.535755 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:53.535714    2567 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 01:05:53.535948 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:53.535792    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 01:05:56.099693 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.099654    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm"]
Sep 12 01:05:56.100102 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.099996    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e92f20-2e25-462f-9fc9-dd91dd6c59f2" containerName="payload-pre-processing"
Sep 12 01:05:56.100102 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.100008    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e92f20-2e25-462f-9fc9-dd91dd6c59f2" containerName="payload-pre-processing"
Sep 12 01:05:56.100102 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.100024    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="654ba166-3867-4788-a4e5-795068aa9391" containerName="revoke-keys"
Sep 12 01:05:56.100102 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.100029    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="654ba166-3867-4788-a4e5-795068aa9391" containerName="revoke-keys"
Sep 12 01:05:56.100102 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.100095    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="654ba166-3867-4788-a4e5-795068aa9391" containerName="revoke-keys"
Sep 12 01:05:56.100102 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.100104    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="49e92f20-2e25-462f-9fc9-dd91dd6c59f2" containerName="payload-pre-processing"
Sep 12 01:05:56.103033 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.103013    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm"
Sep 12 01:05:56.105544 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.105514    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:05:56.105544 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.105519    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 01:05:56.111686 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.111660    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm"]
Sep 12 01:05:56.162825 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.162779    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3f1364-3325-49dd-b268-9d37d0cf2d2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm\" (UID: \"ef3f1364-3325-49dd-b268-9d37d0cf2d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm"
Sep 12 01:05:56.263643 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.263603    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3f1364-3325-49dd-b268-9d37d0cf2d2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm\" (UID: \"ef3f1364-3325-49dd-b268-9d37d0cf2d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm"
Sep 12 01:05:56.264266 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.264242    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3f1364-3325-49dd-b268-9d37d0cf2d2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm\" (UID: \"ef3f1364-3325-49dd-b268-9d37d0cf2d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm"
Sep 12 01:05:56.414630 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.414519    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm"
Sep 12 01:05:56.685369 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.685338    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm"]
Sep 12 01:05:56.687997 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:05:56.687968    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3f1364_3325_49dd_b268_9d37d0cf2d2c.slice/crio-d7f2e8f9e997d6e98b22049895d836ce52aae074670e4529fde3e7fc278591c6 WatchSource:0}: Error finding container d7f2e8f9e997d6e98b22049895d836ce52aae074670e4529fde3e7fc278591c6: Status 404 returned error can't find the container with id d7f2e8f9e997d6e98b22049895d836ce52aae074670e4529fde3e7fc278591c6
Sep 12 01:05:56.943878 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:56.943826    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm" event={"ID":"ef3f1364-3325-49dd-b268-9d37d0cf2d2c","Type":"ContainerStarted","Data":"d7f2e8f9e997d6e98b22049895d836ce52aae074670e4529fde3e7fc278591c6"}
Sep 12 01:05:57.872162 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.872092    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:57.951135 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.951096    2567 generic.go:356] "Generic (PLEG): container finished" podID="ef3f1364-3325-49dd-b268-9d37d0cf2d2c" containerID="89ff766a9049193341528b489e91ec1924426c06eed1a1282333abc3efce40c0" exitCode=0
Sep 12 01:05:57.951328 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.951186    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm" event={"ID":"ef3f1364-3325-49dd-b268-9d37d0cf2d2c","Type":"ContainerDied","Data":"89ff766a9049193341528b489e91ec1924426c06eed1a1282333abc3efce40c0"}
Sep 12 01:05:57.952986 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.952955    2567 generic.go:356] "Generic (PLEG): container finished" podID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerID="828e03956f3e93776d825ea0cb67cba6f32d131bb355ffae562ce055ee4eb52c" exitCode=0
Sep 12 01:05:57.953078 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.953012    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"
Sep 12 01:05:57.953078 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.953039    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" event={"ID":"9e5b8f09-284b-4c99-ac99-ed7c674ab28a","Type":"ContainerDied","Data":"828e03956f3e93776d825ea0cb67cba6f32d131bb355ffae562ce055ee4eb52c"}
Sep 12 01:05:57.953157 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.953094    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw" event={"ID":"9e5b8f09-284b-4c99-ac99-ed7c674ab28a","Type":"ContainerDied","Data":"c6f82f99e96ab9dee4872633122a9dd8691bcc95acb81b8d7ee47853fea444fd"}
Sep 12 01:05:57.953157 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.953119    2567 scope.go:117] "RemoveContainer" containerID="828e03956f3e93776d825ea0cb67cba6f32d131bb355ffae562ce055ee4eb52c"
Sep 12 01:05:57.965527 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.965501    2567 scope.go:117] "RemoveContainer" containerID="d1ad42b9eae54926b60eb8cd67b096900a462d18f9ff3c9d35a67b82fa0fda31"
Sep 12 01:05:57.975093 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.975066    2567 scope.go:117] "RemoveContainer" containerID="828e03956f3e93776d825ea0cb67cba6f32d131bb355ffae562ce055ee4eb52c"
Sep 12 01:05:57.975507 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:05:57.975486    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828e03956f3e93776d825ea0cb67cba6f32d131bb355ffae562ce055ee4eb52c\": container with ID starting with 828e03956f3e93776d825ea0cb67cba6f32d131bb355ffae562ce055ee4eb52c not found: ID does not exist" containerID="828e03956f3e93776d825ea0cb67cba6f32d131bb355ffae562ce055ee4eb52c"
Sep 12 01:05:57.975561 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.975518    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828e03956f3e93776d825ea0cb67cba6f32d131bb355ffae562ce055ee4eb52c"} err="failed to get container status \"828e03956f3e93776d825ea0cb67cba6f32d131bb355ffae562ce055ee4eb52c\": rpc error: code = NotFound desc = could not find container \"828e03956f3e93776d825ea0cb67cba6f32d131bb355ffae562ce055ee4eb52c\": container with ID starting with 828e03956f3e93776d825ea0cb67cba6f32d131bb355ffae562ce055ee4eb52c not found: ID does not exist"
Sep 12 01:05:57.975561 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.975540    2567 scope.go:117] "RemoveContainer" containerID="d1ad42b9eae54926b60eb8cd67b096900a462d18f9ff3c9d35a67b82fa0fda31"
Sep 12 01:05:57.975828 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:05:57.975814    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ad42b9eae54926b60eb8cd67b096900a462d18f9ff3c9d35a67b82fa0fda31\": container with ID starting with d1ad42b9eae54926b60eb8cd67b096900a462d18f9ff3c9d35a67b82fa0fda31 not found: ID does not exist" containerID="d1ad42b9eae54926b60eb8cd67b096900a462d18f9ff3c9d35a67b82fa0fda31"
Sep 12 01:05:57.975878 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.975833    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ad42b9eae54926b60eb8cd67b096900a462d18f9ff3c9d35a67b82fa0fda31"} err="failed to get container status \"d1ad42b9eae54926b60eb8cd67b096900a462d18f9ff3c9d35a67b82fa0fda31\": rpc error: code = NotFound desc = could not find container \"d1ad42b9eae54926b60eb8cd67b096900a462d18f9ff3c9d35a67b82fa0fda31\": container with ID starting with d1ad42b9eae54926b60eb8cd67b096900a462d18f9ff3c9d35a67b82fa0fda31 not found: ID does not exist"
Sep 12 01:05:57.980226 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980192    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-credential-socket\") pod \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") "
Sep 12 01:05:57.980382 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980256    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q85fz\" (UniqueName: \"kubernetes.io/projected/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-kube-api-access-q85fz\") pod \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") "
Sep 12 01:05:57.980382 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980288    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-envoy\") pod \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") "
Sep 12 01:05:57.980382 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980323    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-workload-certs\") pod \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") "
Sep 12 01:05:57.980382 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980346    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-data\") pod \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") "
Sep 12 01:05:57.980638 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980402    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-token\") pod \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") "
Sep 12 01:05:57.980638 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980456    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istiod-ca-cert\") pod \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") "
Sep 12 01:05:57.980638 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980510    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-podinfo\") pod \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") "
Sep 12 01:05:57.980638 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980540    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e5b8f09-284b-4c99-ac99-ed7c674ab28a" (UID: "9e5b8f09-284b-4c99-ac99-ed7c674ab28a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:05:57.980638 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980598    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-workload-socket\") pod \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\" (UID: \"9e5b8f09-284b-4c99-ac99-ed7c674ab28a\") "
Sep 12 01:05:57.980951 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980907    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e5b8f09-284b-4c99-ac99-ed7c674ab28a" (UID: "9e5b8f09-284b-4c99-ac99-ed7c674ab28a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:05:57.980951 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980933    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-credential-socket\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:05:57.981049 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980949    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e5b8f09-284b-4c99-ac99-ed7c674ab28a" (UID: "9e5b8f09-284b-4c99-ac99-ed7c674ab28a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:05:57.981049 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.980986    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e5b8f09-284b-4c99-ac99-ed7c674ab28a" (UID: "9e5b8f09-284b-4c99-ac99-ed7c674ab28a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:05:57.981135 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.981119    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e5b8f09-284b-4c99-ac99-ed7c674ab28a" (UID: "9e5b8f09-284b-4c99-ac99-ed7c674ab28a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:05:57.982625 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.982598    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e5b8f09-284b-4c99-ac99-ed7c674ab28a" (UID: "9e5b8f09-284b-4c99-ac99-ed7c674ab28a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:05:57.982770 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.982722    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e5b8f09-284b-4c99-ac99-ed7c674ab28a" (UID: "9e5b8f09-284b-4c99-ac99-ed7c674ab28a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 01:05:57.982960 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.982926    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-kube-api-access-q85fz" (OuterVolumeSpecName: "kube-api-access-q85fz") pod "9e5b8f09-284b-4c99-ac99-ed7c674ab28a" (UID: "9e5b8f09-284b-4c99-ac99-ed7c674ab28a"). InnerVolumeSpecName "kube-api-access-q85fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:05:57.983158 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:57.983133    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e5b8f09-284b-4c99-ac99-ed7c674ab28a" (UID: "9e5b8f09-284b-4c99-ac99-ed7c674ab28a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:05:58.081886 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:58.081848    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-token\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:05:58.081886 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:58.081878    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istiod-ca-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:05:58.081886 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:58.081887    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-podinfo\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:05:58.082159 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:58.081896    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-workload-socket\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:05:58.082159 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:58.081906    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q85fz\" (UniqueName: \"kubernetes.io/projected/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-kube-api-access-q85fz\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:05:58.082159 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:58.081917    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-envoy\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:05:58.082159 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:58.081925    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-workload-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:05:58.082159 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:58.081932    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5b8f09-284b-4c99-ac99-ed7c674ab28a-istio-data\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:05:58.276474 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:58.276438    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"]
Sep 12 01:05:58.278175 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:58.278150    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-783804-openshift-default-579dbbbcb-x97pw"]
Sep 12 01:05:59.097234 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:59.097201    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm"
Sep 12 01:05:59.190245 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:59.190197    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3f1364-3325-49dd-b268-9d37d0cf2d2c-maas-api-service-ca\") pod \"ef3f1364-3325-49dd-b268-9d37d0cf2d2c\" (UID: \"ef3f1364-3325-49dd-b268-9d37d0cf2d2c\") "
Sep 12 01:05:59.190616 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:59.190563    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3f1364-3325-49dd-b268-9d37d0cf2d2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef3f1364-3325-49dd-b268-9d37d0cf2d2c" (UID: "ef3f1364-3325-49dd-b268-9d37d0cf2d2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:05:59.291764 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:59.291726    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3f1364-3325-49dd-b268-9d37d0cf2d2c-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:05:59.965743 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:59.965705    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm"
Sep 12 01:05:59.965930 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:59.965743    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm" event={"ID":"ef3f1364-3325-49dd-b268-9d37d0cf2d2c","Type":"ContainerDied","Data":"d7f2e8f9e997d6e98b22049895d836ce52aae074670e4529fde3e7fc278591c6"}
Sep 12 01:05:59.965930 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:05:59.965799    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7f2e8f9e997d6e98b22049895d836ce52aae074670e4529fde3e7fc278591c6"
Sep 12 01:06:00.274285 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:00.274238    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" path="/var/lib/kubelet/pods/9e5b8f09-284b-4c99-ac99-ed7c674ab28a/volumes"
Sep 12 01:06:11.768422 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:11.768384    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"]
Sep 12 01:06:11.768980 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:11.768673    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4" podUID="51ce9b50-1da2-4df5-96a0-8ba13cf38088" containerName="payload-pre-processing" containerID="cri-o://f7cd981ffd437e62393e580576509574383f1cebc9ec4dc0664c0eac83937033" gracePeriod=30
Sep 12 01:06:11.949838 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:11.949800    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm"]
Sep 12 01:06:11.953193 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:11.953159    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9a2657-6hlxm"]
Sep 12 01:06:12.009360 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.009333    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"
Sep 12 01:06:12.028469 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.028286    2567 generic.go:356] "Generic (PLEG): container finished" podID="51ce9b50-1da2-4df5-96a0-8ba13cf38088" containerID="f7cd981ffd437e62393e580576509574383f1cebc9ec4dc0664c0eac83937033" exitCode=0
Sep 12 01:06:12.028469 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.028442    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4" event={"ID":"51ce9b50-1da2-4df5-96a0-8ba13cf38088","Type":"ContainerDied","Data":"f7cd981ffd437e62393e580576509574383f1cebc9ec4dc0664c0eac83937033"}
Sep 12 01:06:12.032805 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.028587    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4" event={"ID":"51ce9b50-1da2-4df5-96a0-8ba13cf38088","Type":"ContainerDied","Data":"fb2380777c407c8bfd866c413a558f0a53d956b305cd24e70ac56867fca149f1"}
Sep 12 01:06:12.032805 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.028628    2567 scope.go:117] "RemoveContainer" containerID="f7cd981ffd437e62393e580576509574383f1cebc9ec4dc0664c0eac83937033"
Sep 12 01:06:12.032805 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.028826    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"
Sep 12 01:06:12.046208 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.046181    2567 scope.go:117] "RemoveContainer" containerID="f7cd981ffd437e62393e580576509574383f1cebc9ec4dc0664c0eac83937033"
Sep 12 01:06:12.046687 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:06:12.046658    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7cd981ffd437e62393e580576509574383f1cebc9ec4dc0664c0eac83937033\": container with ID starting with f7cd981ffd437e62393e580576509574383f1cebc9ec4dc0664c0eac83937033 not found: ID does not exist" containerID="f7cd981ffd437e62393e580576509574383f1cebc9ec4dc0664c0eac83937033"
Sep 12 01:06:12.046812 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.046694    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7cd981ffd437e62393e580576509574383f1cebc9ec4dc0664c0eac83937033"} err="failed to get container status \"f7cd981ffd437e62393e580576509574383f1cebc9ec4dc0664c0eac83937033\": rpc error: code = NotFound desc = could not find container \"f7cd981ffd437e62393e580576509574383f1cebc9ec4dc0664c0eac83937033\": container with ID starting with f7cd981ffd437e62393e580576509574383f1cebc9ec4dc0664c0eac83937033 not found: ID does not exist"
Sep 12 01:06:12.118950 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.118913    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvs68\" (UniqueName: \"kubernetes.io/projected/51ce9b50-1da2-4df5-96a0-8ba13cf38088-kube-api-access-cvs68\") pod \"51ce9b50-1da2-4df5-96a0-8ba13cf38088\" (UID: \"51ce9b50-1da2-4df5-96a0-8ba13cf38088\") "
Sep 12 01:06:12.119152 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.119045    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ce9b50-1da2-4df5-96a0-8ba13cf38088-plugins-config-volume\") pod \"51ce9b50-1da2-4df5-96a0-8ba13cf38088\" (UID: \"51ce9b50-1da2-4df5-96a0-8ba13cf38088\") "
Sep 12 01:06:12.119421 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.119395    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ce9b50-1da2-4df5-96a0-8ba13cf38088-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51ce9b50-1da2-4df5-96a0-8ba13cf38088" (UID: "51ce9b50-1da2-4df5-96a0-8ba13cf38088"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:06:12.121103 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.121070    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ce9b50-1da2-4df5-96a0-8ba13cf38088-kube-api-access-cvs68" (OuterVolumeSpecName: "kube-api-access-cvs68") pod "51ce9b50-1da2-4df5-96a0-8ba13cf38088" (UID: "51ce9b50-1da2-4df5-96a0-8ba13cf38088"). InnerVolumeSpecName "kube-api-access-cvs68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:06:12.220250 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.220207    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvs68\" (UniqueName: \"kubernetes.io/projected/51ce9b50-1da2-4df5-96a0-8ba13cf38088-kube-api-access-cvs68\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:06:12.220250 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.220243    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ce9b50-1da2-4df5-96a0-8ba13cf38088-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:06:12.273936 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.273901    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef3f1364-3325-49dd-b268-9d37d0cf2d2c" path="/var/lib/kubelet/pods/ef3f1364-3325-49dd-b268-9d37d0cf2d2c/volumes"
Sep 12 01:06:12.351329 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.351286    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"]
Sep 12 01:06:12.356891 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:12.356845    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9a2657-7d9c5fb7db-mklz4"]
Sep 12 01:06:14.273711 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:14.273676    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51ce9b50-1da2-4df5-96a0-8ba13cf38088" path="/var/lib/kubelet/pods/51ce9b50-1da2-4df5-96a0-8ba13cf38088/volumes"
Sep 12 01:06:17.300887 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:17.300846    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"]
Sep 12 01:06:29.132514 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:29.132479    2567 scope.go:117] "RemoveContainer" containerID="462bfac48f6bf5a56dca3fa710c6f1b09badf36e63aff2ebdd11ce71dfe55e68"
Sep 12 01:06:29.140819 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:29.140799    2567 scope.go:117] "RemoveContainer" containerID="2dd70412cd7a9a8e90e22fc33f7179c8808c209c2a7b466f6a077a303a017289"
Sep 12 01:06:29.148689 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:29.148671    2567 scope.go:117] "RemoveContainer" containerID="ad4d0c1d675a7e7db03fb635bd89ea81ffe79009e9bc71c3064084a1d6eac8c4"
Sep 12 01:06:29.156283 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:29.156265    2567 scope.go:117] "RemoveContainer" containerID="c46a9de9cc191e97efa8310196a1363caebb64aef5b496ba4d8130f6c488acd7"
Sep 12 01:06:29.163834 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:29.163815    2567 scope.go:117] "RemoveContainer" containerID="eb56aacd7e69d8c87482f23e2115750303310066fa72154dc6cd9c733285eca1"
Sep 12 01:06:32.320186 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.320139    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw" podUID="0fd139db-ba36-4baa-ae0f-d8df7ceea65a" containerName="main" containerID="cri-o://e60f0b91310cf59154901800858dbf6922f722f110853f27792a591b3b594443" gracePeriod=45
Sep 12 01:06:32.577872 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.577801    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:06:32.609292 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.609259    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-model-cache\") pod \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") "
Sep 12 01:06:32.609292 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.609298    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-dshm\") pod \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") "
Sep 12 01:06:32.609553 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.609317    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-tls-certs\") pod \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") "
Sep 12 01:06:32.609553 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.609344    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx642\" (UniqueName: \"kubernetes.io/projected/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-kube-api-access-rx642\") pod \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") "
Sep 12 01:06:32.609553 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.609404    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-home\") pod \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") "
Sep 12 01:06:32.609553 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.609451    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-tmp-dir\") pod \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\" (UID: \"0fd139db-ba36-4baa-ae0f-d8df7ceea65a\") "
Sep 12 01:06:32.609837 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.609643    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-model-cache" (OuterVolumeSpecName: "model-cache") pod "0fd139db-ba36-4baa-ae0f-d8df7ceea65a" (UID: "0fd139db-ba36-4baa-ae0f-d8df7ceea65a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:06:32.609837 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.609743    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-model-cache\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:06:32.609960 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.609865    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0fd139db-ba36-4baa-ae0f-d8df7ceea65a" (UID: "0fd139db-ba36-4baa-ae0f-d8df7ceea65a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:06:32.610030 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.609964    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-home" (OuterVolumeSpecName: "home") pod "0fd139db-ba36-4baa-ae0f-d8df7ceea65a" (UID: "0fd139db-ba36-4baa-ae0f-d8df7ceea65a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:06:32.611811 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.611785    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-dshm" (OuterVolumeSpecName: "dshm") pod "0fd139db-ba36-4baa-ae0f-d8df7ceea65a" (UID: "0fd139db-ba36-4baa-ae0f-d8df7ceea65a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:06:32.611962 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.611903    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fd139db-ba36-4baa-ae0f-d8df7ceea65a" (UID: "0fd139db-ba36-4baa-ae0f-d8df7ceea65a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:06:32.611962 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.611952    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-kube-api-access-rx642" (OuterVolumeSpecName: "kube-api-access-rx642") pod "0fd139db-ba36-4baa-ae0f-d8df7ceea65a" (UID: "0fd139db-ba36-4baa-ae0f-d8df7ceea65a"). InnerVolumeSpecName "kube-api-access-rx642". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:06:32.710174 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.710130    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-home\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:06:32.710174 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.710165    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-tmp-dir\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:06:32.710174 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.710175    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-dshm\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:06:32.710174 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.710183    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-tls-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:06:32.710474 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:32.710193    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx642\" (UniqueName: \"kubernetes.io/projected/0fd139db-ba36-4baa-ae0f-d8df7ceea65a-kube-api-access-rx642\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:06:33.135323 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:33.135284    2567 generic.go:356] "Generic (PLEG): container finished" podID="0fd139db-ba36-4baa-ae0f-d8df7ceea65a" containerID="e60f0b91310cf59154901800858dbf6922f722f110853f27792a591b3b594443" exitCode=0
Sep 12 01:06:33.135544 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:33.135334    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"
Sep 12 01:06:33.135544 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:33.135364    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw" event={"ID":"0fd139db-ba36-4baa-ae0f-d8df7ceea65a","Type":"ContainerDied","Data":"e60f0b91310cf59154901800858dbf6922f722f110853f27792a591b3b594443"}
Sep 12 01:06:33.135544 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:33.135412    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw" event={"ID":"0fd139db-ba36-4baa-ae0f-d8df7ceea65a","Type":"ContainerDied","Data":"5605491bb3f1817e71ebcdb758d07d1b971b7e96b3c78f1fee187a9964ce983c"}
Sep 12 01:06:33.135544 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:33.135432    2567 scope.go:117] "RemoveContainer" containerID="e60f0b91310cf59154901800858dbf6922f722f110853f27792a591b3b594443"
Sep 12 01:06:33.145910 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:33.145888    2567 scope.go:117] "RemoveContainer" containerID="e60f0b91310cf59154901800858dbf6922f722f110853f27792a591b3b594443"
Sep 12 01:06:33.146258 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:06:33.146234    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e60f0b91310cf59154901800858dbf6922f722f110853f27792a591b3b594443\": container with ID starting with e60f0b91310cf59154901800858dbf6922f722f110853f27792a591b3b594443 not found: ID does not exist" containerID="e60f0b91310cf59154901800858dbf6922f722f110853f27792a591b3b594443"
Sep 12 01:06:33.146327 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:33.146269    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e60f0b91310cf59154901800858dbf6922f722f110853f27792a591b3b594443"} err="failed to get container status \"e60f0b91310cf59154901800858dbf6922f722f110853f27792a591b3b594443\": rpc error: code = NotFound desc = could not find container \"e60f0b91310cf59154901800858dbf6922f722f110853f27792a591b3b594443\": container with ID starting with e60f0b91310cf59154901800858dbf6922f722f110853f27792a591b3b594443 not found: ID does not exist"
Sep 12 01:06:33.165346 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:33.165310    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"]
Sep 12 01:06:33.171876 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:33.171840    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-9a2657/ipp-route-9a2657-kserve-585b74c-2d8hw"]
Sep 12 01:06:34.273262 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:06:34.273229    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fd139db-ba36-4baa-ae0f-d8df7ceea65a" path="/var/lib/kubelet/pods/0fd139db-ba36-4baa-ae0f-d8df7ceea65a/volumes"
Sep 12 01:07:00.962410 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962315    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l"]
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962693    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ce9b50-1da2-4df5-96a0-8ba13cf38088" containerName="payload-pre-processing"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962707    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ce9b50-1da2-4df5-96a0-8ba13cf38088" containerName="payload-pre-processing"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962734    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerName="istio-proxy"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962739    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerName="istio-proxy"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962745    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3f1364-3325-49dd-b268-9d37d0cf2d2c" containerName="revoke-keys"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962751    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3f1364-3325-49dd-b268-9d37d0cf2d2c" containerName="revoke-keys"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962758    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerName="istio-proxy"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962763    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerName="istio-proxy"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962773    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fd139db-ba36-4baa-ae0f-d8df7ceea65a" containerName="main"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962778    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fd139db-ba36-4baa-ae0f-d8df7ceea65a" containerName="main"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962832    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="51ce9b50-1da2-4df5-96a0-8ba13cf38088" containerName="payload-pre-processing"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962843    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerName="istio-proxy"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962849    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3f1364-3325-49dd-b268-9d37d0cf2d2c" containerName="revoke-keys"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962856    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5b8f09-284b-4c99-ac99-ed7c674ab28a" containerName="istio-proxy"
Sep 12 01:07:00.962892 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.962862    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fd139db-ba36-4baa-ae0f-d8df7ceea65a" containerName="main"
Sep 12 01:07:00.965057 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.965038    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l"
Sep 12 01:07:00.967714 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.967693    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 01:07:00.967714 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.967692    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:07:00.972920 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:00.972896    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l"]
Sep 12 01:07:01.074258 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:01.074215    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b89ba-b4f4-4386-ae17-f7a2376d7a2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l\" (UID: \"675b89ba-b4f4-4386-ae17-f7a2376d7a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l"
Sep 12 01:07:01.175015 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:01.174976    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b89ba-b4f4-4386-ae17-f7a2376d7a2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l\" (UID: \"675b89ba-b4f4-4386-ae17-f7a2376d7a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l"
Sep 12 01:07:01.175637 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:01.175615    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b89ba-b4f4-4386-ae17-f7a2376d7a2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l\" (UID: \"675b89ba-b4f4-4386-ae17-f7a2376d7a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l"
Sep 12 01:07:01.277131 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:01.277090    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l"
Sep 12 01:07:01.548348 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:01.548313    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l"]
Sep 12 01:07:01.549680 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:07:01.549648    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675b89ba_b4f4_4386_ae17_f7a2376d7a2f.slice/crio-85dba7900b786f308652b1a3971dda6ad0384c7b079c9ce133ed2e4926dd0af3 WatchSource:0}: Error finding container 85dba7900b786f308652b1a3971dda6ad0384c7b079c9ce133ed2e4926dd0af3: Status 404 returned error can't find the container with id 85dba7900b786f308652b1a3971dda6ad0384c7b079c9ce133ed2e4926dd0af3
Sep 12 01:07:02.277897 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:02.277855    2567 generic.go:356] "Generic (PLEG): container finished" podID="675b89ba-b4f4-4386-ae17-f7a2376d7a2f" containerID="684a4901674b5c89e04ef1dcb6990b331ae516c75436e96e91eccfdd9ce5c9a4" exitCode=0
Sep 12 01:07:02.278350 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:02.277932    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l" event={"ID":"675b89ba-b4f4-4386-ae17-f7a2376d7a2f","Type":"ContainerDied","Data":"684a4901674b5c89e04ef1dcb6990b331ae516c75436e96e91eccfdd9ce5c9a4"}
Sep 12 01:07:02.278350 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:02.277991    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l" event={"ID":"675b89ba-b4f4-4386-ae17-f7a2376d7a2f","Type":"ContainerStarted","Data":"85dba7900b786f308652b1a3971dda6ad0384c7b079c9ce133ed2e4926dd0af3"}
Sep 12 01:07:03.411548 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:03.411514    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l"
Sep 12 01:07:03.499165 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:03.499122    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b89ba-b4f4-4386-ae17-f7a2376d7a2f-maas-api-service-ca\") pod \"675b89ba-b4f4-4386-ae17-f7a2376d7a2f\" (UID: \"675b89ba-b4f4-4386-ae17-f7a2376d7a2f\") "
Sep 12 01:07:03.499547 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:03.499526    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675b89ba-b4f4-4386-ae17-f7a2376d7a2f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "675b89ba-b4f4-4386-ae17-f7a2376d7a2f" (UID: "675b89ba-b4f4-4386-ae17-f7a2376d7a2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:07:03.600081 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:03.599988    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b89ba-b4f4-4386-ae17-f7a2376d7a2f-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:04.289029 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:04.289000    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l"
Sep 12 01:07:04.289170 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:04.289031    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l" event={"ID":"675b89ba-b4f4-4386-ae17-f7a2376d7a2f","Type":"ContainerDied","Data":"85dba7900b786f308652b1a3971dda6ad0384c7b079c9ce133ed2e4926dd0af3"}
Sep 12 01:07:04.289170 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:04.289078    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85dba7900b786f308652b1a3971dda6ad0384c7b079c9ce133ed2e4926dd0af3"
Sep 12 01:07:11.543843 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:11.543804    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"]
Sep 12 01:07:11.544335 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:11.544061    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g" podUID="8f9b2b42-e02e-41b5-8937-731d81469b5f" containerName="payload-pre-processing" containerID="cri-o://ad022d21945d998b886c9c89abf2ea99653c93080d67fa08813d3624180d155c" gracePeriod=30
Sep 12 01:07:11.705439 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:11.705399    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l"]
Sep 12 01:07:11.712330 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:11.712291    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5b711c-zlr7l"]
Sep 12 01:07:11.803508 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:11.803432    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"
Sep 12 01:07:11.981216 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:11.981180    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9b2b42-e02e-41b5-8937-731d81469b5f-plugins-config-volume\") pod \"8f9b2b42-e02e-41b5-8937-731d81469b5f\" (UID: \"8f9b2b42-e02e-41b5-8937-731d81469b5f\") "
Sep 12 01:07:11.981384 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:11.981281    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwwhg\" (UniqueName: \"kubernetes.io/projected/8f9b2b42-e02e-41b5-8937-731d81469b5f-kube-api-access-dwwhg\") pod \"8f9b2b42-e02e-41b5-8937-731d81469b5f\" (UID: \"8f9b2b42-e02e-41b5-8937-731d81469b5f\") "
Sep 12 01:07:11.981585 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:11.981539    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9b2b42-e02e-41b5-8937-731d81469b5f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f9b2b42-e02e-41b5-8937-731d81469b5f" (UID: "8f9b2b42-e02e-41b5-8937-731d81469b5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:07:11.983446 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:11.983421    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9b2b42-e02e-41b5-8937-731d81469b5f-kube-api-access-dwwhg" (OuterVolumeSpecName: "kube-api-access-dwwhg") pod "8f9b2b42-e02e-41b5-8937-731d81469b5f" (UID: "8f9b2b42-e02e-41b5-8937-731d81469b5f"). InnerVolumeSpecName "kube-api-access-dwwhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:07:12.082996 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:12.082904    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwwhg\" (UniqueName: \"kubernetes.io/projected/8f9b2b42-e02e-41b5-8937-731d81469b5f-kube-api-access-dwwhg\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:12.082996 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:12.082935    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9b2b42-e02e-41b5-8937-731d81469b5f-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:12.274165 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:12.274120    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="675b89ba-b4f4-4386-ae17-f7a2376d7a2f" path="/var/lib/kubelet/pods/675b89ba-b4f4-4386-ae17-f7a2376d7a2f/volumes"
Sep 12 01:07:12.333729 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:12.333645    2567 generic.go:356] "Generic (PLEG): container finished" podID="8f9b2b42-e02e-41b5-8937-731d81469b5f" containerID="ad022d21945d998b886c9c89abf2ea99653c93080d67fa08813d3624180d155c" exitCode=0
Sep 12 01:07:12.333729 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:12.333697    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"
Sep 12 01:07:12.333903 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:12.333725    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g" event={"ID":"8f9b2b42-e02e-41b5-8937-731d81469b5f","Type":"ContainerDied","Data":"ad022d21945d998b886c9c89abf2ea99653c93080d67fa08813d3624180d155c"}
Sep 12 01:07:12.333903 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:12.333782    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g" event={"ID":"8f9b2b42-e02e-41b5-8937-731d81469b5f","Type":"ContainerDied","Data":"35084818ab603a92eb74ef5da7e5c2c2cb51b84756b2d34da083bc3fe52f81ea"}
Sep 12 01:07:12.333903 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:12.333805    2567 scope.go:117] "RemoveContainer" containerID="ad022d21945d998b886c9c89abf2ea99653c93080d67fa08813d3624180d155c"
Sep 12 01:07:12.342304 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:12.342285    2567 scope.go:117] "RemoveContainer" containerID="ad022d21945d998b886c9c89abf2ea99653c93080d67fa08813d3624180d155c"
Sep 12 01:07:12.342590 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:07:12.342555    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad022d21945d998b886c9c89abf2ea99653c93080d67fa08813d3624180d155c\": container with ID starting with ad022d21945d998b886c9c89abf2ea99653c93080d67fa08813d3624180d155c not found: ID does not exist" containerID="ad022d21945d998b886c9c89abf2ea99653c93080d67fa08813d3624180d155c"
Sep 12 01:07:12.342700 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:12.342595    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad022d21945d998b886c9c89abf2ea99653c93080d67fa08813d3624180d155c"} err="failed to get container status \"ad022d21945d998b886c9c89abf2ea99653c93080d67fa08813d3624180d155c\": rpc error: code = NotFound desc = could not find container \"ad022d21945d998b886c9c89abf2ea99653c93080d67fa08813d3624180d155c\": container with ID starting with ad022d21945d998b886c9c89abf2ea99653c93080d67fa08813d3624180d155c not found: ID does not exist"
Sep 12 01:07:12.349653 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:12.349629    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"]
Sep 12 01:07:12.353394 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:12.353372    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5b711c-85865587bb-5f94g"]
Sep 12 01:07:14.273635 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:14.273600    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f9b2b42-e02e-41b5-8937-731d81469b5f" path="/var/lib/kubelet/pods/8f9b2b42-e02e-41b5-8937-731d81469b5f/volumes"
Sep 12 01:07:19.306216 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.306181    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"]
Sep 12 01:07:19.306707 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.306397    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4" podUID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerName="istio-proxy" containerID="cri-o://875bf81c1435c9e8e88423bc3fbcbf09d0220b79bf5863eb94acf4fed1234001" gracePeriod=30
Sep 12 01:07:19.700195 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.700107    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m"]
Sep 12 01:07:19.700484 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.700470    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9b2b42-e02e-41b5-8937-731d81469b5f" containerName="payload-pre-processing"
Sep 12 01:07:19.700529 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.700486    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9b2b42-e02e-41b5-8937-731d81469b5f" containerName="payload-pre-processing"
Sep 12 01:07:19.700529 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.700496    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675b89ba-b4f4-4386-ae17-f7a2376d7a2f" containerName="revoke-keys"
Sep 12 01:07:19.700529 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.700502    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="675b89ba-b4f4-4386-ae17-f7a2376d7a2f" containerName="revoke-keys"
Sep 12 01:07:19.700651 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.700559    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9b2b42-e02e-41b5-8937-731d81469b5f" containerName="payload-pre-processing"
Sep 12 01:07:19.700651 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.700583    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="675b89ba-b4f4-4386-ae17-f7a2376d7a2f" containerName="revoke-keys"
Sep 12 01:07:19.703402 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.703382    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m"
Sep 12 01:07:19.710807 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.710783    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 01:07:19.711098 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.711080    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:07:19.717828 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.717800    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m"]
Sep 12 01:07:19.752031 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.751991    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8299746c-c29c-41ba-9117-9359f49789a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m\" (UID: \"8299746c-c29c-41ba-9117-9359f49789a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m"
Sep 12 01:07:19.853186 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.853141    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8299746c-c29c-41ba-9117-9359f49789a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m\" (UID: \"8299746c-c29c-41ba-9117-9359f49789a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m"
Sep 12 01:07:19.853835 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:19.853815    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8299746c-c29c-41ba-9117-9359f49789a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m\" (UID: \"8299746c-c29c-41ba-9117-9359f49789a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m"
Sep 12 01:07:20.013419 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:20.013351    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m"
Sep 12 01:07:20.292519 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:20.292492    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m"]
Sep 12 01:07:20.295147 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:07:20.295115    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8299746c_c29c_41ba_9117_9359f49789a2.slice/crio-d2e6cadc2ce937aa8b6b8ceef2bf307f56a62b1102e5d87d9b711e34d5b580bc WatchSource:0}: Error finding container d2e6cadc2ce937aa8b6b8ceef2bf307f56a62b1102e5d87d9b711e34d5b580bc: Status 404 returned error can't find the container with id d2e6cadc2ce937aa8b6b8ceef2bf307f56a62b1102e5d87d9b711e34d5b580bc
Sep 12 01:07:20.372666 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:20.372626    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m" event={"ID":"8299746c-c29c-41ba-9117-9359f49789a2","Type":"ContainerStarted","Data":"d2e6cadc2ce937aa8b6b8ceef2bf307f56a62b1102e5d87d9b711e34d5b580bc"}
Sep 12 01:07:21.379337 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:21.379298    2567 generic.go:356] "Generic (PLEG): container finished" podID="8299746c-c29c-41ba-9117-9359f49789a2" containerID="c11cccd84f12ff1a40ceaf12c3bc6920b51eb5e15cb409595f9b9f20365bda55" exitCode=0
Sep 12 01:07:21.379768 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:21.379387    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m" event={"ID":"8299746c-c29c-41ba-9117-9359f49789a2","Type":"ContainerDied","Data":"c11cccd84f12ff1a40ceaf12c3bc6920b51eb5e15cb409595f9b9f20365bda55"}
Sep 12 01:07:22.516084 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:22.516056    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m"
Sep 12 01:07:22.579298 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:22.579269    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8299746c-c29c-41ba-9117-9359f49789a2-maas-api-service-ca\") pod \"8299746c-c29c-41ba-9117-9359f49789a2\" (UID: \"8299746c-c29c-41ba-9117-9359f49789a2\") "
Sep 12 01:07:22.579638 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:22.579615    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8299746c-c29c-41ba-9117-9359f49789a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8299746c-c29c-41ba-9117-9359f49789a2" (UID: "8299746c-c29c-41ba-9117-9359f49789a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:07:22.680061 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:22.679962    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8299746c-c29c-41ba-9117-9359f49789a2-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:23.390258 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:23.390224    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m" event={"ID":"8299746c-c29c-41ba-9117-9359f49789a2","Type":"ContainerDied","Data":"d2e6cadc2ce937aa8b6b8ceef2bf307f56a62b1102e5d87d9b711e34d5b580bc"}
Sep 12 01:07:23.390258 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:23.390244    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m"
Sep 12 01:07:23.390451 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:23.390272    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2e6cadc2ce937aa8b6b8ceef2bf307f56a62b1102e5d87d9b711e34d5b580bc"
Sep 12 01:07:24.547833 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.547802    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:07:24.598379 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.598284    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-workload-certs\") pod \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") "
Sep 12 01:07:24.598565 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.598391    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-data\") pod \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") "
Sep 12 01:07:24.598565 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.598450    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f058bf56-36ef-49d4-a180-6a47e3d4bdd8" (UID: "f058bf56-36ef-49d4-a180-6a47e3d4bdd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:07:24.598565 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.598455    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-credential-socket\") pod \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") "
Sep 12 01:07:24.598565 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.598515    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istiod-ca-cert\") pod \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") "
Sep 12 01:07:24.598565 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.598554    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-workload-socket\") pod \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") "
Sep 12 01:07:24.598839 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.598585    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-token\") pod \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") "
Sep 12 01:07:24.598839 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.598720    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-podinfo\") pod \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") "
Sep 12 01:07:24.598839 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.598709    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-data" (OuterVolumeSpecName: "istio-data") pod "f058bf56-36ef-49d4-a180-6a47e3d4bdd8" (UID: "f058bf56-36ef-49d4-a180-6a47e3d4bdd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:07:24.598839 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.598783    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-envoy\") pod \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") "
Sep 12 01:07:24.598839 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.598781    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f058bf56-36ef-49d4-a180-6a47e3d4bdd8" (UID: "f058bf56-36ef-49d4-a180-6a47e3d4bdd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:07:24.598839 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.598809    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfkrf\" (UniqueName: \"kubernetes.io/projected/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-kube-api-access-lfkrf\") pod \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\" (UID: \"f058bf56-36ef-49d4-a180-6a47e3d4bdd8\") "
Sep 12 01:07:24.599135 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.598974    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f058bf56-36ef-49d4-a180-6a47e3d4bdd8" (UID: "f058bf56-36ef-49d4-a180-6a47e3d4bdd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:07:24.599194 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.599163    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-workload-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:24.599194 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.599184    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-data\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:24.599319 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.599196    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-credential-socket\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:24.599319 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.599209    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istiod-ca-cert\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:24.599319 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.599235    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f058bf56-36ef-49d4-a180-6a47e3d4bdd8" (UID: "f058bf56-36ef-49d4-a180-6a47e3d4bdd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:07:24.601011 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.600982    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-token" (OuterVolumeSpecName: "istio-token") pod "f058bf56-36ef-49d4-a180-6a47e3d4bdd8" (UID: "f058bf56-36ef-49d4-a180-6a47e3d4bdd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:07:24.601126 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.601002    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f058bf56-36ef-49d4-a180-6a47e3d4bdd8" (UID: "f058bf56-36ef-49d4-a180-6a47e3d4bdd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 01:07:24.601126 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.601067    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-kube-api-access-lfkrf" (OuterVolumeSpecName: "kube-api-access-lfkrf") pod "f058bf56-36ef-49d4-a180-6a47e3d4bdd8" (UID: "f058bf56-36ef-49d4-a180-6a47e3d4bdd8"). InnerVolumeSpecName "kube-api-access-lfkrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:07:24.601290 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.601275    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f058bf56-36ef-49d4-a180-6a47e3d4bdd8" (UID: "f058bf56-36ef-49d4-a180-6a47e3d4bdd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:07:24.700082 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.700037    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-envoy\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:24.700082 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.700078    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfkrf\" (UniqueName: \"kubernetes.io/projected/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-kube-api-access-lfkrf\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:24.700082 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.700093    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-workload-socket\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:24.700339 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.700103    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-token\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:24.700339 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:24.700111    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f058bf56-36ef-49d4-a180-6a47e3d4bdd8-istio-podinfo\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:25.409989 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:25.409922    2567 generic.go:356] "Generic (PLEG): container finished" podID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerID="875bf81c1435c9e8e88423bc3fbcbf09d0220b79bf5863eb94acf4fed1234001" exitCode=0
Sep 12 01:07:25.410207 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:25.410081    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"
Sep 12 01:07:25.410207 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:25.410080    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4" event={"ID":"f058bf56-36ef-49d4-a180-6a47e3d4bdd8","Type":"ContainerDied","Data":"875bf81c1435c9e8e88423bc3fbcbf09d0220b79bf5863eb94acf4fed1234001"}
Sep 12 01:07:25.410207 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:25.410186    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4" event={"ID":"f058bf56-36ef-49d4-a180-6a47e3d4bdd8","Type":"ContainerDied","Data":"f897c09e275f95fd8813e93215e65c3fa38f0c958ff698f0b0cdba964e8c4bcf"}
Sep 12 01:07:25.410373 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:25.410220    2567 scope.go:117] "RemoveContainer" containerID="875bf81c1435c9e8e88423bc3fbcbf09d0220b79bf5863eb94acf4fed1234001"
Sep 12 01:07:25.419514 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:25.419304    2567 scope.go:117] "RemoveContainer" containerID="cdcee6b539698c9c2ae8556500b18f910f08f7649e61e0fe8ae20429e8b798d1"
Sep 12 01:07:25.427077 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:25.427057    2567 scope.go:117] "RemoveContainer" containerID="875bf81c1435c9e8e88423bc3fbcbf09d0220b79bf5863eb94acf4fed1234001"
Sep 12 01:07:25.427314 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:07:25.427290    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875bf81c1435c9e8e88423bc3fbcbf09d0220b79bf5863eb94acf4fed1234001\": container with ID starting with 875bf81c1435c9e8e88423bc3fbcbf09d0220b79bf5863eb94acf4fed1234001 not found: ID does not exist" containerID="875bf81c1435c9e8e88423bc3fbcbf09d0220b79bf5863eb94acf4fed1234001"
Sep 12 01:07:25.427383 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:25.427321    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875bf81c1435c9e8e88423bc3fbcbf09d0220b79bf5863eb94acf4fed1234001"} err="failed to get container status \"875bf81c1435c9e8e88423bc3fbcbf09d0220b79bf5863eb94acf4fed1234001\": rpc error: code = NotFound desc = could not find container \"875bf81c1435c9e8e88423bc3fbcbf09d0220b79bf5863eb94acf4fed1234001\": container with ID starting with 875bf81c1435c9e8e88423bc3fbcbf09d0220b79bf5863eb94acf4fed1234001 not found: ID does not exist"
Sep 12 01:07:25.427383 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:25.427339    2567 scope.go:117] "RemoveContainer" containerID="cdcee6b539698c9c2ae8556500b18f910f08f7649e61e0fe8ae20429e8b798d1"
Sep 12 01:07:25.427618 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:07:25.427589    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdcee6b539698c9c2ae8556500b18f910f08f7649e61e0fe8ae20429e8b798d1\": container with ID starting with cdcee6b539698c9c2ae8556500b18f910f08f7649e61e0fe8ae20429e8b798d1 not found: ID does not exist" containerID="cdcee6b539698c9c2ae8556500b18f910f08f7649e61e0fe8ae20429e8b798d1"
Sep 12 01:07:25.427686 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:25.427626    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdcee6b539698c9c2ae8556500b18f910f08f7649e61e0fe8ae20429e8b798d1"} err="failed to get container status \"cdcee6b539698c9c2ae8556500b18f910f08f7649e61e0fe8ae20429e8b798d1\": rpc error: code = NotFound desc = could not find container \"cdcee6b539698c9c2ae8556500b18f910f08f7649e61e0fe8ae20429e8b798d1\": container with ID starting with cdcee6b539698c9c2ae8556500b18f910f08f7649e61e0fe8ae20429e8b798d1 not found: ID does not exist"
Sep 12 01:07:25.437314 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:25.437291    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"]
Sep 12 01:07:25.438958 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:25.438936    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5b711c-openshift-default-76b46dfb86-qq7g4"]
Sep 12 01:07:26.273769 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:26.273735    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" path="/var/lib/kubelet/pods/f058bf56-36ef-49d4-a180-6a47e3d4bdd8/volumes"
Sep 12 01:07:30.216128 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:30.216092    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks"]
Sep 12 01:07:30.220794 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:30.220761    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9511bd-29819580-nv6ks"]
Sep 12 01:07:30.275167 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:30.275124    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6666a48d-595f-4b08-bf45-7ed70c9f0a05" path="/var/lib/kubelet/pods/6666a48d-595f-4b08-bf45-7ed70c9f0a05/volumes"
Sep 12 01:07:30.406729 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:30.406678    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m"]
Sep 12 01:07:30.408607 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:30.408562    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9511bd-slm4m"]
Sep 12 01:07:32.273418 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:32.273386    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8299746c-c29c-41ba-9117-9359f49789a2" path="/var/lib/kubelet/pods/8299746c-c29c-41ba-9117-9359f49789a2/volumes"
Sep 12 01:07:37.623451 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.623415    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8"]
Sep 12 01:07:37.623901 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.623768    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerName="istio-proxy"
Sep 12 01:07:37.623901 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.623781    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerName="istio-proxy"
Sep 12 01:07:37.623901 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.623793    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerName="istio-proxy"
Sep 12 01:07:37.623901 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.623799    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerName="istio-proxy"
Sep 12 01:07:37.623901 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.623811    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8299746c-c29c-41ba-9117-9359f49789a2" containerName="revoke-keys"
Sep 12 01:07:37.623901 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.623817    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8299746c-c29c-41ba-9117-9359f49789a2" containerName="revoke-keys"
Sep 12 01:07:37.623901 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.623866    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8299746c-c29c-41ba-9117-9359f49789a2" containerName="revoke-keys"
Sep 12 01:07:37.623901 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.623874    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerName="istio-proxy"
Sep 12 01:07:37.623901 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.623880    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f058bf56-36ef-49d4-a180-6a47e3d4bdd8" containerName="istio-proxy"
Sep 12 01:07:37.626774 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.626757    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8"
Sep 12 01:07:37.629252 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.629231    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sj5jt\""
Sep 12 01:07:37.629252 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.629232    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:07:37.634998 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.634956    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8"]
Sep 12 01:07:37.712544 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.712498    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212827b4-21af-4cd0-bebc-82b3b668f4dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8\" (UID: \"212827b4-21af-4cd0-bebc-82b3b668f4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8"
Sep 12 01:07:37.813942 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.813899    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212827b4-21af-4cd0-bebc-82b3b668f4dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8\" (UID: \"212827b4-21af-4cd0-bebc-82b3b668f4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8"
Sep 12 01:07:37.814557 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.814533    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212827b4-21af-4cd0-bebc-82b3b668f4dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8\" (UID: \"212827b4-21af-4cd0-bebc-82b3b668f4dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8"
Sep 12 01:07:37.939405 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:37.939296    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8"
Sep 12 01:07:38.212396 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:38.212315    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8"]
Sep 12 01:07:38.215961 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:07:38.215932    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212827b4_21af_4cd0_bebc_82b3b668f4dc.slice/crio-b5ff938f9f824bc899cdbc3b4efb39564ac375882a8dcf72fa7d8fc5b28b2179 WatchSource:0}: Error finding container b5ff938f9f824bc899cdbc3b4efb39564ac375882a8dcf72fa7d8fc5b28b2179: Status 404 returned error can't find the container with id b5ff938f9f824bc899cdbc3b4efb39564ac375882a8dcf72fa7d8fc5b28b2179
Sep 12 01:07:38.476154 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:38.476119    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8" event={"ID":"212827b4-21af-4cd0-bebc-82b3b668f4dc","Type":"ContainerStarted","Data":"b5ff938f9f824bc899cdbc3b4efb39564ac375882a8dcf72fa7d8fc5b28b2179"}
Sep 12 01:07:39.482503 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:39.482470    2567 generic.go:356] "Generic (PLEG): container finished" podID="212827b4-21af-4cd0-bebc-82b3b668f4dc" containerID="088117839e8cca7bcb4f1b46118cfb0abdab0c9bca6e2b0c47b74c7347f6e21f" exitCode=0
Sep 12 01:07:39.482991 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:39.482561    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8" event={"ID":"212827b4-21af-4cd0-bebc-82b3b668f4dc","Type":"ContainerDied","Data":"088117839e8cca7bcb4f1b46118cfb0abdab0c9bca6e2b0c47b74c7347f6e21f"}
Sep 12 01:07:40.620601 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:40.620559    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8"
Sep 12 01:07:40.742483 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:40.742377    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212827b4-21af-4cd0-bebc-82b3b668f4dc-maas-api-service-ca\") pod \"212827b4-21af-4cd0-bebc-82b3b668f4dc\" (UID: \"212827b4-21af-4cd0-bebc-82b3b668f4dc\") "
Sep 12 01:07:40.742778 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:40.742756    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212827b4-21af-4cd0-bebc-82b3b668f4dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "212827b4-21af-4cd0-bebc-82b3b668f4dc" (UID: "212827b4-21af-4cd0-bebc-82b3b668f4dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:07:40.843544 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:40.843503    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212827b4-21af-4cd0-bebc-82b3b668f4dc-maas-api-service-ca\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:41.499738 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:41.499707    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8"
Sep 12 01:07:41.499922 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:41.499746    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8" event={"ID":"212827b4-21af-4cd0-bebc-82b3b668f4dc","Type":"ContainerDied","Data":"b5ff938f9f824bc899cdbc3b4efb39564ac375882a8dcf72fa7d8fc5b28b2179"}
Sep 12 01:07:41.499922 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:41.499818    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5ff938f9f824bc899cdbc3b4efb39564ac375882a8dcf72fa7d8fc5b28b2179"
Sep 12 01:07:48.217138 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.217102    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"]
Sep 12 01:07:48.217972 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.217931    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4" podUID="6d63161a-12bb-4a88-8a2a-840a4cad8eb3" containerName="payload-processing" containerID="cri-o://975b76c35043f9c889d09efef9af2494a75a6978a64a306e49ba0d58d11c87ef" gracePeriod=30
Sep 12 01:07:48.386245 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.386198    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8"]
Sep 12 01:07:48.388769 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.388739    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd25dd-wkcc8"]
Sep 12 01:07:48.472455 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.472432    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"
Sep 12 01:07:48.533840 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.533803    2567 generic.go:356] "Generic (PLEG): container finished" podID="6d63161a-12bb-4a88-8a2a-840a4cad8eb3" containerID="975b76c35043f9c889d09efef9af2494a75a6978a64a306e49ba0d58d11c87ef" exitCode=0
Sep 12 01:07:48.534016 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.533849    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"
Sep 12 01:07:48.534016 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.533888    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4" event={"ID":"6d63161a-12bb-4a88-8a2a-840a4cad8eb3","Type":"ContainerDied","Data":"975b76c35043f9c889d09efef9af2494a75a6978a64a306e49ba0d58d11c87ef"}
Sep 12 01:07:48.534016 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.533941    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4" event={"ID":"6d63161a-12bb-4a88-8a2a-840a4cad8eb3","Type":"ContainerDied","Data":"26489d81552041ea66419d48568d8bc3dab02f389235fde3d085ef3ebe7dd725"}
Sep 12 01:07:48.534016 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.533963    2567 scope.go:117] "RemoveContainer" containerID="975b76c35043f9c889d09efef9af2494a75a6978a64a306e49ba0d58d11c87ef"
Sep 12 01:07:48.542511 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.542492    2567 scope.go:117] "RemoveContainer" containerID="975b76c35043f9c889d09efef9af2494a75a6978a64a306e49ba0d58d11c87ef"
Sep 12 01:07:48.542850 ip-10-0-130-132 kubenswrapper[2567]: E0912 01:07:48.542823    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975b76c35043f9c889d09efef9af2494a75a6978a64a306e49ba0d58d11c87ef\": container with ID starting with 975b76c35043f9c889d09efef9af2494a75a6978a64a306e49ba0d58d11c87ef not found: ID does not exist" containerID="975b76c35043f9c889d09efef9af2494a75a6978a64a306e49ba0d58d11c87ef"
Sep 12 01:07:48.542934 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.542863    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975b76c35043f9c889d09efef9af2494a75a6978a64a306e49ba0d58d11c87ef"} err="failed to get container status \"975b76c35043f9c889d09efef9af2494a75a6978a64a306e49ba0d58d11c87ef\": rpc error: code = NotFound desc = could not find container \"975b76c35043f9c889d09efef9af2494a75a6978a64a306e49ba0d58d11c87ef\": container with ID starting with 975b76c35043f9c889d09efef9af2494a75a6978a64a306e49ba0d58d11c87ef not found: ID does not exist"
Sep 12 01:07:48.616898 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.616861    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h5gl\" (UniqueName: \"kubernetes.io/projected/6d63161a-12bb-4a88-8a2a-840a4cad8eb3-kube-api-access-4h5gl\") pod \"6d63161a-12bb-4a88-8a2a-840a4cad8eb3\" (UID: \"6d63161a-12bb-4a88-8a2a-840a4cad8eb3\") "
Sep 12 01:07:48.616898 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.616900    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d63161a-12bb-4a88-8a2a-840a4cad8eb3-plugins-config-volume\") pod \"6d63161a-12bb-4a88-8a2a-840a4cad8eb3\" (UID: \"6d63161a-12bb-4a88-8a2a-840a4cad8eb3\") "
Sep 12 01:07:48.617309 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.617282    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d63161a-12bb-4a88-8a2a-840a4cad8eb3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d63161a-12bb-4a88-8a2a-840a4cad8eb3" (UID: "6d63161a-12bb-4a88-8a2a-840a4cad8eb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:07:48.619117 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.619097    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d63161a-12bb-4a88-8a2a-840a4cad8eb3-kube-api-access-4h5gl" (OuterVolumeSpecName: "kube-api-access-4h5gl") pod "6d63161a-12bb-4a88-8a2a-840a4cad8eb3" (UID: "6d63161a-12bb-4a88-8a2a-840a4cad8eb3"). InnerVolumeSpecName "kube-api-access-4h5gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:07:48.717716 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.717679    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h5gl\" (UniqueName: \"kubernetes.io/projected/6d63161a-12bb-4a88-8a2a-840a4cad8eb3-kube-api-access-4h5gl\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:48.717716 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.717710    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d63161a-12bb-4a88-8a2a-840a4cad8eb3-plugins-config-volume\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:07:48.857310 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.857278    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"]
Sep 12 01:07:48.861470 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:48.861441    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-dd25dd-57b899966c-t8hq4"]
Sep 12 01:07:50.273480 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:50.273444    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="212827b4-21af-4cd0-bebc-82b3b668f4dc" path="/var/lib/kubelet/pods/212827b4-21af-4cd0-bebc-82b3b668f4dc/volumes"
Sep 12 01:07:50.273882 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:07:50.273821    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d63161a-12bb-4a88-8a2a-840a4cad8eb3" path="/var/lib/kubelet/pods/6d63161a-12bb-4a88-8a2a-840a4cad8eb3/volumes"
Sep 12 01:08:29.223896 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:08:29.223861    2567 scope.go:117] "RemoveContainer" containerID="e27755ed9f9bd3a135cc15166e79cc87d652ca7a1bf266b53eea18c9f929d739"
Sep 12 01:12:26.422194 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.422152    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"]
Sep 12 01:12:26.422775 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.422648    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d63161a-12bb-4a88-8a2a-840a4cad8eb3" containerName="payload-processing"
Sep 12 01:12:26.422775 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.422669    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d63161a-12bb-4a88-8a2a-840a4cad8eb3" containerName="payload-processing"
Sep 12 01:12:26.422775 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.422702    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="212827b4-21af-4cd0-bebc-82b3b668f4dc" containerName="revoke-keys"
Sep 12 01:12:26.422775 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.422711    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="212827b4-21af-4cd0-bebc-82b3b668f4dc" containerName="revoke-keys"
Sep 12 01:12:26.423074 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.422792    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="212827b4-21af-4cd0-bebc-82b3b668f4dc" containerName="revoke-keys"
Sep 12 01:12:26.423074 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.422808    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d63161a-12bb-4a88-8a2a-840a4cad8eb3" containerName="payload-processing"
Sep 12 01:12:26.425792 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.425768    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.428528 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.428501    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-452393ea-kserve-self-signed-certs\""
Sep 12 01:12:26.436365 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.436338    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"]
Sep 12 01:12:26.488000 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.487959    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-model-cache\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.488000 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.488000    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-tls-certs\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.488221 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.488035    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-dshm\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.488221 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.488100    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljh7c\" (UniqueName: \"kubernetes.io/projected/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-kube-api-access-ljh7c\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.488221 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.488143    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-tmp-dir\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.488341 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.488220    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-home\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.589404 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.589365    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-model-cache\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.589404 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.589410    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-tls-certs\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.589740 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.589459    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-dshm\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.589740 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.589507    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljh7c\" (UniqueName: \"kubernetes.io/projected/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-kube-api-access-ljh7c\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.589740 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.589541    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-tmp-dir\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.589910 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.589871    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-model-cache\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.589973 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.589914    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-tmp-dir\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.590025 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.590005    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-home\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.590305 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.590280    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-home\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.592141 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.592115    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-dshm\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.592274 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.592134    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-tls-certs\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.599175 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.599148    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljh7c\" (UniqueName: \"kubernetes.io/projected/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-kube-api-access-ljh7c\") pod \"e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") " pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:26.735746 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:26.735706    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:27.009974 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:27.009945    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"]
Sep 12 01:12:27.012432 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:12:27.012394    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod398d43c1_2a8c_46a2_8fa4_a079e772a5c6.slice/crio-7063583a4d96b48597e19e16134e4b3a0951e1bdfab88461107289add2912fa9 WatchSource:0}: Error finding container 7063583a4d96b48597e19e16134e4b3a0951e1bdfab88461107289add2912fa9: Status 404 returned error can't find the container with id 7063583a4d96b48597e19e16134e4b3a0951e1bdfab88461107289add2912fa9
Sep 12 01:12:27.863748 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:27.863699    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx" event={"ID":"398d43c1-2a8c-46a2-8fa4-a079e772a5c6","Type":"ContainerStarted","Data":"e2464314783b82b6d97023d0732f902b207d249e38bcc33ee9a81de3a98020a7"}
Sep 12 01:12:27.863748 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:27.863749    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx" event={"ID":"398d43c1-2a8c-46a2-8fa4-a079e772a5c6","Type":"ContainerStarted","Data":"7063583a4d96b48597e19e16134e4b3a0951e1bdfab88461107289add2912fa9"}
Sep 12 01:12:27.883096 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:27.883041    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx" podStartSLOduration=1.883015671 podStartE2EDuration="1.883015671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:12:27.880400875 +0000 UTC m=+2580.084401698" watchObservedRunningTime="2026-09-12 01:12:27.883015671 +0000 UTC m=+2580.087016515"
Sep 12 01:12:29.269503 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:29.269470    2567 scope.go:117] "RemoveContainer" containerID="89ff766a9049193341528b489e91ec1924426c06eed1a1282333abc3efce40c0"
Sep 12 01:12:29.279049 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:29.279016    2567 scope.go:117] "RemoveContainer" containerID="1371fdbb4e075c13b731b36e504f1a67a88428d28919a6d1dfd245e37906bc87"
Sep 12 01:12:36.736177 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:36.736131    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:36.736856 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:36.736206    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:36.748711 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:36.748686    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:12:36.918490 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:12:36.918455    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:13:05.939415 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:05.939372    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"]
Sep 12 01:13:20.960363 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:20.960316    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx" podUID="398d43c1-2a8c-46a2-8fa4-a079e772a5c6" containerName="main" containerID="cri-o://e2464314783b82b6d97023d0732f902b207d249e38bcc33ee9a81de3a98020a7" gracePeriod=45
Sep 12 01:13:21.116252 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.116215    2567 generic.go:356] "Generic (PLEG): container finished" podID="398d43c1-2a8c-46a2-8fa4-a079e772a5c6" containerID="e2464314783b82b6d97023d0732f902b207d249e38bcc33ee9a81de3a98020a7" exitCode=0
Sep 12 01:13:21.116426 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.116256    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx" event={"ID":"398d43c1-2a8c-46a2-8fa4-a079e772a5c6","Type":"ContainerDied","Data":"e2464314783b82b6d97023d0732f902b207d249e38bcc33ee9a81de3a98020a7"}
Sep 12 01:13:21.214139 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.214071    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:13:21.300275 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.300232    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-tmp-dir\") pod \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") "
Sep 12 01:13:21.300461 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.300333    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-model-cache\") pod \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") "
Sep 12 01:13:21.300461 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.300380    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljh7c\" (UniqueName: \"kubernetes.io/projected/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-kube-api-access-ljh7c\") pod \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") "
Sep 12 01:13:21.300461 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.300417    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-home\") pod \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") "
Sep 12 01:13:21.300603 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.300472    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-dshm\") pod \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") "
Sep 12 01:13:21.300603 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.300539    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-tls-certs\") pod \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\" (UID: \"398d43c1-2a8c-46a2-8fa4-a079e772a5c6\") "
Sep 12 01:13:21.300603 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.300552    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "398d43c1-2a8c-46a2-8fa4-a079e772a5c6" (UID: "398d43c1-2a8c-46a2-8fa4-a079e772a5c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:13:21.300765 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.300667    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "398d43c1-2a8c-46a2-8fa4-a079e772a5c6" (UID: "398d43c1-2a8c-46a2-8fa4-a079e772a5c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:13:21.300765 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.300733    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-home" (OuterVolumeSpecName: "home") pod "398d43c1-2a8c-46a2-8fa4-a079e772a5c6" (UID: "398d43c1-2a8c-46a2-8fa4-a079e772a5c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:13:21.300881 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.300843    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-tmp-dir\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:13:21.300881 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.300857    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-model-cache\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:13:21.300881 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.300866    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-home\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:13:21.302813 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.302785    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-dshm" (OuterVolumeSpecName: "dshm") pod "398d43c1-2a8c-46a2-8fa4-a079e772a5c6" (UID: "398d43c1-2a8c-46a2-8fa4-a079e772a5c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:13:21.302935 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.302810    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "398d43c1-2a8c-46a2-8fa4-a079e772a5c6" (UID: "398d43c1-2a8c-46a2-8fa4-a079e772a5c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:13:21.302935 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.302911    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-kube-api-access-ljh7c" (OuterVolumeSpecName: "kube-api-access-ljh7c") pod "398d43c1-2a8c-46a2-8fa4-a079e772a5c6" (UID: "398d43c1-2a8c-46a2-8fa4-a079e772a5c6"). InnerVolumeSpecName "kube-api-access-ljh7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:13:21.402448 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.402399    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-tls-certs\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:13:21.402448 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.402442    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljh7c\" (UniqueName: \"kubernetes.io/projected/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-kube-api-access-ljh7c\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:13:21.402448 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:21.402459    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398d43c1-2a8c-46a2-8fa4-a079e772a5c6-dshm\") on node \"ip-10-0-130-132.ec2.internal\" DevicePath \"\""
Sep 12 01:13:22.122429 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:22.122388    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx" event={"ID":"398d43c1-2a8c-46a2-8fa4-a079e772a5c6","Type":"ContainerDied","Data":"7063583a4d96b48597e19e16134e4b3a0951e1bdfab88461107289add2912fa9"}
Sep 12 01:13:22.122973 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:22.122428    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"
Sep 12 01:13:22.122973 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:22.122441    2567 scope.go:117] "RemoveContainer" containerID="e2464314783b82b6d97023d0732f902b207d249e38bcc33ee9a81de3a98020a7"
Sep 12 01:13:22.144143 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:22.144108    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"]
Sep 12 01:13:22.147770 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:22.147739    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-452393ea-kserve-6b4756cb79-2vnfx"]
Sep 12 01:13:22.273924 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:22.273885    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="398d43c1-2a8c-46a2-8fa4-a079e772a5c6" path="/var/lib/kubelet/pods/398d43c1-2a8c-46a2-8fa4-a079e772a5c6/volumes"
Sep 12 01:13:29.304076 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:29.304035    2567 scope.go:117] "RemoveContainer" containerID="684a4901674b5c89e04ef1dcb6990b331ae516c75436e96e91eccfdd9ce5c9a4"
Sep 12 01:13:29.312232 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:29.312206    2567 scope.go:117] "RemoveContainer" containerID="c11cccd84f12ff1a40ceaf12c3bc6920b51eb5e15cb409595f9b9f20365bda55"
Sep 12 01:13:56.651885 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:13:56.651837    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-664f9b94b6-rxqjx_cae3b771-b860-4e3d-b2ab-7fb41fc39b4f/authorino/0.log"
Sep 12 01:14:01.665182 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:01.665141    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-2xddl_482daf06-10b6-4385-95eb-2895bce457fd/manager/0.log"
Sep 12 01:14:02.020456 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:02.020429    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-j7qcq_c1e9be4b-762a-4190-9cd4-6b4d65505de6/server/0.log"
Sep 12 01:14:02.131958 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:02.131929    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-drtcs_7ce16dde-9fbf-4b0a-8148-6ff5955bf039/manager/0.log"
Sep 12 01:14:02.362135 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:02.362054    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b7744c54-mk6xw_db0f42de-90a2-4018-be00-309da0fd271c/copy-manifests/0.log"
Sep 12 01:14:02.367665 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:02.367637    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b7744c54-mk6xw_db0f42de-90a2-4018-be00-309da0fd271c/manager/0.log"
Sep 12 01:14:03.131699 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:03.131670    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819580-x5rmz_15625d32-7a10-4b8c-80b3-b344a80855cb/cleanup/0.log"
Sep 12 01:14:04.034682 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:04.034646    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-664f9b94b6-rxqjx_cae3b771-b860-4e3d-b2ab-7fb41fc39b4f/authorino/0.log"
Sep 12 01:14:04.376312 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:04.376231    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-7v8nj_a6132da6-fa63-4f8b-a6ac-8022da8818b3/kuadrant-console-plugin/0.log"
Sep 12 01:14:04.761860 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:04.761826    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-l9g84_e5db7ffa-9634-4653-af5a-ef87004060fd/manager/0.log"
Sep 12 01:14:05.113104 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:05.113017    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw_d5daf33e-366a-458a-9006-97e0e974afbb/istio-proxy/0.log"
Sep 12 01:14:05.606738 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:05.606708    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cvpvm_e8f16670-c284-4e2e-b6b4-fe54a025c6a9/istio-proxy/1.log"
Sep 12 01:14:06.311492 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:06.311459    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-ntsrn_936ed526-5c84-4985-b69f-63bb71e7727b/main/0.log"
Sep 12 01:14:06.546499 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:06.546469    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-w66c7_30615d1d-857d-4b84-9a65-a186f63f565a/main/0.log"
Sep 12 01:14:06.867631 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:06.867602    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-mjd2z_22554bd1-0bcb-4a6d-8935-09890d1e0b47/main/0.log"
Sep 12 01:14:06.974550 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:06.974523    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-cx5ns_bdcef50e-8eff-4c04-b398-ac7219f37290/main/0.log"
Sep 12 01:14:16.270213 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:16.270181    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q9l42_4967c7dd-797f-4225-a54d-6b83c6827183/global-pull-secret-syncer/0.log"
Sep 12 01:14:16.402048 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:16.402015    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zrcf9_34228a0d-1c7e-49d0-8af7-a9ed2f5ab54e/konnectivity-agent/0.log"
Sep 12 01:14:16.442761 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:16.442734    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-132.ec2.internal_eddb3576bde467541695f2fe0d5bc466/haproxy/0.log"
Sep 12 01:14:21.390971 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:21.390873    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-664f9b94b6-rxqjx_cae3b771-b860-4e3d-b2ab-7fb41fc39b4f/authorino/0.log"
Sep 12 01:14:21.528641 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:21.528613    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-7v8nj_a6132da6-fa63-4f8b-a6ac-8022da8818b3/kuadrant-console-plugin/0.log"
Sep 12 01:14:21.685141 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:21.685055    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-l9g84_e5db7ffa-9634-4653-af5a-ef87004060fd/manager/0.log"
Sep 12 01:14:24.395305 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:24.395268    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jd297_5ba1acf3-4610-4e04-988d-c716bde57c28/node-exporter/0.log"
Sep 12 01:14:24.425318 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:24.425291    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jd297_5ba1acf3-4610-4e04-988d-c716bde57c28/kube-rbac-proxy/0.log"
Sep 12 01:14:24.478823 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:24.478798    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jd297_5ba1acf3-4610-4e04-988d-c716bde57c28/init-textfile/0.log"
Sep 12 01:14:24.958936 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:24.958907    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-cmtqp_411e7923-ac1f-4e50-8ea0-d4119750a83e/prometheus-operator-admission-webhook/0.log"
Sep 12 01:14:26.076390 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.076355    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"]
Sep 12 01:14:26.076814 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.076749    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="398d43c1-2a8c-46a2-8fa4-a079e772a5c6" containerName="main"
Sep 12 01:14:26.076814 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.076763    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="398d43c1-2a8c-46a2-8fa4-a079e772a5c6" containerName="main"
Sep 12 01:14:26.076887 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.076824    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="398d43c1-2a8c-46a2-8fa4-a079e772a5c6" containerName="main"
Sep 12 01:14:26.079490 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.079469    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.082333 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.082304    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fp67s\"/\"kube-root-ca.crt\""
Sep 12 01:14:26.082483 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.082361    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fp67s\"/\"default-dockercfg-wb842\""
Sep 12 01:14:26.083469 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.083447    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fp67s\"/\"openshift-service-ca.crt\""
Sep 12 01:14:26.098642 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.098611    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"]
Sep 12 01:14:26.181379 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.181334    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0bbcdce-d519-4367-917f-0f744a4a018b-lib-modules\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.181611 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.181422    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn24b\" (UniqueName: \"kubernetes.io/projected/e0bbcdce-d519-4367-917f-0f744a4a018b-kube-api-access-qn24b\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.181611 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.181480    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0bbcdce-d519-4367-917f-0f744a4a018b-proc\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.181611 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.181551    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0bbcdce-d519-4367-917f-0f744a4a018b-sys\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.181611 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.181584    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0bbcdce-d519-4367-917f-0f744a4a018b-podres\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.282713 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.282671    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0bbcdce-d519-4367-917f-0f744a4a018b-proc\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.282925 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.282741    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0bbcdce-d519-4367-917f-0f744a4a018b-sys\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.282925 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.282769    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0bbcdce-d519-4367-917f-0f744a4a018b-podres\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.282925 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.282800    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0bbcdce-d519-4367-917f-0f744a4a018b-proc\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.282925 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.282830    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0bbcdce-d519-4367-917f-0f744a4a018b-lib-modules\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.282925 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.282843    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0bbcdce-d519-4367-917f-0f744a4a018b-sys\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.282925 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.282869    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn24b\" (UniqueName: \"kubernetes.io/projected/e0bbcdce-d519-4367-917f-0f744a4a018b-kube-api-access-qn24b\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.282925 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.282914    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0bbcdce-d519-4367-917f-0f744a4a018b-podres\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.283288 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.282969    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0bbcdce-d519-4367-917f-0f744a4a018b-lib-modules\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.291780 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.291752    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn24b\" (UniqueName: \"kubernetes.io/projected/e0bbcdce-d519-4367-917f-0f744a4a018b-kube-api-access-qn24b\") pod \"perf-node-gather-daemonset-hjjvg\" (UID: \"e0bbcdce-d519-4367-917f-0f744a4a018b\") " pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.390000 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.389906    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:26.663037 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:26.662945    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"]
Sep 12 01:14:26.665673 ip-10-0-130-132 kubenswrapper[2567]: W0912 01:14:26.665640    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode0bbcdce_d519_4367_917f_0f744a4a018b.slice/crio-fb893269258c8d730fdef913cdf5823b40fec7538966e3a94470c033c114b222 WatchSource:0}: Error finding container fb893269258c8d730fdef913cdf5823b40fec7538966e3a94470c033c114b222: Status 404 returned error can't find the container with id fb893269258c8d730fdef913cdf5823b40fec7538966e3a94470c033c114b222
Sep 12 01:14:27.413124 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:27.413081    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg" event={"ID":"e0bbcdce-d519-4367-917f-0f744a4a018b","Type":"ContainerStarted","Data":"7fb7d0d35a1934117ff8dc74f72f6ab48f806ba8a287a4254e530a63d9f1278d"}
Sep 12 01:14:27.413613 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:27.413144    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg" event={"ID":"e0bbcdce-d519-4367-917f-0f744a4a018b","Type":"ContainerStarted","Data":"fb893269258c8d730fdef913cdf5823b40fec7538966e3a94470c033c114b222"}
Sep 12 01:14:27.413613 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:27.413276    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:27.442650 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:27.442593    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg" podStartSLOduration=1.442557008 podStartE2EDuration="1.442557008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:14:27.441427358 +0000 UTC m=+2699.645428193" watchObservedRunningTime="2026-09-12 01:14:27.442557008 +0000 UTC m=+2699.646557829"
Sep 12 01:14:29.337003 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:29.336968    2567 scope.go:117] "RemoveContainer" containerID="088117839e8cca7bcb4f1b46118cfb0abdab0c9bca6e2b0c47b74c7347f6e21f"
Sep 12 01:14:30.041043 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:30.041010    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p94bw_5c884033-b8a0-4f39-a5b7-d5006599e229/dns/0.log"
Sep 12 01:14:30.064247 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:30.064220    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p94bw_5c884033-b8a0-4f39-a5b7-d5006599e229/kube-rbac-proxy/0.log"
Sep 12 01:14:30.116805 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:30.116778    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h688x_661a55c6-221b-4cb8-b50b-ff96524e9ac8/dns-node-resolver/0.log"
Sep 12 01:14:30.819763 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:30.819731    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7db65b44b-pjgst_4bb66d73-4366-4eb7-a858-c353d4d4dcb9/registry/0.log"
Sep 12 01:14:30.843968 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:30.843939    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7pvhz_0e01af40-0cf2-43d6-83c5-12215a4c0a4f/node-ca/0.log"
Sep 12 01:14:31.874318 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:31.874283    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw_d5daf33e-366a-458a-9006-97e0e974afbb/istio-proxy/0.log"
Sep 12 01:14:32.625347 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:32.625317    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cvpvm_e8f16670-c284-4e2e-b6b4-fe54a025c6a9/istio-proxy/0.log"
Sep 12 01:14:32.628148 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:32.628128    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cvpvm_e8f16670-c284-4e2e-b6b4-fe54a025c6a9/istio-proxy/1.log"
Sep 12 01:14:33.427996 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:33.427965    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fp67s/perf-node-gather-daemonset-hjjvg"
Sep 12 01:14:33.856437 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:33.856403    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f5zs5_1a25679b-9c33-46d9-a0c5-77b6922c31f6/serve-healthcheck-canary/0.log"
Sep 12 01:14:34.514865 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:34.514836    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ckmv2_87ebd59e-1ecc-4723-853e-89b54ab190a7/kube-rbac-proxy/0.log"
Sep 12 01:14:34.553283 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:34.553256    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ckmv2_87ebd59e-1ecc-4723-853e-89b54ab190a7/exporter/0.log"
Sep 12 01:14:34.587765 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:34.587736    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ckmv2_87ebd59e-1ecc-4723-853e-89b54ab190a7/extractor/0.log"
Sep 12 01:14:36.929475 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:36.929442    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw_d5daf33e-366a-458a-9006-97e0e974afbb/istio-proxy/0.log"
Sep 12 01:14:37.570886 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:37.570854    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cvpvm_e8f16670-c284-4e2e-b6b4-fe54a025c6a9/istio-proxy/0.log"
Sep 12 01:14:37.573440 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:37.573417    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cvpvm_e8f16670-c284-4e2e-b6b4-fe54a025c6a9/istio-proxy/1.log"
Sep 12 01:14:37.681280 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:37.681250    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-2xddl_482daf06-10b6-4385-95eb-2895bce457fd/manager/0.log"
Sep 12 01:14:38.171420 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:38.171393    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-j7qcq_c1e9be4b-762a-4190-9cd4-6b4d65505de6/server/0.log"
Sep 12 01:14:38.313346 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:38.313316    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-drtcs_7ce16dde-9fbf-4b0a-8148-6ff5955bf039/manager/0.log"
Sep 12 01:14:38.504068 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:38.504038    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b7744c54-mk6xw_db0f42de-90a2-4018-be00-309da0fd271c/manager/0.log"
Sep 12 01:14:38.531044 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:38.531014    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67b7744c54-mk6xw_db0f42de-90a2-4018-be00-309da0fd271c/copy-manifests/0.log"
Sep 12 01:14:39.202829 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:39.202797    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c5b6cw_d5daf33e-366a-458a-9006-97e0e974afbb/istio-proxy/0.log"
Sep 12 01:14:39.862154 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:39.862119    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7bc7647c9b-4q9nd_ad2b397a-80b9-47ae-b920-7b052d4c49ac/manager/0.log"
Sep 12 01:14:39.894236 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:39.894209    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cvpvm_e8f16670-c284-4e2e-b6b4-fe54a025c6a9/istio-proxy/0.log"
Sep 12 01:14:39.900815 ip-10-0-130-132 kubenswrapper[2567]: I0912 01:14:39.900783    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cvpvm_e8f16670-c284-4e2e-b6b4-fe54a025c6a9/istio-proxy/1.log"