Sep 08 15:42:27.181035 ip-10-0-134-107 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 15:42:27.181047 ip-10-0-134-107 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 15:42:27.181053 ip-10-0-134-107 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 15:42:27.181419 ip-10-0-134-107 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 15:42:37.867267 ip-10-0-134-107 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 08 15:42:37.867283 ip-10-0-134-107 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1e1c11f023d241058401afd82e75b4b4 --
Sep 08 15:44:13.268697 ip-10-0-134-107 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 15:44:13.699427 ip-10-0-134-107 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 15:44:13.699427 ip-10-0-134-107 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 15:44:13.699427 ip-10-0-134-107 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 15:44:13.699427 ip-10-0-134-107 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 15:44:13.699427 ip-10-0-134-107 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 15:44:13.700514 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.700424    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 15:44:13.707262 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707236    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:44:13.707262 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707257    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:44:13.707262 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707263    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:44:13.707262 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707267    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707271    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707275    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707279    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707283    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707286    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707290    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707294    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707297    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707302    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707306    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707310    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707313    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707317    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707321    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707325    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707329    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707332    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707335    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707339    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:44:13.707556 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707342    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707347    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707352    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707356    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707359    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707363    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707368    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707372    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707376    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707379    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707383    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707387    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707407    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707414    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707421    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707427    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707432    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707436    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707441    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:44:13.708351 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707445    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707449    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707453    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707457    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707462    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707466    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707470    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707474    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707478    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707482    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707487    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707491    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707495    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707499    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707505    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707510    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707514    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707518    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707522    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707526    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:44:13.709209 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707531    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707535    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707539    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707543    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707548    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707551    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707556    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707562    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707566    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707571    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707575    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707579    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707583    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707588    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707592    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707596    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707600    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707605    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707609    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707613    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:44:13.710056 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707618    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707625    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707633    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707640    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707647    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707652    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707657    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707662    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707667    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707671    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707675    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707679    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707683    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707687    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707692    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707695    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.707700    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708900    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708912    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:44:13.710920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708919    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708924    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708929    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708933    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708938    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708942    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708947    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708952    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708957    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708963    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708970    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708974    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708978    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708982    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708986    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708992    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.708996    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709000    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709004    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709009    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:44:13.711757 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709013    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709017    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709021    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709026    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709030    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709034    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709038    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709042    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709046    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709052    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709058    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709063    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709070    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709075    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709080    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709084    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709089    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709093    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709098    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:44:13.712562 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709102    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709106    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709110    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709114    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709118    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709122    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709129    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709134    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709139    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709145    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709149    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709154    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709159    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709163    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709168    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709172    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709178    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709182    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709186    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:44:13.713123 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709191    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709195    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709200    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709204    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709208    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709213    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709217    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709220    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709224    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709228    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709233    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709237    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709241    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709245    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709249    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709254    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709258    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709262    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709266    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709271    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:44:13.713631 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709275    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709280    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709284    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709291    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709295    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709300    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709304    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709309    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709313    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709317    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709322    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709326    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709330    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709334    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709338    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709343    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709347    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709352    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.709356    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710051    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:44:13.714147 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710060    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710064    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710069    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710073    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710077    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710083    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710087    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710092    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710096    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710101    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710106    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710111    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710115    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710119    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710126    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710132    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710137    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710142    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710147    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:44:13.714786 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710151    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710156    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710161    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710166    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710171    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710175    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710179    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710183    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710188    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710193    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710197    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710202    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710206    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710210    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710214    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710219    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710223    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710227    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710231    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710235    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:44:13.715418 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710239    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710244    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710248    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710252    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710255    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710259    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710263    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710267    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710271    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710277    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710282    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710286    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710290    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710294    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710299    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710304    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710310    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710314    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710318    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710322    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:44:13.715920 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710327    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710331    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710335    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710339    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710343    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710348    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710352    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710356    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710361    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710365    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710369    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710373    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710377    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710381    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710386    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710408    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710413    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710417    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710421    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:44:13.716522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710426    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710430    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710436    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710441    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710445    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710449    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710453    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710459    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710463    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710467    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710471    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710478    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710483    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710487    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710493    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710499    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710503    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710509    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710513    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:44:13.717015 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.710518    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710628    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710683    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710693    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710706    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710713    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710718    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710725    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710731    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710737    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710742    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710747    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710753    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710758    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710763    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710767    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710772    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710782    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710786    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710795    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710800    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710805    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710811    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710816    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 15:44:13.717511 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710822    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710827    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710832    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710837    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710842    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710847    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710852    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710857    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710862    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710868    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710873    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710878    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710883    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710888    2581 flags.go:64] FLAG: --enable-server="true"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710893    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710900    2581 flags.go:64] FLAG: --event-burst="100"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710905    2581 flags.go:64] FLAG: --event-qps="50"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710910    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710914    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710919    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710925    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710929    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710933    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710938    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710943    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 15:44:13.718145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710949    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710958    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710963    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710968    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710973    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710977    2581 flags.go:64] FLAG: --feature-gates=""
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710983    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710988    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710993    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.710999    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711004    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711009    2581 flags.go:64] FLAG: --help="false"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711014    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-134-107.ec2.internal"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711019    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711024    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711029    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711037    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711042    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711047    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711052    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711057    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711061    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711066    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711071    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 15:44:13.718776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711076    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711081    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711086    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711091    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711096    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711100    2581 flags.go:64] FLAG: --lock-file=""
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711105    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711109    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711114    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711125    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711131    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711137    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711142    2581 flags.go:64] FLAG: --logging-format="text"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711147    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711152    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711157    2581 flags.go:64] FLAG: --manifest-url=""
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711161    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711169    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711174    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711180    2581 flags.go:64] FLAG: --max-pods="110"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711185    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711190    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711202    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711207    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711214    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 15:44:13.719412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711219    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711224    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711235    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711240    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711245    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711250    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711254    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711264    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711269    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711275    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711280    2581 flags.go:64] FLAG: --port="10250"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711285    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711290    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0753631f05c795195"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711295    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711300    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711305    2581 flags.go:64] FLAG: --register-node="true"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711310    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711318    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711329    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711334    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711339    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711345    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711350    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711355    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711359    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 15:44:13.720062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711364    2581 flags.go:64] FLAG: --runonce="false"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711369    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711374    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711379    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711383    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711405    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711414    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711419    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711424    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711428    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711433    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711438    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711443    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711447    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711452    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711457    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711466    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711471    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711475    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711483    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711488    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711493    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711497    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711502    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711509    2581 flags.go:64] FLAG: --v="2"
Sep 08 15:44:13.720691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711515    2581 flags.go:64] FLAG: --version="false"
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711530    2581 flags.go:64] FLAG: --vmodule=""
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711537    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.711542    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711724    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711730    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711735    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711740    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711745    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711750    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711754    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711758    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711763    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711769    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711773    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711777    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711781    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711786    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711791    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711795    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:44:13.721355 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711799    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711804    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711808    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711812    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711816    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711820    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711824    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711828    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711832    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711837    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711841    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711847    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711851    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711855    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711861    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711865    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711869    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711873    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711877    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711881    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:44:13.721872 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711885    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711889    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711894    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711898    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711902    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711908    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711912    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711916    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711920    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711925    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711929    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711933    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711938    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711942    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711949    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711954    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711959    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711964    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711968    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711972    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:44:13.722356 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711976    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711980    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711985    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711990    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711995    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.711999    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712002    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712007    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712012    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712019    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712024    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712028    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712033    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712037    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712042    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712045    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712049    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712058    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712063    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:44:13.722852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712067    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712071    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712076    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712079    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712084    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712088    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712092    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712097    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712101    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712105    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712109    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712114    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712118    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712122    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712126    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712130    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712138    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712142    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712147    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712151    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:44:13.723327 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712155    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712160    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712164    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.712168    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.712923    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.720998    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.721014    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.722129    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.722137    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722206    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722211    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722214    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722217    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722220    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722223    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:44:13.723868 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722226    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722228    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722231    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722233    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722237    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722239    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722242    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722245    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722247    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722250    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722252    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722255    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722258    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722261    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722263    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722266    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722268    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722271    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722274    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:44:13.724234 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722276    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722280    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722284    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722287    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722291    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722294    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722297    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722299    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722302    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722304    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722307    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722310    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722312    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722315    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722317    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722320    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722322    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722325    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722328    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722330    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:44:13.724700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722332    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722335    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722339    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722343    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722346    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722349    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722352    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722355    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722358    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722361    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722364    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722366    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722369    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722372    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722374    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722377    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722380    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722384    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722387    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:44:13.725217 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722402    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722406    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722410    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722413    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722415    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722418    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722421    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722423    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722426    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722429    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722431    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722434    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722436    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722439    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722441    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722444    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722447    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722449    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722452    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722454    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:44:13.725699 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722457    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722460    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722462    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722465    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722467    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722469    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722472    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722475    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722478    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722483    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722486    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722490    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722493    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722496    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:44:13.726173 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722499    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.722504    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722615    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722620    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722623    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722626    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722629    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722632    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722635    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722639    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722643    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722646    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722648    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722651    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:44:13.726540 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722654    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722656    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722658    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722661    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722663    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722666    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722669    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722672    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722674    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722677    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722680    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722682    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722685    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722688    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722691    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722693    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722697    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722699    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722702    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722704    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:44:13.726879 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722707    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722710    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722712    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722716    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722719    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722722    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722725    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722728    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722730    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722734    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722736    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722739    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722742    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722745    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722748    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722750    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722753    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722756    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722759    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722762    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:44:13.727366 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722765    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722767    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722771    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722773    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722776    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722778    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722781    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722783    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722786    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722789    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722793    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722795    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722798    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722801    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722804    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722807    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722809    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722812    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722815    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722818    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:44:13.727862 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722821    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722823    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722826    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722828    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722830    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722833    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722835    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722838    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722840    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722842    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722845    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722847    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722850    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722853    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722855    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722858    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722861    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722864    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722866    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:44:13.728336 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722869    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:44:13.728808 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722872    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:44:13.728808 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722874    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:44:13.728808 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722877    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:44:13.728808 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722880    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:44:13.728808 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722882    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:44:13.728808 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722885    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:44:13.728808 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:13.722887    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:44:13.728808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.722892    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 15:44:13.728808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.723010    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 15:44:13.728808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.725575    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 15:44:13.728808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.726712    2581 server.go:1019] "Starting client certificate rotation"
Sep 08 15:44:13.728808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.726815    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 15:44:13.728808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.726847    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 15:44:13.730265 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.730248    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 15:44:13.731583 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.731559    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 15:44:13.743281 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.743258    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 08 15:44:13.746565 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.746552    2581 log.go:25] "Validated CRI v1 image API"
Sep 08 15:44:13.747355 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.747339    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 15:44:13.751590 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.751568    2581 fs.go:135] Filesystem UUIDs: map[0a063a79-5c5b-4373-b2b2-3bfe222a296b:/dev/nvme0n1p3 689eff02-93bf-429c-a3d0-113dd6cddf42:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 08 15:44:13.751662 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.751587    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 15:44:13.755897 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.755791    2581 manager.go:216] Machine: {Timestamp:2026-09-08 15:44:13.755297 +0000 UTC m=+0.378989314 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3084585 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28ef7c79586d6e22046cdee23f60a0 SystemUUID:ec28ef7c-7958-6d6e-2204-6cdee23f60a0 BootID:1e1c11f0-23d2-4105-8401-afd82e75b4b4 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5a:2a:6b:d8:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5a:2a:6b:d8:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:64:0c:2a:8f:7d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 15:44:13.755897 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.755893    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 15:44:13.756022 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.756010    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 15:44:13.758808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.758780    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 15:44:13.758950 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.758811    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-107.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 15:44:13.760315 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.760303    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 15:44:13.760347 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.760322    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 15:44:13.760347 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.760331    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 15:44:13.761136 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.761124    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 15:44:13.763155 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.763140    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 15:44:13.767895 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.767879    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 15:44:13.768040 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.768030    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 15:44:13.768074 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.768067    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 15:44:13.768105 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.768081    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 15:44:13.768105 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.768097    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 15:44:13.768105 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.768106    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 08 15:44:13.768203 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.768116    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 15:44:13.768501 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.768489    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 15:44:13.768541 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.768511    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 15:44:13.771832 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.771813    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 15:44:13.774953 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.774938    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 15:44:13.775024 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.774984    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 15:44:13.775156 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775140    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 15:44:13.775156 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775159    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 15:44:13.775297 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775168    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 15:44:13.775297 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775173    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 15:44:13.775297 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775181    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 15:44:13.775297 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775186    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 15:44:13.775808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775794    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 15:44:13.775808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775806    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 15:44:13.775898 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775813    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 15:44:13.775898 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775819    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 15:44:13.775898 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775827    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 15:44:13.775898 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775835    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 15:44:13.775898 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775874    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 15:44:13.775898 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.775880    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 15:44:13.779768 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.779752    2581 server.go:1268] "Started kubelet"
Sep 08 15:44:13.779862 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.779834    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 15:44:13.779927 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.779856    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 15:44:13.779977 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.779950    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 15:44:13.784609 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.784580    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 15:44:13.784932 ip-10-0-134-107 systemd[1]: Started Kubernetes Kubelet.
Sep 08 15:44:13.786067 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.785898    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 15:44:13.786067 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.785918    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 15:44:13.786184 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.786085    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-107.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 15:44:13.786794 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.786777    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 15:44:13.786944 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.786925    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 15:44:13.788078 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.788056    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 15:44:13.789945 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.788606    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-107.ec2.internal.18d362ee918a6375  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-107.ec2.internal,UID:ip-10-0-134-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-107.ec2.internal,},FirstTimestamp:2026-09-08 15:44:13.779723125 +0000 UTC m=+0.403415440,LastTimestamp:2026-09-08 15:44:13.779723125 +0000 UTC m=+0.403415440,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-107.ec2.internal,}"
Sep 08 15:44:13.795401 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.795367    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 15:44:13.796082 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.796067    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 15:44:13.796148 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.796135    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-107.ec2.internal\" not found"
Sep 08 15:44:13.796863 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.796845    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 15:44:13.796863 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.796865    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 15:44:13.797018 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.796848    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 15:44:13.797018 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.796933    2581 factory.go:153] Registering CRI-O factory
Sep 08 15:44:13.797018 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.796949    2581 factory.go:223] Registration of the crio container factory successfully
Sep 08 15:44:13.797018 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.796979    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 15:44:13.797018 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.796986    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 15:44:13.797018 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.796999    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 15:44:13.797018 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.797009    2581 factory.go:55] Registering systemd factory
Sep 08 15:44:13.797018 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.797016    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 08 15:44:13.797364 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.797033    2581 factory.go:103] Registering Raw factory
Sep 08 15:44:13.797364 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.797048    2581 manager.go:1196] Started watching for new ooms in manager
Sep 08 15:44:13.797494 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.797450    2581 manager.go:349] Starting recovery of all containers
Sep 08 15:44:13.802343 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.801715    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 15:44:13.802343 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.801826    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 15:44:13.807661 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.807645    2581 manager.go:354] Recovery completed
Sep 08 15:44:13.812156 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.812141    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:44:13.814322 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.814297    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:44:13.814387 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.814338    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:44:13.814387 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.814349    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:44:13.814866 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.814848    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 15:44:13.814866 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.814865    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 15:44:13.814954 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.814882    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 15:44:13.815302 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.814905    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:44:13.816188 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.816169    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:44:13.816275 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.816206    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:44:13.816275 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.816219    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:44:13.817132 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.817114    2581 policy_none.go:49] "None policy: Start"
Sep 08 15:44:13.817245 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.817137    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 15:44:13.817245 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.817152    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 15:44:13.819304 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.819290    2581 policy_none.go:47] "Start"
Sep 08 15:44:13.828424 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.828331    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-107.ec2.internal.18d362ee939a5007  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-107.ec2.internal,UID:ip-10-0-134-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-107.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-107.ec2.internal,},FirstTimestamp:2026-09-08 15:44:13.814321159 +0000 UTC m=+0.438013478,LastTimestamp:2026-09-08 15:44:13.814321159 +0000 UTC m=+0.438013478,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-107.ec2.internal,}"
Sep 08 15:44:13.843677 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.843608    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-107.ec2.internal.18d362ee939aa6ce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-107.ec2.internal,UID:ip-10-0-134-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-107.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-107.ec2.internal,},FirstTimestamp:2026-09-08 15:44:13.814343374 +0000 UTC m=+0.438035689,LastTimestamp:2026-09-08 15:44:13.814343374 +0000 UTC m=+0.438035689,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-107.ec2.internal,}"
Sep 08 15:44:13.848973 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.848959    2581 manager.go:337] "Starting Device Plugin manager"
Sep 08 15:44:13.871290 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.848993    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 15:44:13.871290 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.849004    2581 server.go:87] "Starting device plugin registration server"
Sep 08 15:44:13.871290 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.849259    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 15:44:13.871290 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.849271    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 15:44:13.871290 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.849467    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 15:44:13.871290 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.849556    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 15:44:13.871290 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.849731    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 15:44:13.871290 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.849984    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 15:44:13.871290 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.850052    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-107.ec2.internal\" not found"
Sep 08 15:44:13.871290 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.854976    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-107.ec2.internal.18d362ee939ace3c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-107.ec2.internal,UID:ip-10-0-134-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-107.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-107.ec2.internal,},FirstTimestamp:2026-09-08 15:44:13.814353468 +0000 UTC m=+0.438045783,LastTimestamp:2026-09-08 15:44:13.814353468 +0000 UTC m=+0.438045783,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-107.ec2.internal,}"
Sep 08 15:44:13.871290 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.868323    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-107.ec2.internal.18d362ee939a5007\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-107.ec2.internal.18d362ee939a5007  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-107.ec2.internal,UID:ip-10-0-134-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-107.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-107.ec2.internal,},FirstTimestamp:2026-09-08 15:44:13.814321159 +0000 UTC m=+0.438013478,LastTimestamp:2026-09-08 15:44:13.816187357 +0000 UTC m=+0.439879676,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-107.ec2.internal,}"
Sep 08 15:44:13.877236 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.877165    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-107.ec2.internal.18d362ee939aa6ce\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-107.ec2.internal.18d362ee939aa6ce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-107.ec2.internal,UID:ip-10-0-134-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-107.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-107.ec2.internal,},FirstTimestamp:2026-09-08 15:44:13.814343374 +0000 UTC m=+0.438035689,LastTimestamp:2026-09-08 15:44:13.816212402 +0000 UTC m=+0.439904718,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-107.ec2.internal,}"
Sep 08 15:44:13.890007 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.889934    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-107.ec2.internal.18d362ee939ace3c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-107.ec2.internal.18d362ee939ace3c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-107.ec2.internal,UID:ip-10-0-134-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-107.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-107.ec2.internal,},FirstTimestamp:2026-09-08 15:44:13.814353468 +0000 UTC m=+0.438045783,LastTimestamp:2026-09-08 15:44:13.816225133 +0000 UTC m=+0.439917448,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-107.ec2.internal,}"
Sep 08 15:44:13.892779 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.892747    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 15:44:13.893917 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.893899    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-chtxm"
Sep 08 15:44:13.893917 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.893904    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 15:44:13.894045 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.893952    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 15:44:13.894045 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.893972    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 15:44:13.894146 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.894055    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 15:44:13.899015 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.898987    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 08 15:44:13.899098 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.899016    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-107.ec2.internal.18d362ee95cf4cf5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-107.ec2.internal,UID:ip-10-0-134-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-134-107.ec2.internal,},FirstTimestamp:2026-09-08 15:44:13.851348213 +0000 UTC m=+0.475040515,LastTimestamp:2026-09-08 15:44:13.851348213 +0000 UTC m=+0.475040515,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-107.ec2.internal,}"
Sep 08 15:44:13.901940 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.901925    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-chtxm"
Sep 08 15:44:13.950291 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.950238    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:44:13.951187 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.951171    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:44:13.951272 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.951209    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:44:13.951272 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.951222    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:44:13.951272 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.951246    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-107.ec2.internal"
Sep 08 15:44:13.965110 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:13.965085    2581 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-134-107.ec2.internal\" not found" node="ip-10-0-134-107.ec2.internal"
Sep 08 15:44:13.994520 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.994492    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-107.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal"]
Sep 08 15:44:13.994633 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.994570    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:44:13.996089 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.996064    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:44:13.996222 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.996099    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:44:13.996222 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.996113    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:44:13.998634 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.998619    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:44:13.998721 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.998712    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:13.998771 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.998741    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:44:13.999434 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.999415    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:44:13.999434 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.999427    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:44:13.999593 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.999445    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:44:13.999593 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.999449    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:44:13.999593 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.999457    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:44:13.999593 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:13.999462    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:44:14.001764 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.001744    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.001857 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.001780    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:44:14.002595 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.002578    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:44:14.002683 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.002601    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:44:14.002683 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.002613    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:44:14.008723 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:14.008706    2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-107.ec2.internal\" not found" node="ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.017919 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:14.017901    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-107.ec2.internal\" not found" node="ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.021282 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:14.021267    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-107.ec2.internal\" not found" node="ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.098060 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.098031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5032d201ce1921952e1262cfc0d74711-config\") pod \"kube-apiserver-proxy-ip-10-0-134-107.ec2.internal\" (UID: \"5032d201ce1921952e1262cfc0d74711\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.166185 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.166148    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:44:14.167223 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.167205    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:44:14.167309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.167237    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:44:14.167309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.167249    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:44:14.167309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.167275    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.178301 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.178283    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.178366 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:14.178304    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-107.ec2.internal\": node \"ip-10-0-134-107.ec2.internal\" not found"
Sep 08 15:44:14.198800 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.198775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5032d201ce1921952e1262cfc0d74711-config\") pod \"kube-apiserver-proxy-ip-10-0-134-107.ec2.internal\" (UID: \"5032d201ce1921952e1262cfc0d74711\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.198911 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.198803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/756114e43effe75943fb30ab8ace2af1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal\" (UID: \"756114e43effe75943fb30ab8ace2af1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.198911 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.198827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/756114e43effe75943fb30ab8ace2af1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal\" (UID: \"756114e43effe75943fb30ab8ace2af1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.198911 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.198901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5032d201ce1921952e1262cfc0d74711-config\") pod \"kube-apiserver-proxy-ip-10-0-134-107.ec2.internal\" (UID: \"5032d201ce1921952e1262cfc0d74711\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.212414 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:14.212351    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-107.ec2.internal\" not found"
Sep 08 15:44:14.299018 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.298995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/756114e43effe75943fb30ab8ace2af1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal\" (UID: \"756114e43effe75943fb30ab8ace2af1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.299116 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.299024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/756114e43effe75943fb30ab8ace2af1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal\" (UID: \"756114e43effe75943fb30ab8ace2af1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.299116 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.299071    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/756114e43effe75943fb30ab8ace2af1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal\" (UID: \"756114e43effe75943fb30ab8ace2af1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.299116 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.299087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/756114e43effe75943fb30ab8ace2af1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal\" (UID: \"756114e43effe75943fb30ab8ace2af1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.313131 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:14.313100    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-107.ec2.internal\" not found"
Sep 08 15:44:14.321304 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.321274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.324422 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.324388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:14.413376 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:14.413335    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-107.ec2.internal\" not found"
Sep 08 15:44:14.513927 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:14.513835    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-107.ec2.internal\" not found"
Sep 08 15:44:14.614224 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:14.614199    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-107.ec2.internal\" not found"
Sep 08 15:44:14.706539 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.706515    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:44:14.714519 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:14.714495    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-107.ec2.internal\" not found"
Sep 08 15:44:14.726532 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.726508    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 15:44:14.726648 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.726630    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 15:44:14.815672 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:14.815612    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-107.ec2.internal\" not found"
Sep 08 15:44:14.904787 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.904748    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 15:39:14 +0000 UTC" deadline="2028-06-05 16:02:38.06195327 +0000 UTC"
Sep 08 15:44:14.904787 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.904780    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15264h18m23.157176726s"
Sep 08 15:44:14.915993 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:14.915959    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-107.ec2.internal\" not found"
Sep 08 15:44:14.927979 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.927956    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:44:14.929019 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.929004    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:44:14.929082 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.929038    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:44:14.929082 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:14.929048    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:44:15.016270 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:15.016225    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-107.ec2.internal\" not found"
Sep 08 15:44:15.107821 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.107746    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:44:15.117359 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:15.117333    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-107.ec2.internal\" not found"
Sep 08 15:44:15.122596 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.122578    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:44:15.180908 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:15.180874    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5032d201ce1921952e1262cfc0d74711.slice/crio-eba1845d4b07abaddf446b02cfc50465f1c0feeb0617092114fce0aca0137dd3 WatchSource:0}: Error finding container eba1845d4b07abaddf446b02cfc50465f1c0feeb0617092114fce0aca0137dd3: Status 404 returned error can't find the container with id eba1845d4b07abaddf446b02cfc50465f1c0feeb0617092114fce0aca0137dd3
Sep 08 15:44:15.188457 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.188437    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:44:15.197188 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.197163    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:15.211007 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.210981    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 15:44:15.212677 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.212659    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-107.ec2.internal"
Sep 08 15:44:15.221495 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.221474    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 15:44:15.370727 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.370645    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:44:15.771659 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.771624    2581 apiserver.go:52] "Watching apiserver"
Sep 08 15:44:15.777975 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.777949    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 15:44:15.778417 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.778391    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x","openshift-dns/node-resolver-w4nwh","openshift-image-registry/node-ca-l6lrc","openshift-multus/network-metrics-daemon-nkn7w","openshift-network-diagnostics/network-check-target-8r2bg","kube-system/konnectivity-agent-thsrk","kube-system/kube-apiserver-proxy-ip-10-0-134-107.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qqmj7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal","openshift-multus/multus-additional-cni-plugins-ph9tx","openshift-multus/multus-jbqc7","openshift-network-operator/iptables-alerter-m4j5s","openshift-ovn-kubernetes/ovnkube-node-v2wc2","kube-system/global-pull-secret-syncer-xbnhj"]
Sep 08 15:44:15.781101 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.781077    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-thsrk"
Sep 08 15:44:15.783625 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.783601    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 15:44:15.783625 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.783618    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.783811 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.783649    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-984vf\""
Sep 08 15:44:15.783876 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.783868    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 15:44:15.786020 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.785995    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 15:44:15.786912 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.786438    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 15:44:15.786912 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.786453    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 15:44:15.786912 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.786664    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 15:44:15.786912 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.786724    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 15:44:15.786912 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.786796    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wl4zn\""
Sep 08 15:44:15.786912 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.786848    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 15:44:15.786912 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.786876    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l6lrc"
Sep 08 15:44:15.788412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.788374    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:15.788526 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:15.788460    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:15.789211 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.789192    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 15:44:15.789946 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.789907    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jq578\""
Sep 08 15:44:15.790438 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.790418    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 15:44:15.790541 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.790422    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 15:44:15.793011 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.792991    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:15.793133 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:15.793110    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:15.795045 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.795024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.796136 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.796116    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 15:44:15.797980 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.797957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.798993 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.798973    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 15:44:15.799362 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.799328    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 15:44:15.799590 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.799204    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2p5x4\""
Sep 08 15:44:15.799590 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.799185    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 15:44:15.800151 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.800132    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 15:44:15.800333 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.800313    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:44:15.800661 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.800642    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:15.800850 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.800802    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m4x2n\""
Sep 08 15:44:15.803196 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.803177    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 15:44:15.803654 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.803632    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 15:44:15.803897 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.803878    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 15:44:15.804136 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.804121    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 15:44:15.804730 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.804304    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5n4jk\""
Sep 08 15:44:15.804730 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.804552    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 15:44:15.805339 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.805255    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.805989 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.805967    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m4j5s"
Sep 08 15:44:15.807576 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.807335    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zxh9j\""
Sep 08 15:44:15.807576 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.807543    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 15:44:15.808155 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.808135    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:44:15.808240 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.808181    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 15:44:15.808347 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.808330    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 15:44:15.808506 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.808486    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6xjrr\""
Sep 08 15:44:15.809491 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-kubelet\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.809580 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-slash\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.809580 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-var-lib-openvswitch\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.809580 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-run-openvswitch\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.809736 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.809736 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-systemd\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.809736 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809645    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2201c616-9356-494d-915c-edee0306a6bf-host\") pod \"node-ca-l6lrc\" (UID: \"2201c616-9356-494d-915c-edee0306a6bf\") " pod="openshift-image-registry/node-ca-l6lrc"
Sep 08 15:44:15.809736 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdg5t\" (UniqueName: \"kubernetes.io/projected/2201c616-9356-494d-915c-edee0306a6bf-kube-api-access-jdg5t\") pod \"node-ca-l6lrc\" (UID: \"2201c616-9356-494d-915c-edee0306a6bf\") " pod="openshift-image-registry/node-ca-l6lrc"
Sep 08 15:44:15.809736 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-systemd-units\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.809736 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-run-ovn\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-cni-bin\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44035804-234e-4b1f-9f69-938469b3446f-env-overrides\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-var-lib-kubelet\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs\") pod \"network-metrics-daemon-nkn7w\" (UID: \"996123f7-ff5c-4ffd-8a4d-ebfb862d664e\") " pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4tvk\" (UniqueName: \"kubernetes.io/projected/44035804-234e-4b1f-9f69-938469b3446f-kube-api-access-d4tvk\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-kubernetes\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ed950cd-31e9-4a85-89ac-83818614ad83-agent-certs\") pod \"konnectivity-agent-thsrk\" (UID: \"4ed950cd-31e9-4a85-89ac-83818614ad83\") " pod="kube-system/konnectivity-agent-thsrk"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-etc-openvswitch\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-node-log\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.809976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-log-socket\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44035804-234e-4b1f-9f69-938469b3446f-ovnkube-config\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ed950cd-31e9-4a85-89ac-83818614ad83-konnectivity-ca\") pod \"konnectivity-agent-thsrk\" (UID: \"4ed950cd-31e9-4a85-89ac-83818614ad83\") " pod="kube-system/konnectivity-agent-thsrk"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44035804-234e-4b1f-9f69-938469b3446f-ovn-node-metrics-cert\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.810090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-sysctl-d\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-lib-modules\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3128409-f035-4e2d-bb79-8c4b77ebb807-tmp\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810163    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6bk\" (UniqueName: \"kubernetes.io/projected/e3128409-f035-4e2d-bb79-8c4b77ebb807-kube-api-access-8t6bk\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2201c616-9356-494d-915c-edee0306a6bf-serviceca\") pod \"node-ca-l6lrc\" (UID: \"2201c616-9356-494d-915c-edee0306a6bf\") " pod="openshift-image-registry/node-ca-l6lrc"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-cni-netd\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-sysconfig\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bxz\" (UniqueName: \"kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz\") pod \"network-check-target-8r2bg\" (UID: \"da5fbd85-bda7-46ab-98ea-255e07a0fb82\") " pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tc8z\" (UniqueName: \"kubernetes.io/projected/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-kube-api-access-4tc8z\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-modprobe-d\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-sysctl-conf\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-run\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-host\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vthfd\" (UniqueName: \"kubernetes.io/projected/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-kube-api-access-vthfd\") pod \"network-metrics-daemon-nkn7w\" (UID: \"996123f7-ff5c-4ffd-8a4d-ebfb862d664e\") " pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.810838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-socket-dir\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.811672 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.811672 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44035804-234e-4b1f-9f69-938469b3446f-ovnkube-script-lib\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.811672 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-sys\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.811672 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-registration-dir\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.811672 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-sys-fs\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.811672 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-run-netns\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.811672 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-run-systemd\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.811672 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-tuned\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.811672 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.810749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-device-dir\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.811672 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.811139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w4nwh"
Sep 08 15:44:15.812262 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.811873    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:15.812411 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:15.812392    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:15.812798 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.812776    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 15:44:15.813008 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.812989    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 15:44:15.813129 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.813081    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qq84x\""
Sep 08 15:44:15.813574 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.813554    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 15:44:15.841726 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.841619    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-w4tw4"
Sep 08 15:44:15.851546 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.851490    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w4tw4"
Sep 08 15:44:15.859985 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:15.859937    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756114e43effe75943fb30ab8ace2af1.slice/crio-a3a56edb1a166f55ded2ecb326528791f5378465dd1aa49feb4096566e71e6c3 WatchSource:0}: Error finding container a3a56edb1a166f55ded2ecb326528791f5378465dd1aa49feb4096566e71e6c3: Status 404 returned error can't find the container with id a3a56edb1a166f55ded2ecb326528791f5378465dd1aa49feb4096566e71e6c3
Sep 08 15:44:15.897956 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.897934    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 15:44:15.899092 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.899049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal" event={"ID":"756114e43effe75943fb30ab8ace2af1","Type":"ContainerStarted","Data":"a3a56edb1a166f55ded2ecb326528791f5378465dd1aa49feb4096566e71e6c3"}
Sep 08 15:44:15.901148 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.901119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-107.ec2.internal" event={"ID":"5032d201ce1921952e1262cfc0d74711","Type":"ContainerStarted","Data":"eba1845d4b07abaddf446b02cfc50465f1c0feeb0617092114fce0aca0137dd3"}
Sep 08 15:44:15.911293 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2201c616-9356-494d-915c-edee0306a6bf-host\") pod \"node-ca-l6lrc\" (UID: \"2201c616-9356-494d-915c-edee0306a6bf\") " pod="openshift-image-registry/node-ca-l6lrc"
Sep 08 15:44:15.911432 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65d13726-37f1-465a-bbd5-1ebf50e5ead8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:15.911432 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-var-lib-cni-multus\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.911432 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-var-lib-kubelet\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.911598 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a23223-f446-4941-9b20-6badf3aee2e3-tmp-dir\") pod \"node-resolver-w4nwh\" (UID: \"20a23223-f446-4941-9b20-6badf3aee2e3\") " pod="openshift-dns/node-resolver-w4nwh"
Sep 08 15:44:15.911598 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-systemd-units\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.911598 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-cni-bin\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.911598 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44035804-234e-4b1f-9f69-938469b3446f-env-overrides\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.911598 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp72d\" (UniqueName: \"kubernetes.io/projected/baadbf65-d286-4e72-90b9-abe37d115061-kube-api-access-wp72d\") pod \"iptables-alerter-m4j5s\" (UID: \"baadbf65-d286-4e72-90b9-abe37d115061\") " pod="openshift-network-operator/iptables-alerter-m4j5s"
Sep 08 15:44:15.911598 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-systemd-units\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.911598 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-cni-bin\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.911598 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw9gn\" (UniqueName: \"kubernetes.io/projected/d6b5446c-990e-467e-a319-150edf10bcc1-kube-api-access-kw9gn\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.911966 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20a23223-f446-4941-9b20-6badf3aee2e3-hosts-file\") pod \"node-resolver-w4nwh\" (UID: \"20a23223-f446-4941-9b20-6badf3aee2e3\") " pod="openshift-dns/node-resolver-w4nwh"
Sep 08 15:44:15.911966 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911790    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2201c616-9356-494d-915c-edee0306a6bf-host\") pod \"node-ca-l6lrc\" (UID: \"2201c616-9356-494d-915c-edee0306a6bf\") " pod="openshift-image-registry/node-ca-l6lrc"
Sep 08 15:44:15.911966 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44035804-234e-4b1f-9f69-938469b3446f-ovn-node-metrics-cert\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.911966 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4tvk\" (UniqueName: \"kubernetes.io/projected/44035804-234e-4b1f-9f69-938469b3446f-kube-api-access-d4tvk\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.911966 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-kubernetes\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.911966 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-etc-openvswitch\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.911966 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-log-socket\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.912309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.911981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ed950cd-31e9-4a85-89ac-83818614ad83-konnectivity-ca\") pod \"konnectivity-agent-thsrk\" (UID: \"4ed950cd-31e9-4a85-89ac-83818614ad83\") " pod="kube-system/konnectivity-agent-thsrk"
Sep 08 15:44:15.912309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-log-socket\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.912309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-sysctl-d\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.912309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-lib-modules\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.912309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6bk\" (UniqueName: \"kubernetes.io/projected/e3128409-f035-4e2d-bb79-8c4b77ebb807-kube-api-access-8t6bk\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.912309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912141    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65d13726-37f1-465a-bbd5-1ebf50e5ead8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:15.912309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-run-netns\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.912309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912193    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-hostroot\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.912309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.912309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-etc-kubernetes\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.912309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-modprobe-d\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-run\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/baadbf65-d286-4e72-90b9-abe37d115061-host-slash\") pod \"iptables-alerter-m4j5s\" (UID: \"baadbf65-d286-4e72-90b9-abe37d115061\") " pod="openshift-network-operator/iptables-alerter-m4j5s"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e92972bf-2b13-43de-bc6c-ccf50050fa24-dbus\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44035804-234e-4b1f-9f69-938469b3446f-ovnkube-script-lib\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-sys\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-multus-cni-dir\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6b5446c-990e-467e-a319-150edf10bcc1-cni-binary-copy\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6b5446c-990e-467e-a319-150edf10bcc1-multus-daemon-config\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-run-netns\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-run-systemd\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912734    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-run-netns\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44035804-234e-4b1f-9f69-938469b3446f-env-overrides\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-system-cni-dir\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.912900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912816    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-run-systemd\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-kubernetes\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-modprobe-d\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-run\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.912979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-etc-openvswitch\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-sysctl-d\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-lib-modules\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913148    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-sys\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-run-multus-certs\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44035804-234e-4b1f-9f69-938469b3446f-ovnkube-script-lib\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-kubelet\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-slash\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-var-lib-openvswitch\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-run-openvswitch\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-kubelet\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.913742 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-systemd\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913786    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-var-lib-openvswitch\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdg5t\" (UniqueName: \"kubernetes.io/projected/2201c616-9356-494d-915c-edee0306a6bf-kube-api-access-jdg5t\") pod \"node-ca-l6lrc\" (UID: \"2201c616-9356-494d-915c-edee0306a6bf\") " pod="openshift-image-registry/node-ca-l6lrc"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913797    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ed950cd-31e9-4a85-89ac-83818614ad83-konnectivity-ca\") pod \"konnectivity-agent-thsrk\" (UID: \"4ed950cd-31e9-4a85-89ac-83818614ad83\") " pod="kube-system/konnectivity-agent-thsrk"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54th5\" (UniqueName: \"kubernetes.io/projected/65d13726-37f1-465a-bbd5-1ebf50e5ead8-kube-api-access-54th5\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-systemd\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-run-ovn\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-var-lib-kubelet\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs\") pod \"network-metrics-daemon-nkn7w\" (UID: \"996123f7-ff5c-4ffd-8a4d-ebfb862d664e\") " pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-slash\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.913994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e92972bf-2b13-43de-bc6c-ccf50050fa24-kubelet-config\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3128409-f035-4e2d-bb79-8c4b77ebb807-tmp\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914026    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-run-openvswitch\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-var-lib-kubelet\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914107    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-socket-dir\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.914544 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:15.914183    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914216    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:15.914270    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs podName:996123f7-ff5c-4ffd-8a4d-ebfb862d664e nodeName:}" failed. No retries permitted until 2026-09-08 15:44:16.414238687 +0000 UTC m=+3.037931006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs") pod "network-metrics-daemon-nkn7w" (UID: "996123f7-ff5c-4ffd-8a4d-ebfb862d664e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-run-ovn\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-socket-dir\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ed950cd-31e9-4a85-89ac-83818614ad83-agent-certs\") pod \"konnectivity-agent-thsrk\" (UID: \"4ed950cd-31e9-4a85-89ac-83818614ad83\") " pod="kube-system/konnectivity-agent-thsrk"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-node-log\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44035804-234e-4b1f-9f69-938469b3446f-ovnkube-config\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914560    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/baadbf65-d286-4e72-90b9-abe37d115061-iptables-alerter-script\") pod \"iptables-alerter-m4j5s\" (UID: \"baadbf65-d286-4e72-90b9-abe37d115061\") " pod="openshift-network-operator/iptables-alerter-m4j5s"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2201c616-9356-494d-915c-edee0306a6bf-serviceca\") pod \"node-ca-l6lrc\" (UID: \"2201c616-9356-494d-915c-edee0306a6bf\") " pod="openshift-image-registry/node-ca-l6lrc"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tc8z\" (UniqueName: \"kubernetes.io/projected/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-kube-api-access-4tc8z\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65d13726-37f1-465a-bbd5-1ebf50e5ead8-cnibin\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65d13726-37f1-465a-bbd5-1ebf50e5ead8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-multus-socket-dir-parent\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-var-lib-cni-bin\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-node-log\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.915456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.914844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-cni-netd\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-sysconfig\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bxz\" (UniqueName: \"kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz\") pod \"network-check-target-8r2bg\" (UID: \"da5fbd85-bda7-46ab-98ea-255e07a0fb82\") " pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65d13726-37f1-465a-bbd5-1ebf50e5ead8-system-cni-dir\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44035804-234e-4b1f-9f69-938469b3446f-host-cni-netd\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-cnibin\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915268    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-sysconfig\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-run-k8s-cni-cncf-io\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-sysctl-conf\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2201c616-9356-494d-915c-edee0306a6bf-serviceca\") pod \"node-ca-l6lrc\" (UID: \"2201c616-9356-494d-915c-edee0306a6bf\") " pod="openshift-image-registry/node-ca-l6lrc"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-host\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vthfd\" (UniqueName: \"kubernetes.io/projected/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-kube-api-access-vthfd\") pod \"network-metrics-daemon-nkn7w\" (UID: \"996123f7-ff5c-4ffd-8a4d-ebfb862d664e\") " pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915518    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-sysctl-conf\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-sys-fs\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915560    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3128409-f035-4e2d-bb79-8c4b77ebb807-host\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.916349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-multus-conf-dir\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:15.917145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-registration-dir\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.917145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65d13726-37f1-465a-bbd5-1ebf50e5ead8-cni-binary-copy\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:15.917145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-tuned\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.917145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-device-dir\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.917145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-sys-fs\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.917145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckskc\" (UniqueName: \"kubernetes.io/projected/20a23223-f446-4941-9b20-6badf3aee2e3-kube-api-access-ckskc\") pod \"node-resolver-w4nwh\" (UID: \"20a23223-f446-4941-9b20-6badf3aee2e3\") " pod="openshift-dns/node-resolver-w4nwh"
Sep 08 15:44:15.917145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915877    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.917145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.915955    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-registration-dir\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.917145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.916019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-device-dir\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:15.917145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.916051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44035804-234e-4b1f-9f69-938469b3446f-ovnkube-config\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.917145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.917013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44035804-234e-4b1f-9f69-938469b3446f-ovn-node-metrics-cert\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.917781 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.917677    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3128409-f035-4e2d-bb79-8c4b77ebb807-tmp\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.918760 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.918728    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3128409-f035-4e2d-bb79-8c4b77ebb807-etc-tuned\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.919701 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.919679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ed950cd-31e9-4a85-89ac-83818614ad83-agent-certs\") pod \"konnectivity-agent-thsrk\" (UID: \"4ed950cd-31e9-4a85-89ac-83818614ad83\") " pod="kube-system/konnectivity-agent-thsrk"
Sep 08 15:44:15.921203 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.921179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6bk\" (UniqueName: \"kubernetes.io/projected/e3128409-f035-4e2d-bb79-8c4b77ebb807-kube-api-access-8t6bk\") pod \"tuned-qqmj7\" (UID: \"e3128409-f035-4e2d-bb79-8c4b77ebb807\") " pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:15.921832 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.921810    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4tvk\" (UniqueName: \"kubernetes.io/projected/44035804-234e-4b1f-9f69-938469b3446f-kube-api-access-d4tvk\") pod \"ovnkube-node-v2wc2\" (UID: \"44035804-234e-4b1f-9f69-938469b3446f\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:15.923515 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:15.923491    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:44:15.923652 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:15.923518    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:44:15.923652 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:15.923534    2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8bxz for pod openshift-network-diagnostics/network-check-target-8r2bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:15.923652 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:15.923602    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz podName:da5fbd85-bda7-46ab-98ea-255e07a0fb82 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:16.423584237 +0000 UTC m=+3.047276541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k8bxz" (UniqueName: "kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz") pod "network-check-target-8r2bg" (UID: "da5fbd85-bda7-46ab-98ea-255e07a0fb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:15.924804 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.924777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdg5t\" (UniqueName: \"kubernetes.io/projected/2201c616-9356-494d-915c-edee0306a6bf-kube-api-access-jdg5t\") pod \"node-ca-l6lrc\" (UID: \"2201c616-9356-494d-915c-edee0306a6bf\") " pod="openshift-image-registry/node-ca-l6lrc"
Sep 08 15:44:15.926516 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.926389    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vthfd\" (UniqueName: \"kubernetes.io/projected/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-kube-api-access-vthfd\") pod \"network-metrics-daemon-nkn7w\" (UID: \"996123f7-ff5c-4ffd-8a4d-ebfb862d664e\") " pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:15.926516 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:15.926510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tc8z\" (UniqueName: \"kubernetes.io/projected/ebfad9a3-ef15-4dcc-a831-9d55a6cd659c-kube-api-access-4tc8z\") pod \"aws-ebs-csi-driver-node-ptd7x\" (UID: \"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:16.016538 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54th5\" (UniqueName: \"kubernetes.io/projected/65d13726-37f1-465a-bbd5-1ebf50e5ead8-kube-api-access-54th5\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.016700 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e92972bf-2b13-43de-bc6c-ccf50050fa24-kubelet-config\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:16.016700 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/baadbf65-d286-4e72-90b9-abe37d115061-iptables-alerter-script\") pod \"iptables-alerter-m4j5s\" (UID: \"baadbf65-d286-4e72-90b9-abe37d115061\") " pod="openshift-network-operator/iptables-alerter-m4j5s"
Sep 08 15:44:16.016700 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65d13726-37f1-465a-bbd5-1ebf50e5ead8-cnibin\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.016700 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65d13726-37f1-465a-bbd5-1ebf50e5ead8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.016700 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-multus-socket-dir-parent\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-var-lib-cni-bin\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65d13726-37f1-465a-bbd5-1ebf50e5ead8-cnibin\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65d13726-37f1-465a-bbd5-1ebf50e5ead8-system-cni-dir\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-cnibin\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-var-lib-cni-bin\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-multus-socket-dir-parent\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-run-k8s-cni-cncf-io\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-multus-conf-dir\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016894    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-cnibin\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65d13726-37f1-465a-bbd5-1ebf50e5ead8-cni-binary-copy\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:16.016917    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016935    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-run-k8s-cni-cncf-io\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckskc\" (UniqueName: \"kubernetes.io/projected/20a23223-f446-4941-9b20-6badf3aee2e3-kube-api-access-ckskc\") pod \"node-resolver-w4nwh\" (UID: \"20a23223-f446-4941-9b20-6badf3aee2e3\") " pod="openshift-dns/node-resolver-w4nwh"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65d13726-37f1-465a-bbd5-1ebf50e5ead8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.016967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.016975    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-multus-conf-dir\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:16.016983    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret podName:e92972bf-2b13-43de-bc6c-ccf50050fa24 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:16.51696515 +0000 UTC m=+3.140657473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret") pod "global-pull-secret-syncer-xbnhj" (UID: "e92972bf-2b13-43de-bc6c-ccf50050fa24") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-var-lib-cni-multus\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-var-lib-kubelet\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a23223-f446-4941-9b20-6badf3aee2e3-tmp-dir\") pod \"node-resolver-w4nwh\" (UID: \"20a23223-f446-4941-9b20-6badf3aee2e3\") " pod="openshift-dns/node-resolver-w4nwh"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp72d\" (UniqueName: \"kubernetes.io/projected/baadbf65-d286-4e72-90b9-abe37d115061-kube-api-access-wp72d\") pod \"iptables-alerter-m4j5s\" (UID: \"baadbf65-d286-4e72-90b9-abe37d115061\") " pod="openshift-network-operator/iptables-alerter-m4j5s"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw9gn\" (UniqueName: \"kubernetes.io/projected/d6b5446c-990e-467e-a319-150edf10bcc1-kube-api-access-kw9gn\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65d13726-37f1-465a-bbd5-1ebf50e5ead8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20a23223-f446-4941-9b20-6badf3aee2e3-hosts-file\") pod \"node-resolver-w4nwh\" (UID: \"20a23223-f446-4941-9b20-6badf3aee2e3\") " pod="openshift-dns/node-resolver-w4nwh"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/baadbf65-d286-4e72-90b9-abe37d115061-iptables-alerter-script\") pod \"iptables-alerter-m4j5s\" (UID: \"baadbf65-d286-4e72-90b9-abe37d115061\") " pod="openshift-network-operator/iptables-alerter-m4j5s"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-var-lib-cni-multus\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017354    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65d13726-37f1-465a-bbd5-1ebf50e5ead8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017366    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-var-lib-kubelet\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017427    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65d13726-37f1-465a-bbd5-1ebf50e5ead8-cni-binary-copy\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e92972bf-2b13-43de-bc6c-ccf50050fa24-kubelet-config\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017527    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20a23223-f446-4941-9b20-6badf3aee2e3-hosts-file\") pod \"node-resolver-w4nwh\" (UID: \"20a23223-f446-4941-9b20-6badf3aee2e3\") " pod="openshift-dns/node-resolver-w4nwh"
Sep 08 15:44:16.017813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65d13726-37f1-465a-bbd5-1ebf50e5ead8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-run-netns\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-hostroot\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65d13726-37f1-465a-bbd5-1ebf50e5ead8-system-cni-dir\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017613    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a23223-f446-4941-9b20-6badf3aee2e3-tmp-dir\") pod \"node-resolver-w4nwh\" (UID: \"20a23223-f446-4941-9b20-6badf3aee2e3\") " pod="openshift-dns/node-resolver-w4nwh"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-etc-kubernetes\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-run-netns\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/baadbf65-d286-4e72-90b9-abe37d115061-host-slash\") pod \"iptables-alerter-m4j5s\" (UID: \"baadbf65-d286-4e72-90b9-abe37d115061\") " pod="openshift-network-operator/iptables-alerter-m4j5s"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/baadbf65-d286-4e72-90b9-abe37d115061-host-slash\") pod \"iptables-alerter-m4j5s\" (UID: \"baadbf65-d286-4e72-90b9-abe37d115061\") " pod="openshift-network-operator/iptables-alerter-m4j5s"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e92972bf-2b13-43de-bc6c-ccf50050fa24-dbus\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017661    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-etc-kubernetes\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65d13726-37f1-465a-bbd5-1ebf50e5ead8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-multus-cni-dir\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-hostroot\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6b5446c-990e-467e-a319-150edf10bcc1-cni-binary-copy\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6b5446c-990e-467e-a319-150edf10bcc1-multus-daemon-config\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e92972bf-2b13-43de-bc6c-ccf50050fa24-dbus\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-multus-cni-dir\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.018458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-system-cni-dir\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.019214 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-system-cni-dir\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.019214 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-run-multus-certs\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.019214 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.017899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6b5446c-990e-467e-a319-150edf10bcc1-host-run-multus-certs\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.019214 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.018212    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6b5446c-990e-467e-a319-150edf10bcc1-cni-binary-copy\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.019214 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.018296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6b5446c-990e-467e-a319-150edf10bcc1-multus-daemon-config\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.029342 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.029285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54th5\" (UniqueName: \"kubernetes.io/projected/65d13726-37f1-465a-bbd5-1ebf50e5ead8-kube-api-access-54th5\") pod \"multus-additional-cni-plugins-ph9tx\" (UID: \"65d13726-37f1-465a-bbd5-1ebf50e5ead8\") " pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.029520 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.029495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckskc\" (UniqueName: \"kubernetes.io/projected/20a23223-f446-4941-9b20-6badf3aee2e3-kube-api-access-ckskc\") pod \"node-resolver-w4nwh\" (UID: \"20a23223-f446-4941-9b20-6badf3aee2e3\") " pod="openshift-dns/node-resolver-w4nwh"
Sep 08 15:44:16.029878 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.029860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp72d\" (UniqueName: \"kubernetes.io/projected/baadbf65-d286-4e72-90b9-abe37d115061-kube-api-access-wp72d\") pod \"iptables-alerter-m4j5s\" (UID: \"baadbf65-d286-4e72-90b9-abe37d115061\") " pod="openshift-network-operator/iptables-alerter-m4j5s"
Sep 08 15:44:16.030228 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.030207    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw9gn\" (UniqueName: \"kubernetes.io/projected/d6b5446c-990e-467e-a319-150edf10bcc1-kube-api-access-kw9gn\") pod \"multus-jbqc7\" (UID: \"d6b5446c-990e-467e-a319-150edf10bcc1\") " pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.100808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.100779    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-thsrk"
Sep 08 15:44:16.112304 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:16.112277    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed950cd_31e9_4a85_89ac_83818614ad83.slice/crio-5d151317f3852c9ff1d1a4db03ce8e4b6857da6845314887a4cd991404de6ec0 WatchSource:0}: Error finding container 5d151317f3852c9ff1d1a4db03ce8e4b6857da6845314887a4cd991404de6ec0: Status 404 returned error can't find the container with id 5d151317f3852c9ff1d1a4db03ce8e4b6857da6845314887a4cd991404de6ec0
Sep 08 15:44:16.116127 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.116106    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:16.121287 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.121262    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l6lrc"
Sep 08 15:44:16.127714 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.127673    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x"
Sep 08 15:44:16.128027 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:16.127999    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44035804_234e_4b1f_9f69_938469b3446f.slice/crio-d6b0ab8a7d53ac85d1886d5d6c81ec8df59fb2b2ea4918a54116696817b39c3b WatchSource:0}: Error finding container d6b0ab8a7d53ac85d1886d5d6c81ec8df59fb2b2ea4918a54116696817b39c3b: Status 404 returned error can't find the container with id d6b0ab8a7d53ac85d1886d5d6c81ec8df59fb2b2ea4918a54116696817b39c3b
Sep 08 15:44:16.131440 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:16.131412    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2201c616_9356_494d_915c_edee0306a6bf.slice/crio-49db7c2bf7e99e385c00f7933fbf29b513054bec3d9ee27557e82ce3c3d6b45b WatchSource:0}: Error finding container 49db7c2bf7e99e385c00f7933fbf29b513054bec3d9ee27557e82ce3c3d6b45b: Status 404 returned error can't find the container with id 49db7c2bf7e99e385c00f7933fbf29b513054bec3d9ee27557e82ce3c3d6b45b
Sep 08 15:44:16.137914 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.137890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qqmj7"
Sep 08 15:44:16.138016 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:16.137995    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfad9a3_ef15_4dcc_a831_9d55a6cd659c.slice/crio-e1afb6aeeb94d1d9da9849b1a40719bef55f9e6944fbfb632f968c7b3219dc3b WatchSource:0}: Error finding container e1afb6aeeb94d1d9da9849b1a40719bef55f9e6944fbfb632f968c7b3219dc3b: Status 404 returned error can't find the container with id e1afb6aeeb94d1d9da9849b1a40719bef55f9e6944fbfb632f968c7b3219dc3b
Sep 08 15:44:16.144958 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.144940    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ph9tx"
Sep 08 15:44:16.148416 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:16.148390    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3128409_f035_4e2d_bb79_8c4b77ebb807.slice/crio-a926403ee48b8aeb8ed7f82506c9eb1f189bca65cb3fa960e57ed4393e2d1068 WatchSource:0}: Error finding container a926403ee48b8aeb8ed7f82506c9eb1f189bca65cb3fa960e57ed4393e2d1068: Status 404 returned error can't find the container with id a926403ee48b8aeb8ed7f82506c9eb1f189bca65cb3fa960e57ed4393e2d1068
Sep 08 15:44:16.153416 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.153389    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbqc7"
Sep 08 15:44:16.153921 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:16.153902    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d13726_37f1_465a_bbd5_1ebf50e5ead8.slice/crio-0269f91cf4d74d125800d932bdd8f3697e640db83deceead3dac999659834e3c WatchSource:0}: Error finding container 0269f91cf4d74d125800d932bdd8f3697e640db83deceead3dac999659834e3c: Status 404 returned error can't find the container with id 0269f91cf4d74d125800d932bdd8f3697e640db83deceead3dac999659834e3c
Sep 08 15:44:16.163433 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.163405    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m4j5s"
Sep 08 15:44:16.170462 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.170444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w4nwh"
Sep 08 15:44:16.421212 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.421176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs\") pod \"network-metrics-daemon-nkn7w\" (UID: \"996123f7-ff5c-4ffd-8a4d-ebfb862d664e\") " pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:16.421379 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:16.421324    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:16.421442 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:16.421396    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs podName:996123f7-ff5c-4ffd-8a4d-ebfb862d664e nodeName:}" failed. No retries permitted until 2026-09-08 15:44:17.421375312 +0000 UTC m=+4.045067643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs") pod "network-metrics-daemon-nkn7w" (UID: "996123f7-ff5c-4ffd-8a4d-ebfb862d664e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:16.508379 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:16.508348    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b5446c_990e_467e_a319_150edf10bcc1.slice/crio-f14641dbbe1244cfd949216e5925734ab6df74d8bdc24bfed12a586fca35f1c4 WatchSource:0}: Error finding container f14641dbbe1244cfd949216e5925734ab6df74d8bdc24bfed12a586fca35f1c4: Status 404 returned error can't find the container with id f14641dbbe1244cfd949216e5925734ab6df74d8bdc24bfed12a586fca35f1c4
Sep 08 15:44:16.522008 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.521974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bxz\" (UniqueName: \"kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz\") pod \"network-check-target-8r2bg\" (UID: \"da5fbd85-bda7-46ab-98ea-255e07a0fb82\") " pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:16.522138 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.522017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:16.522138 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:16.522088    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:44:16.522138 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:16.522110    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:44:16.522138 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:16.522120    2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8bxz for pod openshift-network-diagnostics/network-check-target-8r2bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:16.522138 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:16.522127    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:16.522415 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:16.522166    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz podName:da5fbd85-bda7-46ab-98ea-255e07a0fb82 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:17.522151984 +0000 UTC m=+4.145844288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8bxz" (UniqueName: "kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz") pod "network-check-target-8r2bg" (UID: "da5fbd85-bda7-46ab-98ea-255e07a0fb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:16.522415 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:16.522182    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret podName:e92972bf-2b13-43de-bc6c-ccf50050fa24 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:17.522175724 +0000 UTC m=+4.145868026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret") pod "global-pull-secret-syncer-xbnhj" (UID: "e92972bf-2b13-43de-bc6c-ccf50050fa24") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:16.855938 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.855848    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 15:39:16 +0000 UTC" deadline="2028-06-21 06:02:34.97691093 +0000 UTC"
Sep 08 15:44:16.855938 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.855882    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15638h18m18.121033446s"
Sep 08 15:44:16.904318 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.904280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-thsrk" event={"ID":"4ed950cd-31e9-4a85-89ac-83818614ad83","Type":"ContainerStarted","Data":"5d151317f3852c9ff1d1a4db03ce8e4b6857da6845314887a4cd991404de6ec0"}
Sep 08 15:44:16.906070 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.906035    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qqmj7" event={"ID":"e3128409-f035-4e2d-bb79-8c4b77ebb807","Type":"ContainerStarted","Data":"a926403ee48b8aeb8ed7f82506c9eb1f189bca65cb3fa960e57ed4393e2d1068"}
Sep 08 15:44:16.908652 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.908622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l6lrc" event={"ID":"2201c616-9356-494d-915c-edee0306a6bf","Type":"ContainerStarted","Data":"49db7c2bf7e99e385c00f7933fbf29b513054bec3d9ee27557e82ce3c3d6b45b"}
Sep 08 15:44:16.910910 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.910511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbqc7" event={"ID":"d6b5446c-990e-467e-a319-150edf10bcc1","Type":"ContainerStarted","Data":"f14641dbbe1244cfd949216e5925734ab6df74d8bdc24bfed12a586fca35f1c4"}
Sep 08 15:44:16.911992 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:16.911969    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaadbf65_d286_4e72_90b9_abe37d115061.slice/crio-85615970a507fe9f71e2128c7e53158c213aadc2dce68d3f1c25a5c1c530a802 WatchSource:0}: Error finding container 85615970a507fe9f71e2128c7e53158c213aadc2dce68d3f1c25a5c1c530a802: Status 404 returned error can't find the container with id 85615970a507fe9f71e2128c7e53158c213aadc2dce68d3f1c25a5c1c530a802
Sep 08 15:44:16.912569 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.912536    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph9tx" event={"ID":"65d13726-37f1-465a-bbd5-1ebf50e5ead8","Type":"ContainerStarted","Data":"0269f91cf4d74d125800d932bdd8f3697e640db83deceead3dac999659834e3c"}
Sep 08 15:44:16.914488 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:16.914372    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a23223_f446_4941_9b20_6badf3aee2e3.slice/crio-04ced1e649a4d4bb25c96c4defe3714ef5110fa87bab342d407be7eb14d17b1a WatchSource:0}: Error finding container 04ced1e649a4d4bb25c96c4defe3714ef5110fa87bab342d407be7eb14d17b1a: Status 404 returned error can't find the container with id 04ced1e649a4d4bb25c96c4defe3714ef5110fa87bab342d407be7eb14d17b1a
Sep 08 15:44:16.917192 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.916035    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x" event={"ID":"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c","Type":"ContainerStarted","Data":"e1afb6aeeb94d1d9da9849b1a40719bef55f9e6944fbfb632f968c7b3219dc3b"}
Sep 08 15:44:16.920713 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:16.920686    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2" event={"ID":"44035804-234e-4b1f-9f69-938469b3446f","Type":"ContainerStarted","Data":"d6b0ab8a7d53ac85d1886d5d6c81ec8df59fb2b2ea4918a54116696817b39c3b"}
Sep 08 15:44:17.166455 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:17.166403    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:44:17.430237 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:17.429275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs\") pod \"network-metrics-daemon-nkn7w\" (UID: \"996123f7-ff5c-4ffd-8a4d-ebfb862d664e\") " pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:17.430237 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:17.429763    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:17.430237 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:17.429832    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs podName:996123f7-ff5c-4ffd-8a4d-ebfb862d664e nodeName:}" failed. No retries permitted until 2026-09-08 15:44:19.429812824 +0000 UTC m=+6.053505133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs") pod "network-metrics-daemon-nkn7w" (UID: "996123f7-ff5c-4ffd-8a4d-ebfb862d664e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:17.531445 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:17.530462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bxz\" (UniqueName: \"kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz\") pod \"network-check-target-8r2bg\" (UID: \"da5fbd85-bda7-46ab-98ea-255e07a0fb82\") " pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:17.531445 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:17.530516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:17.531445 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:17.530652    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:17.531445 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:17.530712    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret podName:e92972bf-2b13-43de-bc6c-ccf50050fa24 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:19.530693464 +0000 UTC m=+6.154385768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret") pod "global-pull-secret-syncer-xbnhj" (UID: "e92972bf-2b13-43de-bc6c-ccf50050fa24") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:17.531445 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:17.531175    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:44:17.531445 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:17.531199    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:44:17.531445 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:17.531218    2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8bxz for pod openshift-network-diagnostics/network-check-target-8r2bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:17.531445 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:17.531275    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz podName:da5fbd85-bda7-46ab-98ea-255e07a0fb82 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:19.531255009 +0000 UTC m=+6.154947316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8bxz" (UniqueName: "kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz") pod "network-check-target-8r2bg" (UID: "da5fbd85-bda7-46ab-98ea-255e07a0fb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:17.856721 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:17.856625    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 15:39:16 +0000 UTC" deadline="2028-03-06 03:07:55.016647192 +0000 UTC"
Sep 08 15:44:17.856721 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:17.856669    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13067h23m37.159982807s"
Sep 08 15:44:17.894690 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:17.894661    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:17.894857 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:17.894811    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:17.895275 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:17.895255    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:17.895386 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:17.895349    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:17.895459 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:17.895422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:17.898249 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:17.898165    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:17.931585 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:17.931557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m4j5s" event={"ID":"baadbf65-d286-4e72-90b9-abe37d115061","Type":"ContainerStarted","Data":"85615970a507fe9f71e2128c7e53158c213aadc2dce68d3f1c25a5c1c530a802"}
Sep 08 15:44:17.936629 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:17.936600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-107.ec2.internal" event={"ID":"5032d201ce1921952e1262cfc0d74711","Type":"ContainerStarted","Data":"ab9e2c13a155311ab6c71ddb8ec639041130b375bf3b8d67e823f95a82eccbfe"}
Sep 08 15:44:17.943358 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:17.943249    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w4nwh" event={"ID":"20a23223-f446-4941-9b20-6badf3aee2e3","Type":"ContainerStarted","Data":"04ced1e649a4d4bb25c96c4defe3714ef5110fa87bab342d407be7eb14d17b1a"}
Sep 08 15:44:19.453725 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:19.453682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs\") pod \"network-metrics-daemon-nkn7w\" (UID: \"996123f7-ff5c-4ffd-8a4d-ebfb862d664e\") " pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:19.454235 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:19.453917    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:19.454235 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:19.453983    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs podName:996123f7-ff5c-4ffd-8a4d-ebfb862d664e nodeName:}" failed. No retries permitted until 2026-09-08 15:44:23.45396284 +0000 UTC m=+10.077655148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs") pod "network-metrics-daemon-nkn7w" (UID: "996123f7-ff5c-4ffd-8a4d-ebfb862d664e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:19.555146 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:19.555099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bxz\" (UniqueName: \"kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz\") pod \"network-check-target-8r2bg\" (UID: \"da5fbd85-bda7-46ab-98ea-255e07a0fb82\") " pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:19.555375 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:19.555166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:19.555375 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:19.555270    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:19.555375 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:19.555286    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:44:19.555375 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:19.555304    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:44:19.555375 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:19.555317    2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8bxz for pod openshift-network-diagnostics/network-check-target-8r2bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:19.555375 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:19.555341    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret podName:e92972bf-2b13-43de-bc6c-ccf50050fa24 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:23.555322218 +0000 UTC m=+10.179014543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret") pod "global-pull-secret-syncer-xbnhj" (UID: "e92972bf-2b13-43de-bc6c-ccf50050fa24") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:19.555375 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:19.555360    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz podName:da5fbd85-bda7-46ab-98ea-255e07a0fb82 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:23.555351057 +0000 UTC m=+10.179043366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8bxz" (UniqueName: "kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz") pod "network-check-target-8r2bg" (UID: "da5fbd85-bda7-46ab-98ea-255e07a0fb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:19.900221 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:19.900190    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:19.900427 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:19.900337    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:19.900772 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:19.900749    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:19.900893 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:19.900860    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:19.901029 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:19.900926    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:19.901029 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:19.901002    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:21.894241 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:21.894188    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:21.894754 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:21.894330    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:21.894817 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:21.894764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:21.894901 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:21.894879    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:21.894949 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:21.894936    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:21.895053 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:21.895017    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:23.484909 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:23.484875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs\") pod \"network-metrics-daemon-nkn7w\" (UID: \"996123f7-ff5c-4ffd-8a4d-ebfb862d664e\") " pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:23.485416 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:23.485057    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:23.485416 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:23.485101    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs podName:996123f7-ff5c-4ffd-8a4d-ebfb862d664e nodeName:}" failed. No retries permitted until 2026-09-08 15:44:31.485088379 +0000 UTC m=+18.108780681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs") pod "network-metrics-daemon-nkn7w" (UID: "996123f7-ff5c-4ffd-8a4d-ebfb862d664e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:23.585559 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:23.585522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bxz\" (UniqueName: \"kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz\") pod \"network-check-target-8r2bg\" (UID: \"da5fbd85-bda7-46ab-98ea-255e07a0fb82\") " pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:23.585716 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:23.585568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:23.585716 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:23.585670    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:44:23.585716 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:23.585687    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:44:23.585716 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:23.585698    2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8bxz for pod openshift-network-diagnostics/network-check-target-8r2bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:23.585847 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:23.585737    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz podName:da5fbd85-bda7-46ab-98ea-255e07a0fb82 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:31.585724937 +0000 UTC m=+18.209417252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8bxz" (UniqueName: "kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz") pod "network-check-target-8r2bg" (UID: "da5fbd85-bda7-46ab-98ea-255e07a0fb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:23.585847 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:23.585733    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:23.585847 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:23.585800    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret podName:e92972bf-2b13-43de-bc6c-ccf50050fa24 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:31.585783035 +0000 UTC m=+18.209475343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret") pod "global-pull-secret-syncer-xbnhj" (UID: "e92972bf-2b13-43de-bc6c-ccf50050fa24") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:23.896936 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:23.896576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:23.896936 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:23.896595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:23.896936 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:23.896623    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:23.896936 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:23.896826    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:23.898027 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:23.897981    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:23.898169 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:23.898101    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:25.898846 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:25.898812    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:25.899322 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:25.898938    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:25.899322 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:25.898985    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:25.899322 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:25.899052    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:25.899322 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:25.899101    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:25.899322 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:25.899181    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:27.897761 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:27.897726    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:27.898196 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:27.897848    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:27.898196 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:27.897857    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:27.898196 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:27.897868    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:27.898196 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:27.897962    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:27.898196 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:27.898038    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:29.894893 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:29.894815    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:29.894893 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:29.894833    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:29.894893 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:29.894842    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:29.895465 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:29.894938    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:29.895465 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:29.895038    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:29.895465 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:29.895133    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:31.543692 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:31.543651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs\") pod \"network-metrics-daemon-nkn7w\" (UID: \"996123f7-ff5c-4ffd-8a4d-ebfb862d664e\") " pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:31.544304 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:31.543825    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:31.544304 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:31.543915    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs podName:996123f7-ff5c-4ffd-8a4d-ebfb862d664e nodeName:}" failed. No retries permitted until 2026-09-08 15:44:47.543885741 +0000 UTC m=+34.167578057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs") pod "network-metrics-daemon-nkn7w" (UID: "996123f7-ff5c-4ffd-8a4d-ebfb862d664e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:31.644125 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:31.644088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bxz\" (UniqueName: \"kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz\") pod \"network-check-target-8r2bg\" (UID: \"da5fbd85-bda7-46ab-98ea-255e07a0fb82\") " pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:31.644125 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:31.644130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:31.644358 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:31.644241    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:31.644358 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:31.644254    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:44:31.644358 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:31.644277    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:44:31.644358 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:31.644290    2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8bxz for pod openshift-network-diagnostics/network-check-target-8r2bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:31.644358 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:31.644313    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret podName:e92972bf-2b13-43de-bc6c-ccf50050fa24 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:47.644296131 +0000 UTC m=+34.267988433 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret") pod "global-pull-secret-syncer-xbnhj" (UID: "e92972bf-2b13-43de-bc6c-ccf50050fa24") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:31.644358 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:31.644331    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz podName:da5fbd85-bda7-46ab-98ea-255e07a0fb82 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:47.64432216 +0000 UTC m=+34.268014462 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8bxz" (UniqueName: "kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz") pod "network-check-target-8r2bg" (UID: "da5fbd85-bda7-46ab-98ea-255e07a0fb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:31.897767 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:31.897734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:31.897925 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:31.897815    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:31.897925 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:31.897903    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:31.898013 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:31.897922    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:31.898013 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:31.897997    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:31.898111 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:31.898068    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:33.897465 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:33.897432    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:33.897895 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:33.897478    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:33.897895 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:33.897478    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:33.897895 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:33.897550    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:33.898057 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:33.897907    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:33.898057 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:33.897957    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:34.994510 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:34.994288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2" event={"ID":"44035804-234e-4b1f-9f69-938469b3446f","Type":"ContainerStarted","Data":"0bc8c551350904fa0c8890427a3b322cf8fdf96ba47dd54845e351ca3e300c58"}
Sep 08 15:44:34.995173 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:34.994585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2" event={"ID":"44035804-234e-4b1f-9f69-938469b3446f","Type":"ContainerStarted","Data":"09a919e29acc9e63de7230a43e4bf3d533a5ce050a293fa73d106184f19a1e99"}
Sep 08 15:44:34.998187 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:34.998156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qqmj7" event={"ID":"e3128409-f035-4e2d-bb79-8c4b77ebb807","Type":"ContainerStarted","Data":"44faf3ff2d0fbbd9d88a0d1cf270bd2e1ac400ac0dec42e121593502ccae8311"}
Sep 08 15:44:35.001919 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:35.001888    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbqc7" event={"ID":"d6b5446c-990e-467e-a319-150edf10bcc1","Type":"ContainerStarted","Data":"1f61620bdbebf14b5924c4b4270d25cb0a8adf5b1a295713887a7d8aae598e4a"}
Sep 08 15:44:35.017013 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:35.016948    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-107.ec2.internal" podStartSLOduration=20.016927325 podStartE2EDuration="20.016927325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:17.954445984 +0000 UTC m=+4.578138303" watchObservedRunningTime="2026-09-08 15:44:35.016927325 +0000 UTC m=+21.640619649"
Sep 08 15:44:35.017117 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:35.017055    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qqmj7" podStartSLOduration=1.428783333 podStartE2EDuration="20.017043531s" totalImagesPullingTime="18.588260198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:35.015863684 +0000 UTC m=+21.639556009" watchObservedRunningTime="2026-09-08 15:44:35.017043531 +0000 UTC m=+21.640735856"
Sep 08 15:44:35.896979 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:35.896819    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:35.897124 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:35.896870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:35.897124 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:35.897046    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:35.897124 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:35.896876    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:35.897285 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:35.897120    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:35.897285 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:35.897182    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:36.005556 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.005471    2581 generic.go:356] "Generic (PLEG): container finished" podID="756114e43effe75943fb30ab8ace2af1" containerID="30ca3fad844191763b965b066f329b5dafcfb580fe09ae8761d82dfa0aa01c17" exitCode=0
Sep 08 15:44:36.006045 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.005555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal" event={"ID":"756114e43effe75943fb30ab8ace2af1","Type":"ContainerDied","Data":"30ca3fad844191763b965b066f329b5dafcfb580fe09ae8761d82dfa0aa01c17"}
Sep 08 15:44:36.007328 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.007307    2581 generic.go:356] "Generic (PLEG): container finished" podID="65d13726-37f1-465a-bbd5-1ebf50e5ead8" containerID="14973484d7e9b25a3bcc43eb92b980ba151bc404bd2d47c656fb85abb5ae8dd6" exitCode=0
Sep 08 15:44:36.007432 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.007378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph9tx" event={"ID":"65d13726-37f1-465a-bbd5-1ebf50e5ead8","Type":"ContainerDied","Data":"14973484d7e9b25a3bcc43eb92b980ba151bc404bd2d47c656fb85abb5ae8dd6"}
Sep 08 15:44:36.008993 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.008972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x" event={"ID":"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c","Type":"ContainerStarted","Data":"e7585c7759eec3ca2de9c9bcc5225521679aa2990ce1d0cdddec2c57caa8d48a"}
Sep 08 15:44:36.011857 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.011838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2" event={"ID":"44035804-234e-4b1f-9f69-938469b3446f","Type":"ContainerStarted","Data":"4ede16d505c444ce35d3cbf8e70227dedbe32b6fd92404f7c068b7745d4cbf11"}
Sep 08 15:44:36.011924 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.011897    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2" event={"ID":"44035804-234e-4b1f-9f69-938469b3446f","Type":"ContainerStarted","Data":"1d70072039d9ea1ca8a0478034cb19ad0989fd6c24aa0b5abdbefb4fbd9e8e8e"}
Sep 08 15:44:36.011975 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.011938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2" event={"ID":"44035804-234e-4b1f-9f69-938469b3446f","Type":"ContainerStarted","Data":"c4254b10ee32c86067efc56c307dc2a8c533dfdd783fbbde19485e51f8c17b06"}
Sep 08 15:44:36.012020 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.011975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2" event={"ID":"44035804-234e-4b1f-9f69-938469b3446f","Type":"ContainerStarted","Data":"b98b05f6590640c13f970c60b7f9e0e91a7b075a4040a97cb788090cca7fc4dc"}
Sep 08 15:44:36.013152 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.013127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-thsrk" event={"ID":"4ed950cd-31e9-4a85-89ac-83818614ad83","Type":"ContainerStarted","Data":"34c4cc238f9dd6ba7f83ef3668087b78ceaacf351db83282915ba3d6308be681"}
Sep 08 15:44:36.014379 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.014358    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l6lrc" event={"ID":"2201c616-9356-494d-915c-edee0306a6bf","Type":"ContainerStarted","Data":"5ee0d15412a6ddb135e0234da1e88fef7cb88c8b2a3d862c03de169e9f8f0d92"}
Sep 08 15:44:36.015543 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.015522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w4nwh" event={"ID":"20a23223-f446-4941-9b20-6badf3aee2e3","Type":"ContainerStarted","Data":"7b6f0d49a43a1048b53d4b94c8801b23a0477d96fd569d83775bb060f73006d4"}
Sep 08 15:44:36.020255 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.020218    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jbqc7" podStartSLOduration=2.750976798 podStartE2EDuration="21.020206855s" totalImagesPullingTime="18.269230057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:35.032051087 +0000 UTC m=+21.655743439" watchObservedRunningTime="2026-09-08 15:44:36.020206855 +0000 UTC m=+22.643899179"
Sep 08 15:44:36.034770 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.034735    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-thsrk" podStartSLOduration=3.431422428 podStartE2EDuration="22.034723617s" totalImagesPullingTime="18.603301189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:36.034543301 +0000 UTC m=+22.658235625" watchObservedRunningTime="2026-09-08 15:44:36.034723617 +0000 UTC m=+22.658415981"
Sep 08 15:44:36.075164 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.075104    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l6lrc" podStartSLOduration=3.491845491 podStartE2EDuration="22.075090013s" totalImagesPullingTime="18.583244522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:36.074491446 +0000 UTC m=+22.698183782" watchObservedRunningTime="2026-09-08 15:44:36.075090013 +0000 UTC m=+22.698782337"
Sep 08 15:44:36.088544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:36.088505    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w4nwh" podStartSLOduration=3.275578775 podStartE2EDuration="21.08849257s" totalImagesPullingTime="17.812913795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:36.088170461 +0000 UTC m=+22.711862786" watchObservedRunningTime="2026-09-08 15:44:36.08849257 +0000 UTC m=+22.712184893"
Sep 08 15:44:37.019203 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:37.019169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal" event={"ID":"756114e43effe75943fb30ab8ace2af1","Type":"ContainerStarted","Data":"f561f736dd8e2461aa7d6e5ef2645592a4f304da05581a8ce60cc65dc09a4dc6"}
Sep 08 15:44:37.020594 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:37.020568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m4j5s" event={"ID":"baadbf65-d286-4e72-90b9-abe37d115061","Type":"ContainerStarted","Data":"0958e731e8058e63278211f4af0797f7119b3fc8eecb7d44185b123588daf9e6"}
Sep 08 15:44:37.035702 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:37.035653    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-107.ec2.internal" podStartSLOduration=22.035640249 podStartE2EDuration="22.035640249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:37.035641266 +0000 UTC m=+23.659333590" watchObservedRunningTime="2026-09-08 15:44:37.035640249 +0000 UTC m=+23.659332574"
Sep 08 15:44:37.049969 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:37.049925    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m4j5s" podStartSLOduration=4.249019142 podStartE2EDuration="22.049914128s" totalImagesPullingTime="17.800894986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:37.049778997 +0000 UTC m=+23.673471321" watchObservedRunningTime="2026-09-08 15:44:37.049914128 +0000 UTC m=+23.673606451"
Sep 08 15:44:37.251634 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:37.251605    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-thsrk"
Sep 08 15:44:37.252203 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:37.252174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-thsrk"
Sep 08 15:44:37.894496 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:37.894458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:37.894678 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:37.894458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:37.894678 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:37.894569    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:37.894678 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:37.894564    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:37.894678 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:37.894664    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:37.894868 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:37.894747    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:38.026597 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:38.026553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2" event={"ID":"44035804-234e-4b1f-9f69-938469b3446f","Type":"ContainerStarted","Data":"4816b0a928d7265b2f8517ea21e8bad8f9441082f69270bbc1104c792472b78b"}
Sep 08 15:44:38.027095 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:38.026954    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-thsrk"
Sep 08 15:44:38.030623 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:38.030599    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-thsrk"
Sep 08 15:44:38.512296 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:38.512136    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 15:44:38.861823 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:38.861673    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T15:44:38.512290923Z","UUID":"9d377af7-8189-4a75-9d96-b083796dd89e","Handler":null,"Name":"","Endpoint":""}
Sep 08 15:44:38.863109 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:38.863091    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 15:44:38.863233 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:38.863117    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 15:44:39.029892 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:39.029858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x" event={"ID":"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c","Type":"ContainerStarted","Data":"cceb9975fa43ef116b906d0d8e05a45f7b004db42ac4634097917a93ab8b3619"}
Sep 08 15:44:39.894710 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:39.894683    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:39.894880 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:39.894689    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:39.894880 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:39.894782    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:39.896195 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:39.894689    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:39.896195 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:39.894875    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:39.896195 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:39.895009    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:40.036299 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:40.036265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2" event={"ID":"44035804-234e-4b1f-9f69-938469b3446f","Type":"ContainerStarted","Data":"7b9d9f6e6a2988ed7a4f9d0d36d2f4c69ef23852282642aa97f7268e6cb167e3"}
Sep 08 15:44:40.036782 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:40.036602    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:40.036782 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:40.036668    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:40.036782 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:40.036714    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:40.052901 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:40.052876    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:40.053088 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:40.053071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:44:40.069025 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:40.068968    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2" podStartSLOduration=7.418912908 podStartE2EDuration="26.068950328s" totalImagesPullingTime="18.65003742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:40.067511853 +0000 UTC m=+26.691204177" watchObservedRunningTime="2026-09-08 15:44:40.068950328 +0000 UTC m=+26.692642676"
Sep 08 15:44:41.495534 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:41.495291    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xbnhj"]
Sep 08 15:44:41.495953 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:41.495683    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:41.495953 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:41.495820    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:41.498007 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:41.497974    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nkn7w"]
Sep 08 15:44:41.498136 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:41.498074    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:41.498198 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:41.498166    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:41.507069 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:41.507047    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8r2bg"]
Sep 08 15:44:41.507173 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:41.507139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:41.507266 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:41.507248    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:42.041987 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:42.041952    2581 generic.go:356] "Generic (PLEG): container finished" podID="65d13726-37f1-465a-bbd5-1ebf50e5ead8" containerID="a5f572d018de430d44b0bfac50a89cdb456d8c2dc713e5165552f96d59fb23ac" exitCode=0
Sep 08 15:44:42.042162 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:42.042032    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph9tx" event={"ID":"65d13726-37f1-465a-bbd5-1ebf50e5ead8","Type":"ContainerDied","Data":"a5f572d018de430d44b0bfac50a89cdb456d8c2dc713e5165552f96d59fb23ac"}
Sep 08 15:44:42.894739 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:42.894716    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:42.895219 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:42.894817    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:42.895219 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:42.894869    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:42.895219 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:42.894954    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:43.046791 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:43.046758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x" event={"ID":"ebfad9a3-ef15-4dcc-a831-9d55a6cd659c","Type":"ContainerStarted","Data":"b4fe0f4de44566ce390cbe46f2964d221cc3319cccef109413588335c7ff77fa"}
Sep 08 15:44:43.066571 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:43.066524    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptd7x" podStartSLOduration=2.256585474 podStartE2EDuration="28.066511593s" totalImagesPullingTime="25.809926119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:44:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:43.066090925 +0000 UTC m=+29.689783248" watchObservedRunningTime="2026-09-08 15:44:43.066511593 +0000 UTC m=+29.690203908"
Sep 08 15:44:43.894843 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:43.894817    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:43.895334 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:43.894895    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:44.051063 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:44.050993    2581 generic.go:356] "Generic (PLEG): container finished" podID="65d13726-37f1-465a-bbd5-1ebf50e5ead8" containerID="dffa268676f178412ded71b5d499fc2d2ddfc072e17585b13ac29720b768a54e" exitCode=0
Sep 08 15:44:44.051176 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:44.051075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph9tx" event={"ID":"65d13726-37f1-465a-bbd5-1ebf50e5ead8","Type":"ContainerDied","Data":"dffa268676f178412ded71b5d499fc2d2ddfc072e17585b13ac29720b768a54e"}
Sep 08 15:44:44.895241 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:44.895079    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:44.895714 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:44.895136    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:44.895714 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:44.895316    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:44.895714 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:44.895377    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:45.896845 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:45.896822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:45.897158 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:45.896914    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkn7w" podUID="996123f7-ff5c-4ffd-8a4d-ebfb862d664e"
Sep 08 15:44:46.056795 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:46.056761    2581 generic.go:356] "Generic (PLEG): container finished" podID="65d13726-37f1-465a-bbd5-1ebf50e5ead8" containerID="cf58b4ab60f512fcef9ad7e447110d850b02c0c7387c89aece50194b83dba153" exitCode=0
Sep 08 15:44:46.056914 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:46.056808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph9tx" event={"ID":"65d13726-37f1-465a-bbd5-1ebf50e5ead8","Type":"ContainerDied","Data":"cf58b4ab60f512fcef9ad7e447110d850b02c0c7387c89aece50194b83dba153"}
Sep 08 15:44:46.894921 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:46.894889    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:46.895082 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:46.894889    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:46.895082 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:46.895010    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r2bg" podUID="da5fbd85-bda7-46ab-98ea-255e07a0fb82"
Sep 08 15:44:46.895082 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:46.895069    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbnhj" podUID="e92972bf-2b13-43de-bc6c-ccf50050fa24"
Sep 08 15:44:47.565505 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.565423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs\") pod \"network-metrics-daemon-nkn7w\" (UID: \"996123f7-ff5c-4ffd-8a4d-ebfb862d664e\") " pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:47.566036 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:47.565552    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:47.566036 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:47.565626    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs podName:996123f7-ff5c-4ffd-8a4d-ebfb862d664e nodeName:}" failed. No retries permitted until 2026-09-08 15:45:19.565603705 +0000 UTC m=+66.189296012 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs") pod "network-metrics-daemon-nkn7w" (UID: "996123f7-ff5c-4ffd-8a4d-ebfb862d664e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:44:47.666303 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.666274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bxz\" (UniqueName: \"kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz\") pod \"network-check-target-8r2bg\" (UID: \"da5fbd85-bda7-46ab-98ea-255e07a0fb82\") " pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:47.666303 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.666308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:47.666527 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:47.666435    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:47.666527 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:47.666446    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:44:47.666527 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:47.666461    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:44:47.666527 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:47.666473    2581 projected.go:196] Error preparing data for projected volume kube-api-access-k8bxz for pod openshift-network-diagnostics/network-check-target-8r2bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:47.666527 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:47.666496    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret podName:e92972bf-2b13-43de-bc6c-ccf50050fa24 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:19.666478617 +0000 UTC m=+66.290170943 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret") pod "global-pull-secret-syncer-xbnhj" (UID: "e92972bf-2b13-43de-bc6c-ccf50050fa24") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:44:47.666527 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:47.666512    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz podName:da5fbd85-bda7-46ab-98ea-255e07a0fb82 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:19.666504202 +0000 UTC m=+66.290196505 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k8bxz" (UniqueName: "kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz") pod "network-check-target-8r2bg" (UID: "da5fbd85-bda7-46ab-98ea-255e07a0fb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:44:47.728205 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.728179    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-107.ec2.internal" event="NodeReady"
Sep 08 15:44:47.728350 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.728342    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 15:44:47.761941 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.761915    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-z67qz"]
Sep 08 15:44:47.783459 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.783203    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"]
Sep 08 15:44:47.784178 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.784156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-z67qz"
Sep 08 15:44:47.787354 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.787331    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:44:47.787515 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.787474    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-j6mtd\""
Sep 08 15:44:47.788120 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.788012    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 08 15:44:47.802236 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.802109    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74f5f6df75-6fqds"]
Sep 08 15:44:47.804369 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.804347    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"
Sep 08 15:44:47.806488 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.806468    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 08 15:44:47.806595 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.806484    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:44:47.806752 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.806734    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 08 15:44:47.806849 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.806784    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 08 15:44:47.806849 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.806825    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wlzx9\""
Sep 08 15:44:47.820219 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.820174    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-fzq5z"]
Sep 08 15:44:47.820447 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.820331    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:47.822864 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.822842    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 15:44:47.823091 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.823073    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 15:44:47.823177 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.823114    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 15:44:47.823357 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.823345    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6lw5f\""
Sep 08 15:44:47.829305 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.829278    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 15:44:47.844367 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.844341    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"]
Sep 08 15:44:47.844460 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.844436    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:47.846634 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.846611    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:44:47.846748 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.846731    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 08 15:44:47.846872 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.846853    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dq5vc\""
Sep 08 15:44:47.847262 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.847053    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 08 15:44:47.847262 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.847139    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 08 15:44:47.852997 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.852982    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 08 15:44:47.868563 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.868543    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"]
Sep 08 15:44:47.868685 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.868669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"
Sep 08 15:44:47.870878 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.870858    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 08 15:44:47.870966 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.870915    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:44:47.870966 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.870927    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 08 15:44:47.871142 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.871111    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 08 15:44:47.871266 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.871195    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2zpbn\""
Sep 08 15:44:47.884717 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.884696    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"]
Sep 08 15:44:47.885879 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.884840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:44:47.887191 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.887170    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 08 15:44:47.887861 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.887845    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 08 15:44:47.888474 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.888456    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:44:47.888723 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.888698    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p7plt\""
Sep 08 15:44:47.896797 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.896780    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:44:47.899275 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.899257    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 08 15:44:47.899371 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.899277    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 15:44:47.899371 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.899283    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-d8bb6\""
Sep 08 15:44:47.899508 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.899376    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 08 15:44:47.899639 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.899621    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 15:44:47.901012 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.900978    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-nxt9p"]
Sep 08 15:44:47.901113 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.901097    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:44:47.903034 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.903015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 15:44:47.903115 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.903075    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-78hmt\""
Sep 08 15:44:47.914886 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.914867    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5f6c69889d-c7hpw"]
Sep 08 15:44:47.914995 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.914982    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:47.917412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.917251    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 15:44:47.917412 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.917251    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 15:44:47.917523 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.917414    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 08 15:44:47.917523 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.917481    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 08 15:44:47.917607 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.917589    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-x5nn4\""
Sep 08 15:44:47.923336 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.923315    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 08 15:44:47.924922 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.924905    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-z67qz"]
Sep 08 15:44:47.925003 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.924938    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"]
Sep 08 15:44:47.925003 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.924958    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"]
Sep 08 15:44:47.925003 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.924990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:47.925003 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.924999    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"]
Sep 08 15:44:47.925202 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.925023    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-62rfg"]
Sep 08 15:44:47.926965 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.926946    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 08 15:44:47.927291 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.927272    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 08 15:44:47.927435 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.927418    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 15:44:47.927533 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.927444    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 08 15:44:47.927663 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.927422    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 15:44:47.927663 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.927282    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 08 15:44:47.927783 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.927368    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-l2fz9\""
Sep 08 15:44:47.941110 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.941089    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-fzq5z"]
Sep 08 15:44:47.941200 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.941120    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-ztgpn"]
Sep 08 15:44:47.941262 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.941198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62rfg"
Sep 08 15:44:47.943271 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.943249    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 15:44:47.943355 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.943285    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 15:44:47.947346 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.943641    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 15:44:47.947346 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.943712    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z2m58\""
Sep 08 15:44:47.955050 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.955032    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"]
Sep 08 15:44:47.955146 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.955065    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-nxt9p"]
Sep 08 15:44:47.955146 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.955085    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f6c69889d-c7hpw"]
Sep 08 15:44:47.955146 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.955103    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-ztgpn"]
Sep 08 15:44:47.955146 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.955134    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62rfg"]
Sep 08 15:44:47.955146 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.955139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-ztgpn"
Sep 08 15:44:47.955380 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.955152    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74f5f6df75-6fqds"]
Sep 08 15:44:47.955380 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.955188    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6vlx8"]
Sep 08 15:44:47.957138 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.957121    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cn9hv\""
Sep 08 15:44:47.957217 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.957149    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 15:44:47.957217 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.957120    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 15:44:47.967323 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a27cac25-7d24-41e2-bb79-a2412491978b-serving-cert\") pod \"console-operator-85b454874b-fzq5z\" (UID: \"a27cac25-7d24-41e2-bb79-a2412491978b\") " pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:47.967444 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v7f9\" (UniqueName: \"kubernetes.io/projected/593f6dab-17b1-48a1-a4bb-a9164f694d90-kube-api-access-5v7f9\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vdxpx\" (UID: \"593f6dab-17b1-48a1-a4bb-a9164f694d90\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"
Sep 08 15:44:47.967444 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38cac5cb-e76e-4799-a54c-eaf3158d7158-trusted-ca\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:47.967444 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a27cac25-7d24-41e2-bb79-a2412491978b-config\") pod \"console-operator-85b454874b-fzq5z\" (UID: \"a27cac25-7d24-41e2-bb79-a2412491978b\") " pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:47.967607 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/38cac5cb-e76e-4799-a54c-eaf3158d7158-installation-pull-secrets\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:47.967607 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5qqc\" (UniqueName: \"kubernetes.io/projected/c30ad658-852c-412f-af05-83940b0551b2-kube-api-access-g5qqc\") pod \"volume-data-source-validator-695c895bd6-z67qz\" (UID: \"c30ad658-852c-412f-af05-83940b0551b2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-z67qz"
Sep 08 15:44:47.967607 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62fvl\" (UniqueName: \"kubernetes.io/projected/a27cac25-7d24-41e2-bb79-a2412491978b-kube-api-access-62fvl\") pod \"console-operator-85b454874b-fzq5z\" (UID: \"a27cac25-7d24-41e2-bb79-a2412491978b\") " pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:47.967757 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-bound-sa-token\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:47.967757 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/593f6dab-17b1-48a1-a4bb-a9164f694d90-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vdxpx\" (UID: \"593f6dab-17b1-48a1-a4bb-a9164f694d90\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"
Sep 08 15:44:47.967757 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/38cac5cb-e76e-4799-a54c-eaf3158d7158-ca-trust-extracted\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:47.967757 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-certificates\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:47.967947 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d818a8bd-49ec-4d08-9602-78e859ddf43e-serving-cert\") pod \"service-ca-operator-859f94c4b-xk8g2\" (UID: \"d818a8bd-49ec-4d08-9602-78e859ddf43e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"
Sep 08 15:44:47.967947 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967804    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/593f6dab-17b1-48a1-a4bb-a9164f694d90-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vdxpx\" (UID: \"593f6dab-17b1-48a1-a4bb-a9164f694d90\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"
Sep 08 15:44:47.967947 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a27cac25-7d24-41e2-bb79-a2412491978b-trusted-ca\") pod \"console-operator-85b454874b-fzq5z\" (UID: \"a27cac25-7d24-41e2-bb79-a2412491978b\") " pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:47.967947 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:47.968134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.967949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d818a8bd-49ec-4d08-9602-78e859ddf43e-config\") pod \"service-ca-operator-859f94c4b-xk8g2\" (UID: \"d818a8bd-49ec-4d08-9602-78e859ddf43e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"
Sep 08 15:44:47.968134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.968035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/38cac5cb-e76e-4799-a54c-eaf3158d7158-image-registry-private-configuration\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:47.968134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.968069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq4qg\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-kube-api-access-cq4qg\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:47.968134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.968098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm2fp\" (UniqueName: \"kubernetes.io/projected/d818a8bd-49ec-4d08-9602-78e859ddf43e-kube-api-access-wm2fp\") pod \"service-ca-operator-859f94c4b-xk8g2\" (UID: \"d818a8bd-49ec-4d08-9602-78e859ddf43e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"
Sep 08 15:44:47.969220 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.969201    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6vlx8"]
Sep 08 15:44:47.969373 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.969354    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:47.971544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.971527    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 15:44:47.971635 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.971572    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 15:44:47.971695 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:47.971642    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qhh8b\""
Sep 08 15:44:48.031452 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.031417    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:48.068599 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/38cac5cb-e76e-4799-a54c-eaf3158d7158-image-registry-private-configuration\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.068705 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2fp\" (UniqueName: \"kubernetes.io/projected/d818a8bd-49ec-4d08-9602-78e859ddf43e-kube-api-access-wm2fp\") pod \"service-ca-operator-859f94c4b-xk8g2\" (UID: \"d818a8bd-49ec-4d08-9602-78e859ddf43e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"
Sep 08 15:44:48.068705 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6a25233-dc78-40bb-9a74-25f3aae52b3c-serving-cert\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.068705 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38cac5cb-e76e-4799-a54c-eaf3158d7158-trusted-ca\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.068705 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvvz6\" (UniqueName: \"kubernetes.io/projected/b43463d7-de74-46e8-af40-2db7ab47f8ff-kube-api-access-dvvz6\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:48.068705 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/38cac5cb-e76e-4799-a54c-eaf3158d7158-installation-pull-secrets\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.068907 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b43463d7-de74-46e8-af40-2db7ab47f8ff-config-volume\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:48.068907 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-bound-sa-token\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.068907 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2mv5\" (UniqueName: \"kubernetes.io/projected/280e0b1c-6fa9-4d0a-8769-8589179684a3-kube-api-access-d2mv5\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:44:48.068907 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mb5v\" (UniqueName: \"kubernetes.io/projected/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-kube-api-access-2mb5v\") pod \"cluster-samples-operator-7cdcfcf76c-hckv7\" (UID: \"8c9f9660-e1c4-4433-a181-e8eee81bbe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:44:48.068907 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/38cac5cb-e76e-4799-a54c-eaf3158d7158-ca-trust-extracted\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.069139 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a27cac25-7d24-41e2-bb79-a2412491978b-trusted-ca\") pod \"console-operator-85b454874b-fzq5z\" (UID: \"a27cac25-7d24-41e2-bb79-a2412491978b\") " pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:48.069139 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/280e0b1c-6fa9-4d0a-8769-8589179684a3-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:44:48.069139 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.068991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6a25233-dc78-40bb-9a74-25f3aae52b3c-tmp\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.069139 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-certificates\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.069139 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d818a8bd-49ec-4d08-9602-78e859ddf43e-serving-cert\") pod \"service-ca-operator-859f94c4b-xk8g2\" (UID: \"d818a8bd-49ec-4d08-9602-78e859ddf43e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"
Sep 08 15:44:48.069139 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b6a25233-dc78-40bb-9a74-25f3aae52b3c-snapshots\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.069139 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmmgf\" (UniqueName: \"kubernetes.io/projected/b3c368f6-fe5b-4cd8-8d03-0c75a4478a19-kube-api-access-bmmgf\") pod \"ingress-canary-62rfg\" (UID: \"b3c368f6-fe5b-4cd8-8d03-0c75a4478a19\") " pod="openshift-ingress-canary/ingress-canary-62rfg"
Sep 08 15:44:48.069490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d818a8bd-49ec-4d08-9602-78e859ddf43e-config\") pod \"service-ca-operator-859f94c4b-xk8g2\" (UID: \"d818a8bd-49ec-4d08-9602-78e859ddf43e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"
Sep 08 15:44:48.069490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq4qg\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-kube-api-access-cq4qg\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.069490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a27cac25-7d24-41e2-bb79-a2412491978b-serving-cert\") pod \"console-operator-85b454874b-fzq5z\" (UID: \"a27cac25-7d24-41e2-bb79-a2412491978b\") " pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:48.069490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.069490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:48.069490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/38cac5cb-e76e-4799-a54c-eaf3158d7158-ca-trust-extracted\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.069490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v7f9\" (UniqueName: \"kubernetes.io/projected/593f6dab-17b1-48a1-a4bb-a9164f694d90-kube-api-access-5v7f9\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vdxpx\" (UID: \"593f6dab-17b1-48a1-a4bb-a9164f694d90\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"
Sep 08 15:44:48.069490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a27cac25-7d24-41e2-bb79-a2412491978b-config\") pod \"console-operator-85b454874b-fzq5z\" (UID: \"a27cac25-7d24-41e2-bb79-a2412491978b\") " pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:48.069490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.069490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9x2h\" (UniqueName: \"kubernetes.io/projected/b6a25233-dc78-40bb-9a74-25f3aae52b3c-kube-api-access-z9x2h\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.069490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-default-certificate\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.069490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62fvl\" (UniqueName: \"kubernetes.io/projected/a27cac25-7d24-41e2-bb79-a2412491978b-kube-api-access-62fvl\") pod \"console-operator-85b454874b-fzq5z\" (UID: \"a27cac25-7d24-41e2-bb79-a2412491978b\") " pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:48.069490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5qqc\" (UniqueName: \"kubernetes.io/projected/c30ad658-852c-412f-af05-83940b0551b2-kube-api-access-g5qqc\") pod \"volume-data-source-validator-695c895bd6-z67qz\" (UID: \"c30ad658-852c-412f-af05-83940b0551b2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-z67qz"
Sep 08 15:44:48.070126 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hckv7\" (UID: \"8c9f9660-e1c4-4433-a181-e8eee81bbe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:44:48.070126 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a25233-dc78-40bb-9a74-25f3aae52b3c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.070126 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a25233-dc78-40bb-9a74-25f3aae52b3c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.070126 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/593f6dab-17b1-48a1-a4bb-a9164f694d90-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vdxpx\" (UID: \"593f6dab-17b1-48a1-a4bb-a9164f694d90\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"
Sep 08 15:44:48.070126 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:44:48.070126 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-stats-auth\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.070126 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/593f6dab-17b1-48a1-a4bb-a9164f694d90-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vdxpx\" (UID: \"593f6dab-17b1-48a1-a4bb-a9164f694d90\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"
Sep 08 15:44:48.070126 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3c368f6-fe5b-4cd8-8d03-0c75a4478a19-cert\") pod \"ingress-canary-62rfg\" (UID: \"b3c368f6-fe5b-4cd8-8d03-0c75a4478a19\") " pod="openshift-ingress-canary/ingress-canary-62rfg"
Sep 08 15:44:48.070126 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d818a8bd-49ec-4d08-9602-78e859ddf43e-config\") pod \"service-ca-operator-859f94c4b-xk8g2\" (UID: \"d818a8bd-49ec-4d08-9602-78e859ddf43e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"
Sep 08 15:44:48.070126 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8ctn\" (UniqueName: \"kubernetes.io/projected/73240d9c-1436-48bd-90b2-a665d22d7006-kube-api-access-t8ctn\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.070126 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a27cac25-7d24-41e2-bb79-a2412491978b-trusted-ca\") pod \"console-operator-85b454874b-fzq5z\" (UID: \"a27cac25-7d24-41e2-bb79-a2412491978b\") " pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:48.070126 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcxhx\" (UniqueName: \"kubernetes.io/projected/8e8e89d2-9be9-4976-84ce-6abf114256f0-kube-api-access-bcxhx\") pod \"network-check-source-685456655f-ztgpn\" (UID: \"8e8e89d2-9be9-4976-84ce-6abf114256f0\") " pod="openshift-network-diagnostics/network-check-source-685456655f-ztgpn"
Sep 08 15:44:48.070126 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.069876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38cac5cb-e76e-4799-a54c-eaf3158d7158-trusted-ca\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.070764 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.070179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a27cac25-7d24-41e2-bb79-a2412491978b-config\") pod \"console-operator-85b454874b-fzq5z\" (UID: \"a27cac25-7d24-41e2-bb79-a2412491978b\") " pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:48.070764 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.070315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-certificates\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.070764 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.070354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43463d7-de74-46e8-af40-2db7ab47f8ff-tmp-dir\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:48.070764 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.070414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.070764 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.070421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/593f6dab-17b1-48a1-a4bb-a9164f694d90-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vdxpx\" (UID: \"593f6dab-17b1-48a1-a4bb-a9164f694d90\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"
Sep 08 15:44:48.070764 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.070711    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:44:48.070764 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.070726    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5f6df75-6fqds: secret "image-registry-tls" not found
Sep 08 15:44:48.071086 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.070794    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls podName:38cac5cb-e76e-4799-a54c-eaf3158d7158 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:48.570777322 +0000 UTC m=+35.194469624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls") pod "image-registry-74f5f6df75-6fqds" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158") : secret "image-registry-tls" not found
Sep 08 15:44:48.073104 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.073084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a27cac25-7d24-41e2-bb79-a2412491978b-serving-cert\") pod \"console-operator-85b454874b-fzq5z\" (UID: \"a27cac25-7d24-41e2-bb79-a2412491978b\") " pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:48.073187 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.073110    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d818a8bd-49ec-4d08-9602-78e859ddf43e-serving-cert\") pod \"service-ca-operator-859f94c4b-xk8g2\" (UID: \"d818a8bd-49ec-4d08-9602-78e859ddf43e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"
Sep 08 15:44:48.073187 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.073161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/593f6dab-17b1-48a1-a4bb-a9164f694d90-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vdxpx\" (UID: \"593f6dab-17b1-48a1-a4bb-a9164f694d90\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"
Sep 08 15:44:48.073353 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.073337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/38cac5cb-e76e-4799-a54c-eaf3158d7158-image-registry-private-configuration\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.073407 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.073355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/38cac5cb-e76e-4799-a54c-eaf3158d7158-installation-pull-secrets\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.078598 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.078556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2fp\" (UniqueName: \"kubernetes.io/projected/d818a8bd-49ec-4d08-9602-78e859ddf43e-kube-api-access-wm2fp\") pod \"service-ca-operator-859f94c4b-xk8g2\" (UID: \"d818a8bd-49ec-4d08-9602-78e859ddf43e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"
Sep 08 15:44:48.079518 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.079492    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq4qg\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-kube-api-access-cq4qg\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.080107 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.080052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-bound-sa-token\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.080688 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.080462    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v7f9\" (UniqueName: \"kubernetes.io/projected/593f6dab-17b1-48a1-a4bb-a9164f694d90-kube-api-access-5v7f9\") pod \"kube-storage-version-migrator-operator-8d7dd4675-vdxpx\" (UID: \"593f6dab-17b1-48a1-a4bb-a9164f694d90\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"
Sep 08 15:44:48.081568 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.081550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5qqc\" (UniqueName: \"kubernetes.io/projected/c30ad658-852c-412f-af05-83940b0551b2-kube-api-access-g5qqc\") pod \"volume-data-source-validator-695c895bd6-z67qz\" (UID: \"c30ad658-852c-412f-af05-83940b0551b2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-z67qz"
Sep 08 15:44:48.083578 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.083557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62fvl\" (UniqueName: \"kubernetes.io/projected/a27cac25-7d24-41e2-bb79-a2412491978b-kube-api-access-62fvl\") pod \"console-operator-85b454874b-fzq5z\" (UID: \"a27cac25-7d24-41e2-bb79-a2412491978b\") " pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:48.110252 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.110233    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-z67qz"
Sep 08 15:44:48.113344 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.113321    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"
Sep 08 15:44:48.124825 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.124800    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:48.154343 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.153664    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:48.171309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.171440 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:48.171440 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.171440 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9x2h\" (UniqueName: \"kubernetes.io/projected/b6a25233-dc78-40bb-9a74-25f3aae52b3c-kube-api-access-z9x2h\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.171440 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-default-certificate\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.171647 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hckv7\" (UID: \"8c9f9660-e1c4-4433-a181-e8eee81bbe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:44:48.171647 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a25233-dc78-40bb-9a74-25f3aae52b3c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.171647 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a25233-dc78-40bb-9a74-25f3aae52b3c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.171647 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:44:48.171647 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-stats-auth\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.171647 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3c368f6-fe5b-4cd8-8d03-0c75a4478a19-cert\") pod \"ingress-canary-62rfg\" (UID: \"b3c368f6-fe5b-4cd8-8d03-0c75a4478a19\") " pod="openshift-ingress-canary/ingress-canary-62rfg"
Sep 08 15:44:48.171647 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ctn\" (UniqueName: \"kubernetes.io/projected/73240d9c-1436-48bd-90b2-a665d22d7006-kube-api-access-t8ctn\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.171974 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcxhx\" (UniqueName: \"kubernetes.io/projected/8e8e89d2-9be9-4976-84ce-6abf114256f0-kube-api-access-bcxhx\") pod \"network-check-source-685456655f-ztgpn\" (UID: \"8e8e89d2-9be9-4976-84ce-6abf114256f0\") " pod="openshift-network-diagnostics/network-check-source-685456655f-ztgpn"
Sep 08 15:44:48.171974 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43463d7-de74-46e8-af40-2db7ab47f8ff-tmp-dir\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:48.171974 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6a25233-dc78-40bb-9a74-25f3aae52b3c-serving-cert\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.171974 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvvz6\" (UniqueName: \"kubernetes.io/projected/b43463d7-de74-46e8-af40-2db7ab47f8ff-kube-api-access-dvvz6\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:48.171974 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b43463d7-de74-46e8-af40-2db7ab47f8ff-config-volume\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:48.171974 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2mv5\" (UniqueName: \"kubernetes.io/projected/280e0b1c-6fa9-4d0a-8769-8589179684a3-kube-api-access-d2mv5\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:44:48.171974 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mb5v\" (UniqueName: \"kubernetes.io/projected/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-kube-api-access-2mb5v\") pod \"cluster-samples-operator-7cdcfcf76c-hckv7\" (UID: \"8c9f9660-e1c4-4433-a181-e8eee81bbe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:44:48.171974 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/280e0b1c-6fa9-4d0a-8769-8589179684a3-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:44:48.171974 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6a25233-dc78-40bb-9a74-25f3aae52b3c-tmp\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.171974 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b6a25233-dc78-40bb-9a74-25f3aae52b3c-snapshots\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.172449 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.171979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmmgf\" (UniqueName: \"kubernetes.io/projected/b3c368f6-fe5b-4cd8-8d03-0c75a4478a19-kube-api-access-bmmgf\") pod \"ingress-canary-62rfg\" (UID: \"b3c368f6-fe5b-4cd8-8d03-0c75a4478a19\") " pod="openshift-ingress-canary/ingress-canary-62rfg"
Sep 08 15:44:48.172909 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.172688    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6a25233-dc78-40bb-9a74-25f3aae52b3c-tmp\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.173452 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.173043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/280e0b1c-6fa9-4d0a-8769-8589179684a3-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:44:48.173452 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.173111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43463d7-de74-46e8-af40-2db7ab47f8ff-tmp-dir\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:48.173452 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.173189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b6a25233-dc78-40bb-9a74-25f3aae52b3c-snapshots\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.173679 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.173514    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:44:48.173679 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.173523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b43463d7-de74-46e8-af40-2db7ab47f8ff-config-volume\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:48.173679 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.173584    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls podName:b43463d7-de74-46e8-af40-2db7ab47f8ff nodeName:}" failed. No retries permitted until 2026-09-08 15:44:48.673564152 +0000 UTC m=+35.297256460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls") pod "dns-default-6vlx8" (UID: "b43463d7-de74-46e8-af40-2db7ab47f8ff") : secret "dns-default-metrics-tls" not found
Sep 08 15:44:48.173679 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.173645    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 15:44:48.173679 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.173676    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs podName:73240d9c-1436-48bd-90b2-a665d22d7006 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:48.673666085 +0000 UTC m=+35.297358396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs") pod "router-default-5f6c69889d-c7hpw" (UID: "73240d9c-1436-48bd-90b2-a665d22d7006") : secret "router-metrics-certs-default" not found
Sep 08 15:44:48.173964 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.173772    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle podName:73240d9c-1436-48bd-90b2-a665d22d7006 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:48.673752359 +0000 UTC m=+35.297444686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle") pod "router-default-5f6c69889d-c7hpw" (UID: "73240d9c-1436-48bd-90b2-a665d22d7006") : configmap references non-existent config key: service-ca.crt
Sep 08 15:44:48.173964 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.173903    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 15:44:48.173964 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.173951    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls podName:280e0b1c-6fa9-4d0a-8769-8589179684a3 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:48.673936572 +0000 UTC m=+35.297628875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-n5fbl" (UID: "280e0b1c-6fa9-4d0a-8769-8589179684a3") : secret "cluster-monitoring-operator-tls" not found
Sep 08 15:44:48.174337 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.174181    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:44:48.174337 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.174303    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls podName:8c9f9660-e1c4-4433-a181-e8eee81bbe2e nodeName:}" failed. No retries permitted until 2026-09-08 15:44:48.674263931 +0000 UTC m=+35.297956250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hckv7" (UID: "8c9f9660-e1c4-4433-a181-e8eee81bbe2e") : secret "samples-operator-tls" not found
Sep 08 15:44:48.174664 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.174380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a25233-dc78-40bb-9a74-25f3aae52b3c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.174841 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.174775    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a25233-dc78-40bb-9a74-25f3aae52b3c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.176827 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.176783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3c368f6-fe5b-4cd8-8d03-0c75a4478a19-cert\") pod \"ingress-canary-62rfg\" (UID: \"b3c368f6-fe5b-4cd8-8d03-0c75a4478a19\") " pod="openshift-ingress-canary/ingress-canary-62rfg"
Sep 08 15:44:48.178912 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.178861    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-default-certificate\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.179183 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.179158    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6a25233-dc78-40bb-9a74-25f3aae52b3c-serving-cert\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.179319 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.179264    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"
Sep 08 15:44:48.179572 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.179529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-stats-auth\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.184082 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.183986    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9x2h\" (UniqueName: \"kubernetes.io/projected/b6a25233-dc78-40bb-9a74-25f3aae52b3c-kube-api-access-z9x2h\") pod \"insights-operator-76bcd6ccf5-nxt9p\" (UID: \"b6a25233-dc78-40bb-9a74-25f3aae52b3c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.185145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.185109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ctn\" (UniqueName: \"kubernetes.io/projected/73240d9c-1436-48bd-90b2-a665d22d7006-kube-api-access-t8ctn\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.185765 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.185686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmmgf\" (UniqueName: \"kubernetes.io/projected/b3c368f6-fe5b-4cd8-8d03-0c75a4478a19-kube-api-access-bmmgf\") pod \"ingress-canary-62rfg\" (UID: \"b3c368f6-fe5b-4cd8-8d03-0c75a4478a19\") " pod="openshift-ingress-canary/ingress-canary-62rfg"
Sep 08 15:44:48.186087 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.186034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcxhx\" (UniqueName: \"kubernetes.io/projected/8e8e89d2-9be9-4976-84ce-6abf114256f0-kube-api-access-bcxhx\") pod \"network-check-source-685456655f-ztgpn\" (UID: \"8e8e89d2-9be9-4976-84ce-6abf114256f0\") " pod="openshift-network-diagnostics/network-check-source-685456655f-ztgpn"
Sep 08 15:44:48.186409 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.186307    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mb5v\" (UniqueName: \"kubernetes.io/projected/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-kube-api-access-2mb5v\") pod \"cluster-samples-operator-7cdcfcf76c-hckv7\" (UID: \"8c9f9660-e1c4-4433-a181-e8eee81bbe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:44:48.186937 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.186897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2mv5\" (UniqueName: \"kubernetes.io/projected/280e0b1c-6fa9-4d0a-8769-8589179684a3-kube-api-access-d2mv5\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:44:48.188554 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.187652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvvz6\" (UniqueName: \"kubernetes.io/projected/b43463d7-de74-46e8-af40-2db7ab47f8ff-kube-api-access-dvvz6\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:48.226589 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.226347    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p"
Sep 08 15:44:48.250802 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.250515    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62rfg"
Sep 08 15:44:48.264862 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.264692    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-ztgpn"
Sep 08 15:44:48.446801 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.446778    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-z67qz"]
Sep 08 15:44:48.449504 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.449444    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2"]
Sep 08 15:44:48.450738 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:48.450709    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30ad658_852c_412f_af05_83940b0551b2.slice/crio-69dfbc3dc04a2e65f768aa2366139b200c85758f8e3ea2da731004183a4b07a0 WatchSource:0}: Error finding container 69dfbc3dc04a2e65f768aa2366139b200c85758f8e3ea2da731004183a4b07a0: Status 404 returned error can't find the container with id 69dfbc3dc04a2e65f768aa2366139b200c85758f8e3ea2da731004183a4b07a0
Sep 08 15:44:48.453522 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:48.453487    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd818a8bd_49ec_4d08_9602_78e859ddf43e.slice/crio-f1a558e42d9e4de7feaea03d4b11977a93f3ceefeae098be6e50cea24a1fdf35 WatchSource:0}: Error finding container f1a558e42d9e4de7feaea03d4b11977a93f3ceefeae098be6e50cea24a1fdf35: Status 404 returned error can't find the container with id f1a558e42d9e4de7feaea03d4b11977a93f3ceefeae098be6e50cea24a1fdf35
Sep 08 15:44:48.459040 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.459020    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-fzq5z"]
Sep 08 15:44:48.461724 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:48.461703    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27cac25_7d24_41e2_bb79_a2412491978b.slice/crio-8a3b53dd4301eaea54a607d57638a1d888abae93b09d609779d595ea604c8c43 WatchSource:0}: Error finding container 8a3b53dd4301eaea54a607d57638a1d888abae93b09d609779d595ea604c8c43: Status 404 returned error can't find the container with id 8a3b53dd4301eaea54a607d57638a1d888abae93b09d609779d595ea604c8c43
Sep 08 15:44:48.497774 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.496422    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx"]
Sep 08 15:44:48.538044 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.538015    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-nxt9p"]
Sep 08 15:44:48.542319 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:48.542292    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a25233_dc78_40bb_9a74_25f3aae52b3c.slice/crio-bc34872902ebbc4842008888b917c7cce4c55e575045756f3e9bd32d7c72a4e3 WatchSource:0}: Error finding container bc34872902ebbc4842008888b917c7cce4c55e575045756f3e9bd32d7c72a4e3: Status 404 returned error can't find the container with id bc34872902ebbc4842008888b917c7cce4c55e575045756f3e9bd32d7c72a4e3
Sep 08 15:44:48.560678 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.560624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62rfg"]
Sep 08 15:44:48.562550 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:48.562525    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c368f6_fe5b_4cd8_8d03_0c75a4478a19.slice/crio-1730f65b961ffb6937bf31118706a02a7d56e183f446096e74743e717d92c30c WatchSource:0}: Error finding container 1730f65b961ffb6937bf31118706a02a7d56e183f446096e74743e717d92c30c: Status 404 returned error can't find the container with id 1730f65b961ffb6937bf31118706a02a7d56e183f446096e74743e717d92c30c
Sep 08 15:44:48.575733 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.575706    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-ztgpn"]
Sep 08 15:44:48.576956 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.576938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:48.577117 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.577059    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:44:48.577117 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.577074    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5f6df75-6fqds: secret "image-registry-tls" not found
Sep 08 15:44:48.577224 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.577138    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls podName:38cac5cb-e76e-4799-a54c-eaf3158d7158 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:49.5771101 +0000 UTC m=+36.200802418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls") pod "image-registry-74f5f6df75-6fqds" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158") : secret "image-registry-tls" not found
Sep 08 15:44:48.578684 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:48.578666    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8e89d2_9be9_4976_84ce_6abf114256f0.slice/crio-f5b5cd683d934e382d1b806dba1d8a952e5ff8f75bd2a85c485861cd438c225f WatchSource:0}: Error finding container f5b5cd683d934e382d1b806dba1d8a952e5ff8f75bd2a85c485861cd438c225f: Status 404 returned error can't find the container with id f5b5cd683d934e382d1b806dba1d8a952e5ff8f75bd2a85c485861cd438c225f
Sep 08 15:44:48.677569 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.677542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:44:48.677689 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.677646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.677689 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.677667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:48.677689 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.677685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:48.677856 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.677699    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 15:44:48.677856 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.677717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hckv7\" (UID: \"8c9f9660-e1c4-4433-a181-e8eee81bbe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:44:48.677856 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.677768    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls podName:280e0b1c-6fa9-4d0a-8769-8589179684a3 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:49.677745552 +0000 UTC m=+36.301437868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-n5fbl" (UID: "280e0b1c-6fa9-4d0a-8769-8589179684a3") : secret "cluster-monitoring-operator-tls" not found
Sep 08 15:44:48.677856 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.677806    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:44:48.677856 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.677809    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:44:48.677856 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.677837    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 15:44:48.677856 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.677852    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle podName:73240d9c-1436-48bd-90b2-a665d22d7006 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:49.677834827 +0000 UTC m=+36.301527136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle") pod "router-default-5f6c69889d-c7hpw" (UID: "73240d9c-1436-48bd-90b2-a665d22d7006") : configmap references non-existent config key: service-ca.crt
Sep 08 15:44:48.678177 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.677872    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls podName:8c9f9660-e1c4-4433-a181-e8eee81bbe2e nodeName:}" failed. No retries permitted until 2026-09-08 15:44:49.677864916 +0000 UTC m=+36.301557218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hckv7" (UID: "8c9f9660-e1c4-4433-a181-e8eee81bbe2e") : secret "samples-operator-tls" not found
Sep 08 15:44:48.678177 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.677892    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs podName:73240d9c-1436-48bd-90b2-a665d22d7006 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:49.677876489 +0000 UTC m=+36.301568797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs") pod "router-default-5f6c69889d-c7hpw" (UID: "73240d9c-1436-48bd-90b2-a665d22d7006") : secret "router-metrics-certs-default" not found
Sep 08 15:44:48.678177 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:48.677911    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls podName:b43463d7-de74-46e8-af40-2db7ab47f8ff nodeName:}" failed. No retries permitted until 2026-09-08 15:44:49.677901807 +0000 UTC m=+36.301594113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls") pod "dns-default-6vlx8" (UID: "b43463d7-de74-46e8-af40-2db7ab47f8ff") : secret "dns-default-metrics-tls" not found
Sep 08 15:44:48.894252 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.894226    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:44:48.894418 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.894234    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:44:48.896737 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.896716    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 15:44:48.896867 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:48.896794    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xkdlq\""
Sep 08 15:44:49.067731 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:49.067650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2" event={"ID":"d818a8bd-49ec-4d08-9602-78e859ddf43e","Type":"ContainerStarted","Data":"f1a558e42d9e4de7feaea03d4b11977a93f3ceefeae098be6e50cea24a1fdf35"}
Sep 08 15:44:49.069113 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:49.069086    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p" event={"ID":"b6a25233-dc78-40bb-9a74-25f3aae52b3c","Type":"ContainerStarted","Data":"bc34872902ebbc4842008888b917c7cce4c55e575045756f3e9bd32d7c72a4e3"}
Sep 08 15:44:49.070433 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:49.070406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-z67qz" event={"ID":"c30ad658-852c-412f-af05-83940b0551b2","Type":"ContainerStarted","Data":"69dfbc3dc04a2e65f768aa2366139b200c85758f8e3ea2da731004183a4b07a0"}
Sep 08 15:44:49.071787 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:49.071724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-ztgpn" event={"ID":"8e8e89d2-9be9-4976-84ce-6abf114256f0","Type":"ContainerStarted","Data":"f5b5cd683d934e382d1b806dba1d8a952e5ff8f75bd2a85c485861cd438c225f"}
Sep 08 15:44:49.073231 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:49.073209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62rfg" event={"ID":"b3c368f6-fe5b-4cd8-8d03-0c75a4478a19","Type":"ContainerStarted","Data":"1730f65b961ffb6937bf31118706a02a7d56e183f446096e74743e717d92c30c"}
Sep 08 15:44:49.074546 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:49.074521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx" event={"ID":"593f6dab-17b1-48a1-a4bb-a9164f694d90","Type":"ContainerStarted","Data":"141caf9d85d2ff0abcff855c5ad023b161c64767e8d755e14e21b0e4dcb40059"}
Sep 08 15:44:49.075758 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:49.075731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-fzq5z" event={"ID":"a27cac25-7d24-41e2-bb79-a2412491978b","Type":"ContainerStarted","Data":"8a3b53dd4301eaea54a607d57638a1d888abae93b09d609779d595ea604c8c43"}
Sep 08 15:44:49.586522 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:49.586483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:49.586970 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:49.586615    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:44:49.586970 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:49.586633    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5f6df75-6fqds: secret "image-registry-tls" not found
Sep 08 15:44:49.586970 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:49.586697    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls podName:38cac5cb-e76e-4799-a54c-eaf3158d7158 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:51.586682903 +0000 UTC m=+38.210375205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls") pod "image-registry-74f5f6df75-6fqds" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158") : secret "image-registry-tls" not found
Sep 08 15:44:49.687066 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:49.687035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:49.687066 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:49.687070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:49.687302 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:49.687089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:49.687302 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:49.687221    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 15:44:49.687302 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:49.687236    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle podName:73240d9c-1436-48bd-90b2-a665d22d7006 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:51.687216707 +0000 UTC m=+38.310909031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle") pod "router-default-5f6c69889d-c7hpw" (UID: "73240d9c-1436-48bd-90b2-a665d22d7006") : configmap references non-existent config key: service-ca.crt
Sep 08 15:44:49.687302 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:49.687234    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:44:49.687302 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:49.687287    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs podName:73240d9c-1436-48bd-90b2-a665d22d7006 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:51.687268681 +0000 UTC m=+38.310961000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs") pod "router-default-5f6c69889d-c7hpw" (UID: "73240d9c-1436-48bd-90b2-a665d22d7006") : secret "router-metrics-certs-default" not found
Sep 08 15:44:49.687562 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:49.687311    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls podName:b43463d7-de74-46e8-af40-2db7ab47f8ff nodeName:}" failed. No retries permitted until 2026-09-08 15:44:51.687294113 +0000 UTC m=+38.310986416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls") pod "dns-default-6vlx8" (UID: "b43463d7-de74-46e8-af40-2db7ab47f8ff") : secret "dns-default-metrics-tls" not found
Sep 08 15:44:49.687562 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:49.687341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hckv7\" (UID: \"8c9f9660-e1c4-4433-a181-e8eee81bbe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:44:49.687562 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:49.687420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:44:49.687562 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:49.687544    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 15:44:49.687734 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:49.687584    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls podName:280e0b1c-6fa9-4d0a-8769-8589179684a3 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:51.687571928 +0000 UTC m=+38.311264236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-n5fbl" (UID: "280e0b1c-6fa9-4d0a-8769-8589179684a3") : secret "cluster-monitoring-operator-tls" not found
Sep 08 15:44:49.687734 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:49.687658    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:44:49.687734 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:49.687714    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls podName:8c9f9660-e1c4-4433-a181-e8eee81bbe2e nodeName:}" failed. No retries permitted until 2026-09-08 15:44:51.687699928 +0000 UTC m=+38.311392258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hckv7" (UID: "8c9f9660-e1c4-4433-a181-e8eee81bbe2e") : secret "samples-operator-tls" not found
Sep 08 15:44:51.206345 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:51.206259    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:51.287166 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:51.287132    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:51.343373 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:51.343322    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:51.436186 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:51.436069    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:51.536588 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:51.536556    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:51.606536 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:51.605939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:51.606536 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:51.606079    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:44:51.606536 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:51.606095    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5f6df75-6fqds: secret "image-registry-tls" not found
Sep 08 15:44:51.606536 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:51.606151    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls podName:38cac5cb-e76e-4799-a54c-eaf3158d7158 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:55.606131389 +0000 UTC m=+42.229823710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls") pod "image-registry-74f5f6df75-6fqds" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158") : secret "image-registry-tls" not found
Sep 08 15:44:51.640547 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:51.640472    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:51.706920 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:51.706833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hckv7\" (UID: \"8c9f9660-e1c4-4433-a181-e8eee81bbe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:44:51.706920 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:51.706912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:44:51.707134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:51.707034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:51.707134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:51.707062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:51.707134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:51.707094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:51.707289 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:51.707234    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle podName:73240d9c-1436-48bd-90b2-a665d22d7006 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:55.707216002 +0000 UTC m=+42.330908322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle") pod "router-default-5f6c69889d-c7hpw" (UID: "73240d9c-1436-48bd-90b2-a665d22d7006") : configmap references non-existent config key: service-ca.crt
Sep 08 15:44:51.707354 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:51.707321    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:44:51.707733 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:51.707359    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls podName:8c9f9660-e1c4-4433-a181-e8eee81bbe2e nodeName:}" failed. No retries permitted until 2026-09-08 15:44:55.707347232 +0000 UTC m=+42.331039541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hckv7" (UID: "8c9f9660-e1c4-4433-a181-e8eee81bbe2e") : secret "samples-operator-tls" not found
Sep 08 15:44:51.707733 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:51.707435    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 15:44:51.707733 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:51.707471    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls podName:280e0b1c-6fa9-4d0a-8769-8589179684a3 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:55.707459156 +0000 UTC m=+42.331151472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-n5fbl" (UID: "280e0b1c-6fa9-4d0a-8769-8589179684a3") : secret "cluster-monitoring-operator-tls" not found
Sep 08 15:44:51.707733 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:51.707509    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:44:51.707733 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:51.707577    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls podName:b43463d7-de74-46e8-af40-2db7ab47f8ff nodeName:}" failed. No retries permitted until 2026-09-08 15:44:55.707558801 +0000 UTC m=+42.331251105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls") pod "dns-default-6vlx8" (UID: "b43463d7-de74-46e8-af40-2db7ab47f8ff") : secret "dns-default-metrics-tls" not found
Sep 08 15:44:51.707733 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:51.707646    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 15:44:51.707733 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:51.707680    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs podName:73240d9c-1436-48bd-90b2-a665d22d7006 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:55.707669119 +0000 UTC m=+42.331361424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs") pod "router-default-5f6c69889d-c7hpw" (UID: "73240d9c-1436-48bd-90b2-a665d22d7006") : secret "router-metrics-certs-default" not found
Sep 08 15:44:52.240970 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:52.240881    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:52.346269 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:52.346217    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:52.636887 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:52.636800    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:52.691252 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:52.691219    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:52.953235 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:52.953158    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:53.023104 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:53.023070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log" limit=1024
Sep 08 15:44:55.643569 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:55.643544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:44:55.643960 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:55.643676    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:44:55.643960 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:55.643693    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5f6df75-6fqds: secret "image-registry-tls" not found
Sep 08 15:44:55.643960 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:55.643754    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls podName:38cac5cb-e76e-4799-a54c-eaf3158d7158 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:03.643735583 +0000 UTC m=+50.267427886 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls") pod "image-registry-74f5f6df75-6fqds" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158") : secret "image-registry-tls" not found
Sep 08 15:44:55.744030 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:55.743950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:55.744030 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:55.743993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:44:55.744030 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:55.744024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:44:55.744295 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:55.744055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hckv7\" (UID: \"8c9f9660-e1c4-4433-a181-e8eee81bbe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:44:55.744295 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:55.744101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:44:55.744295 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:55.744111    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 15:44:55.744295 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:55.744173    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs podName:73240d9c-1436-48bd-90b2-a665d22d7006 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:03.744153831 +0000 UTC m=+50.367846151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs") pod "router-default-5f6c69889d-c7hpw" (UID: "73240d9c-1436-48bd-90b2-a665d22d7006") : secret "router-metrics-certs-default" not found
Sep 08 15:44:55.744295 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:55.744111    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:44:55.744295 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:55.744202    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle podName:73240d9c-1436-48bd-90b2-a665d22d7006 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:03.744183877 +0000 UTC m=+50.367876199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle") pod "router-default-5f6c69889d-c7hpw" (UID: "73240d9c-1436-48bd-90b2-a665d22d7006") : configmap references non-existent config key: service-ca.crt
Sep 08 15:44:55.744295 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:55.744200    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 15:44:55.744295 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:55.744230    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:44:55.744295 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:55.744271    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls podName:280e0b1c-6fa9-4d0a-8769-8589179684a3 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:03.744241008 +0000 UTC m=+50.367933332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-n5fbl" (UID: "280e0b1c-6fa9-4d0a-8769-8589179684a3") : secret "cluster-monitoring-operator-tls" not found
Sep 08 15:44:55.744647 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:55.744305    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls podName:8c9f9660-e1c4-4433-a181-e8eee81bbe2e nodeName:}" failed. No retries permitted until 2026-09-08 15:45:03.744277501 +0000 UTC m=+50.367969807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hckv7" (UID: "8c9f9660-e1c4-4433-a181-e8eee81bbe2e") : secret "samples-operator-tls" not found
Sep 08 15:44:55.744647 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:55.744324    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls podName:b43463d7-de74-46e8-af40-2db7ab47f8ff nodeName:}" failed. No retries permitted until 2026-09-08 15:45:03.744314091 +0000 UTC m=+50.368006398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls") pod "dns-default-6vlx8" (UID: "b43463d7-de74-46e8-af40-2db7ab47f8ff") : secret "dns-default-metrics-tls" not found
Sep 08 15:44:58.102856 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.102822    2581 generic.go:356] "Generic (PLEG): container finished" podID="65d13726-37f1-465a-bbd5-1ebf50e5ead8" containerID="35ce90bdc809e46bcfe175aff3fef00c9cfdcb93104ad35561ba3134e02bcfb7" exitCode=0
Sep 08 15:44:58.103221 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.102909    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph9tx" event={"ID":"65d13726-37f1-465a-bbd5-1ebf50e5ead8","Type":"ContainerDied","Data":"35ce90bdc809e46bcfe175aff3fef00c9cfdcb93104ad35561ba3134e02bcfb7"}
Sep 08 15:44:58.104860 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.104803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-ztgpn" event={"ID":"8e8e89d2-9be9-4976-84ce-6abf114256f0","Type":"ContainerStarted","Data":"1004d4fe11f634a635eec46cf35659ed71a936dad487db58a353a7e0d9cfe0a6"}
Sep 08 15:44:58.106537 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.106509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62rfg" event={"ID":"b3c368f6-fe5b-4cd8-8d03-0c75a4478a19","Type":"ContainerStarted","Data":"f81abb17553d479b9654e3ed65688011f4bc23bb3f29869c8bbe43af5bbb73f3"}
Sep 08 15:44:58.108140 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.108113    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx" event={"ID":"593f6dab-17b1-48a1-a4bb-a9164f694d90","Type":"ContainerStarted","Data":"f428acbe143abea5aff4c8e8fa5f666fb0b797ffb71c9d3c1e931c3fceb7d069"}
Sep 08 15:44:58.110119 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.110069    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/0.log"
Sep 08 15:44:58.110119 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.110103    2581 generic.go:356] "Generic (PLEG): container finished" podID="a27cac25-7d24-41e2-bb79-a2412491978b" containerID="f8270814e368e2f7a2b9dbceee7952f769c292e3ffd2560bddd9f04beea512d3" exitCode=255
Sep 08 15:44:58.110320 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.110172    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-fzq5z" event={"ID":"a27cac25-7d24-41e2-bb79-a2412491978b","Type":"ContainerDied","Data":"f8270814e368e2f7a2b9dbceee7952f769c292e3ffd2560bddd9f04beea512d3"}
Sep 08 15:44:58.110377 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.110350    2581 scope.go:117] "RemoveContainer" containerID="f8270814e368e2f7a2b9dbceee7952f769c292e3ffd2560bddd9f04beea512d3"
Sep 08 15:44:58.111878 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.111858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2" event={"ID":"d818a8bd-49ec-4d08-9602-78e859ddf43e","Type":"ContainerStarted","Data":"2b00d570ce009527d5f71c4c5ab8b88b1100ea427755a2333ea6060d024c0605"}
Sep 08 15:44:58.113658 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.113634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p" event={"ID":"b6a25233-dc78-40bb-9a74-25f3aae52b3c","Type":"ContainerStarted","Data":"76b04771ce6c458646b46922ccc8d8a34f3c465bce0d83fb4aee236dd960b314"}
Sep 08 15:44:58.115096 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.115075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-z67qz" event={"ID":"c30ad658-852c-412f-af05-83940b0551b2","Type":"ContainerStarted","Data":"60e514759d8c29a89fdc0ed733368ce9d1141ad47d1d38390546355a404d0a65"}
Sep 08 15:44:58.154161 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.154104    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:58.154329 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.154317    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:44:58.170734 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.170331    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p" podStartSLOduration=19.715866401 podStartE2EDuration="28.170311966s" totalImagesPullingTime="8.454445565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:58.146089265 +0000 UTC m=+44.769781606" watchObservedRunningTime="2026-09-08 15:44:58.170311966 +0000 UTC m=+44.794004292"
Sep 08 15:44:58.171653 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.171613    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-z67qz" podStartSLOduration=19.179834959 podStartE2EDuration="28.171599514s" totalImagesPullingTime="8.991764555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:58.169629207 +0000 UTC m=+44.793321555" watchObservedRunningTime="2026-09-08 15:44:58.171599514 +0000 UTC m=+44.795291840"
Sep 08 15:44:58.209812 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.209760    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2" podStartSLOduration=19.675010467 podStartE2EDuration="28.209741143s" totalImagesPullingTime="8.534730676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:58.207482138 +0000 UTC m=+44.831174463" watchObservedRunningTime="2026-09-08 15:44:58.209741143 +0000 UTC m=+44.833433466"
Sep 08 15:44:58.210485 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.210466    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-q5cth"]
Sep 08 15:44:58.229112 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.228318    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-q5cth"]
Sep 08 15:44:58.229112 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.228471    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-q5cth"
Sep 08 15:44:58.231116 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.231096    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 15:44:58.231484 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.231469    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5ln9t\""
Sep 08 15:44:58.232668 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.231473    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 15:44:58.250700 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.250654    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx" podStartSLOduration=19.75890419 podStartE2EDuration="28.250635276s" totalImagesPullingTime="8.491731086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:58.249145284 +0000 UTC m=+44.872837611" watchObservedRunningTime="2026-09-08 15:44:58.250635276 +0000 UTC m=+44.874327595"
Sep 08 15:44:58.268990 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.268942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4lcd\" (UniqueName: \"kubernetes.io/projected/a588199a-f0f0-476f-8806-b158e9539913-kube-api-access-p4lcd\") pod \"migrator-fdbb8fdff-q5cth\" (UID: \"a588199a-f0f0-476f-8806-b158e9539913\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-q5cth"
Sep 08 15:44:58.277937 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.277747    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-62rfg" podStartSLOduration=2.851708766 podStartE2EDuration="11.277729673s" totalImagesPullingTime="8.426020907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:58.276826617 +0000 UTC m=+44.900518968" watchObservedRunningTime="2026-09-08 15:44:58.277729673 +0000 UTC m=+44.901421990"
Sep 08 15:44:58.300383 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.298862    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-ztgpn" podStartSLOduration=19.888647034999998 podStartE2EDuration="28.29884367s" totalImagesPullingTime="8.410196635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:44:58.298032581 +0000 UTC m=+44.921724905" watchObservedRunningTime="2026-09-08 15:44:58.29884367 +0000 UTC m=+44.922535996"
Sep 08 15:44:58.370068 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.370037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4lcd\" (UniqueName: \"kubernetes.io/projected/a588199a-f0f0-476f-8806-b158e9539913-kube-api-access-p4lcd\") pod \"migrator-fdbb8fdff-q5cth\" (UID: \"a588199a-f0f0-476f-8806-b158e9539913\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-q5cth"
Sep 08 15:44:58.380152 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.380121    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4lcd\" (UniqueName: \"kubernetes.io/projected/a588199a-f0f0-476f-8806-b158e9539913-kube-api-access-p4lcd\") pod \"migrator-fdbb8fdff-q5cth\" (UID: \"a588199a-f0f0-476f-8806-b158e9539913\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-q5cth"
Sep 08 15:44:58.562499 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.562418    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-q5cth"
Sep 08 15:44:58.726883 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.726853    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7qvc9"]
Sep 08 15:44:58.751080 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.751057    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qvc9"]
Sep 08 15:44:58.751214 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.751172    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.753436 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.753414    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 15:44:58.753569 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.753445    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rhb4d\""
Sep 08 15:44:58.753569 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.753490    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 15:44:58.773721 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.773694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmgz5\" (UniqueName: \"kubernetes.io/projected/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-api-access-pmgz5\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.773812 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.773725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.773890 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.773873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-data-volume\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.773946 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.773930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.773983 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.773971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-crio-socket\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.819571 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.819514    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-jjj7p"]
Sep 08 15:44:58.841187 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.841163    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-jjj7p"]
Sep 08 15:44:58.841292 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.841262    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p"
Sep 08 15:44:58.843268 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.843249    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 08 15:44:58.843367 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.843326    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-b4msn\""
Sep 08 15:44:58.843453 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.843371    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 08 15:44:58.845377 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.845359    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-q5cth"]
Sep 08 15:44:58.848928 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:44:58.848910    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda588199a_f0f0_476f_8806_b158e9539913.slice/crio-6f663fcc7c967ba32702ea0d4eb24429749488a8fc31ed816a5dc0c2925a57bc WatchSource:0}: Error finding container 6f663fcc7c967ba32702ea0d4eb24429749488a8fc31ed816a5dc0c2925a57bc: Status 404 returned error can't find the container with id 6f663fcc7c967ba32702ea0d4eb24429749488a8fc31ed816a5dc0c2925a57bc
Sep 08 15:44:58.875012 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.874993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-crio-socket\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.875098 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.875031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmgz5\" (UniqueName: \"kubernetes.io/projected/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-api-access-pmgz5\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.875098 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.875049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.875171 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:58.875130    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:44:58.875202 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:58.875183    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls podName:c82d6a57-745e-47b9-afd5-b5c466bdf5d6 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:59.375162103 +0000 UTC m=+45.998854412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qvc9" (UID: "c82d6a57-745e-47b9-afd5-b5c466bdf5d6") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:44:58.875202 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.875190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-crio-socket\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.875320 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.875305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jjj7p\" (UID: \"7ff830b7-be75-496c-8e4e-bf04e35912ee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p"
Sep 08 15:44:58.875355 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.875331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-data-volume\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.875387 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.875374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ff830b7-be75-496c-8e4e-bf04e35912ee-nginx-conf\") pod \"networking-console-plugin-79977f9844-jjj7p\" (UID: \"7ff830b7-be75-496c-8e4e-bf04e35912ee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p"
Sep 08 15:44:58.875433 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.875420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.875517 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:58.875506    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:44:58.875553 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:58.875544    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm podName:c82d6a57-745e-47b9-afd5-b5c466bdf5d6 nodeName:}" failed. No retries permitted until 2026-09-08 15:44:59.375535232 +0000 UTC m=+45.999227535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qvc9" (UID: "c82d6a57-745e-47b9-afd5-b5c466bdf5d6") : configmap "kube-rbac-proxy" not found
Sep 08 15:44:58.875606 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.875590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-data-volume\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.882833 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.882817    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmgz5\" (UniqueName: \"kubernetes.io/projected/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-api-access-pmgz5\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:58.976558 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.976533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jjj7p\" (UID: \"7ff830b7-be75-496c-8e4e-bf04e35912ee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p"
Sep 08 15:44:58.976674 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.976582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ff830b7-be75-496c-8e4e-bf04e35912ee-nginx-conf\") pod \"networking-console-plugin-79977f9844-jjj7p\" (UID: \"7ff830b7-be75-496c-8e4e-bf04e35912ee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p"
Sep 08 15:44:58.976722 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:58.976679    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:44:58.976757 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:58.976733    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert podName:7ff830b7-be75-496c-8e4e-bf04e35912ee nodeName:}" failed. No retries permitted until 2026-09-08 15:44:59.476716177 +0000 UTC m=+46.100408479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-jjj7p" (UID: "7ff830b7-be75-496c-8e4e-bf04e35912ee") : secret "networking-console-plugin-cert" not found
Sep 08 15:44:58.977160 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:58.977144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ff830b7-be75-496c-8e4e-bf04e35912ee-nginx-conf\") pod \"networking-console-plugin-79977f9844-jjj7p\" (UID: \"7ff830b7-be75-496c-8e4e-bf04e35912ee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p"
Sep 08 15:44:59.122053 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:59.121990    2581 generic.go:356] "Generic (PLEG): container finished" podID="65d13726-37f1-465a-bbd5-1ebf50e5ead8" containerID="7ee3312904aa6301c9a017e5f9db2aad12f0b075c77fbf7d7671fdffb6c34519" exitCode=0
Sep 08 15:44:59.122442 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:59.122077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph9tx" event={"ID":"65d13726-37f1-465a-bbd5-1ebf50e5ead8","Type":"ContainerDied","Data":"7ee3312904aa6301c9a017e5f9db2aad12f0b075c77fbf7d7671fdffb6c34519"}
Sep 08 15:44:59.123724 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:59.123708    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 15:44:59.124101 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:59.124088    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/0.log"
Sep 08 15:44:59.124147 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:59.124118    2581 generic.go:356] "Generic (PLEG): container finished" podID="a27cac25-7d24-41e2-bb79-a2412491978b" containerID="ef9ea3b00e1aea22d3ca03c4125df6e762d9ed899a9842d30c946bfae270f4de" exitCode=255
Sep 08 15:44:59.124186 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:59.124143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-fzq5z" event={"ID":"a27cac25-7d24-41e2-bb79-a2412491978b","Type":"ContainerDied","Data":"ef9ea3b00e1aea22d3ca03c4125df6e762d9ed899a9842d30c946bfae270f4de"}
Sep 08 15:44:59.124227 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:59.124191    2581 scope.go:117] "RemoveContainer" containerID="f8270814e368e2f7a2b9dbceee7952f769c292e3ffd2560bddd9f04beea512d3"
Sep 08 15:44:59.124363 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:59.124347    2581 scope.go:117] "RemoveContainer" containerID="ef9ea3b00e1aea22d3ca03c4125df6e762d9ed899a9842d30c946bfae270f4de"
Sep 08 15:44:59.124536 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:59.124518    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-fzq5z_openshift-console-operator(a27cac25-7d24-41e2-bb79-a2412491978b)\"" pod="openshift-console-operator/console-operator-85b454874b-fzq5z" podUID="a27cac25-7d24-41e2-bb79-a2412491978b"
Sep 08 15:44:59.125678 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:59.125631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-q5cth" event={"ID":"a588199a-f0f0-476f-8806-b158e9539913","Type":"ContainerStarted","Data":"6f663fcc7c967ba32702ea0d4eb24429749488a8fc31ed816a5dc0c2925a57bc"}
Sep 08 15:44:59.386818 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:59.386788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:59.386960 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:59.386927    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:44:59.387019 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:59.386964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:44:59.387019 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:59.386991    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls podName:c82d6a57-745e-47b9-afd5-b5c466bdf5d6 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:00.386975491 +0000 UTC m=+47.010667794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qvc9" (UID: "c82d6a57-745e-47b9-afd5-b5c466bdf5d6") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:44:59.387124 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:59.387028    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:44:59.387124 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:59.387082    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm podName:c82d6a57-745e-47b9-afd5-b5c466bdf5d6 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:00.387066711 +0000 UTC m=+47.010759030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qvc9" (UID: "c82d6a57-745e-47b9-afd5-b5c466bdf5d6") : configmap "kube-rbac-proxy" not found
Sep 08 15:44:59.488336 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:44:59.488307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jjj7p\" (UID: \"7ff830b7-be75-496c-8e4e-bf04e35912ee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p"
Sep 08 15:44:59.488484 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:59.488468    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:44:59.488541 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:44:59.488527    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert podName:7ff830b7-be75-496c-8e4e-bf04e35912ee nodeName:}" failed. No retries permitted until 2026-09-08 15:45:00.488512702 +0000 UTC m=+47.112205018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-jjj7p" (UID: "7ff830b7-be75-496c-8e4e-bf04e35912ee") : secret "networking-console-plugin-cert" not found
Sep 08 15:45:00.130422 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:00.130333    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 15:45:00.130818 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:00.130696    2581 scope.go:117] "RemoveContainer" containerID="ef9ea3b00e1aea22d3ca03c4125df6e762d9ed899a9842d30c946bfae270f4de"
Sep 08 15:45:00.130889 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:00.130865    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-fzq5z_openshift-console-operator(a27cac25-7d24-41e2-bb79-a2412491978b)\"" pod="openshift-console-operator/console-operator-85b454874b-fzq5z" podUID="a27cac25-7d24-41e2-bb79-a2412491978b"
Sep 08 15:45:00.134159 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:00.134128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph9tx" event={"ID":"65d13726-37f1-465a-bbd5-1ebf50e5ead8","Type":"ContainerStarted","Data":"56cabde792f9ff7014a1d515a13f27990768c82b9c3a6b672d68c18d2d117465"}
Sep 08 15:45:00.175101 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:00.175051    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ph9tx" podStartSLOduration=9.267461777 podStartE2EDuration="45.175033133s" totalImagesPullingTime="35.907571356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:44:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:00.173970989 +0000 UTC m=+46.797663313" watchObservedRunningTime="2026-09-08 15:45:00.175033133 +0000 UTC m=+46.798725459"
Sep 08 15:45:00.396019 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:00.395979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:45:00.396189 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:00.396126    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:45:00.396189 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:00.396161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:45:00.396312 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:00.396191    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls podName:c82d6a57-745e-47b9-afd5-b5c466bdf5d6 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:02.39617505 +0000 UTC m=+49.019867352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qvc9" (UID: "c82d6a57-745e-47b9-afd5-b5c466bdf5d6") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:45:00.396312 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:00.396226    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:45:00.396448 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:00.396302    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm podName:c82d6a57-745e-47b9-afd5-b5c466bdf5d6 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:02.396282988 +0000 UTC m=+49.019975306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qvc9" (UID: "c82d6a57-745e-47b9-afd5-b5c466bdf5d6") : configmap "kube-rbac-proxy" not found
Sep 08 15:45:00.496697 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:00.496672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jjj7p\" (UID: \"7ff830b7-be75-496c-8e4e-bf04e35912ee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p"
Sep 08 15:45:00.496838 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:00.496824    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:45:00.496903 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:00.496887    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert podName:7ff830b7-be75-496c-8e4e-bf04e35912ee nodeName:}" failed. No retries permitted until 2026-09-08 15:45:02.49686982 +0000 UTC m=+49.120562141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-jjj7p" (UID: "7ff830b7-be75-496c-8e4e-bf04e35912ee") : secret "networking-console-plugin-cert" not found
Sep 08 15:45:00.910010 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:00.909988    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w4nwh_20a23223-f446-4941-9b20-6badf3aee2e3/dns-node-resolver/0.log"
Sep 08 15:45:01.146916 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.146884    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-z44f2"]
Sep 08 15:45:01.173424 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.173346    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-z44f2"]
Sep 08 15:45:01.173566 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.173472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-z44f2"
Sep 08 15:45:01.175673 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.175653    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 15:45:01.176007 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.175988    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 15:45:01.176299 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.176283    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 15:45:01.176539 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.176523    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 15:45:01.176918 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.176900    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5vzw2\""
Sep 08 15:45:01.303307 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.303276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58ea5830-31f8-498b-82c6-3cd8fcb22044-signing-cabundle\") pod \"service-ca-75d99c484-z44f2\" (UID: \"58ea5830-31f8-498b-82c6-3cd8fcb22044\") " pod="openshift-service-ca/service-ca-75d99c484-z44f2"
Sep 08 15:45:01.303453 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.303337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58ea5830-31f8-498b-82c6-3cd8fcb22044-signing-key\") pod \"service-ca-75d99c484-z44f2\" (UID: \"58ea5830-31f8-498b-82c6-3cd8fcb22044\") " pod="openshift-service-ca/service-ca-75d99c484-z44f2"
Sep 08 15:45:01.303525 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.303483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5s6t\" (UniqueName: \"kubernetes.io/projected/58ea5830-31f8-498b-82c6-3cd8fcb22044-kube-api-access-r5s6t\") pod \"service-ca-75d99c484-z44f2\" (UID: \"58ea5830-31f8-498b-82c6-3cd8fcb22044\") " pod="openshift-service-ca/service-ca-75d99c484-z44f2"
Sep 08 15:45:01.403943 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.403917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5s6t\" (UniqueName: \"kubernetes.io/projected/58ea5830-31f8-498b-82c6-3cd8fcb22044-kube-api-access-r5s6t\") pod \"service-ca-75d99c484-z44f2\" (UID: \"58ea5830-31f8-498b-82c6-3cd8fcb22044\") " pod="openshift-service-ca/service-ca-75d99c484-z44f2"
Sep 08 15:45:01.404082 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.404047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58ea5830-31f8-498b-82c6-3cd8fcb22044-signing-cabundle\") pod \"service-ca-75d99c484-z44f2\" (UID: \"58ea5830-31f8-498b-82c6-3cd8fcb22044\") " pod="openshift-service-ca/service-ca-75d99c484-z44f2"
Sep 08 15:45:01.404137 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.404089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58ea5830-31f8-498b-82c6-3cd8fcb22044-signing-key\") pod \"service-ca-75d99c484-z44f2\" (UID: \"58ea5830-31f8-498b-82c6-3cd8fcb22044\") " pod="openshift-service-ca/service-ca-75d99c484-z44f2"
Sep 08 15:45:01.404880 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.404846    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58ea5830-31f8-498b-82c6-3cd8fcb22044-signing-cabundle\") pod \"service-ca-75d99c484-z44f2\" (UID: \"58ea5830-31f8-498b-82c6-3cd8fcb22044\") " pod="openshift-service-ca/service-ca-75d99c484-z44f2"
Sep 08 15:45:01.407095 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.407067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58ea5830-31f8-498b-82c6-3cd8fcb22044-signing-key\") pod \"service-ca-75d99c484-z44f2\" (UID: \"58ea5830-31f8-498b-82c6-3cd8fcb22044\") " pod="openshift-service-ca/service-ca-75d99c484-z44f2"
Sep 08 15:45:01.412606 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.412580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5s6t\" (UniqueName: \"kubernetes.io/projected/58ea5830-31f8-498b-82c6-3cd8fcb22044-kube-api-access-r5s6t\") pod \"service-ca-75d99c484-z44f2\" (UID: \"58ea5830-31f8-498b-82c6-3cd8fcb22044\") " pod="openshift-service-ca/service-ca-75d99c484-z44f2"
Sep 08 15:45:01.482213 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.482153    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-z44f2"
Sep 08 15:45:01.711033 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.711010    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l6lrc_2201c616-9356-494d-915c-edee0306a6bf/node-ca/0.log"
Sep 08 15:45:01.770166 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:01.770101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-z44f2"]
Sep 08 15:45:02.026628 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:02.026562    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ea5830_31f8_498b_82c6_3cd8fcb22044.slice/crio-faf75ef74146568892d4de739b47791cf58d0e6f23d1540803514639e95f4608 WatchSource:0}: Error finding container faf75ef74146568892d4de739b47791cf58d0e6f23d1540803514639e95f4608: Status 404 returned error can't find the container with id faf75ef74146568892d4de739b47791cf58d0e6f23d1540803514639e95f4608
Sep 08 15:45:02.144198 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:02.144173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-z44f2" event={"ID":"58ea5830-31f8-498b-82c6-3cd8fcb22044","Type":"ContainerStarted","Data":"3f0a24698317429d784c70bae165212abc3076a8d59fd8ec90d96dbbcd0b0fc6"}
Sep 08 15:45:02.144299 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:02.144244    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-z44f2" event={"ID":"58ea5830-31f8-498b-82c6-3cd8fcb22044","Type":"ContainerStarted","Data":"faf75ef74146568892d4de739b47791cf58d0e6f23d1540803514639e95f4608"}
Sep 08 15:45:02.161729 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:02.161682    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-z44f2" podStartSLOduration=1.161666356 podStartE2EDuration="1.161666356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:02.160619848 +0000 UTC m=+48.784312172" watchObservedRunningTime="2026-09-08 15:45:02.161666356 +0000 UTC m=+48.785358680"
Sep 08 15:45:02.310545 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:02.310468    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-62rfg_b3c368f6-fe5b-4cd8-8d03-0c75a4478a19/serve-healthcheck-canary/0.log"
Sep 08 15:45:02.414750 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:02.414635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:45:02.414827 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:02.414814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:45:02.414878 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:02.414817    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:45:02.414968 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:02.414884    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls podName:c82d6a57-745e-47b9-afd5-b5c466bdf5d6 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:06.414861279 +0000 UTC m=+53.038553599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qvc9" (UID: "c82d6a57-745e-47b9-afd5-b5c466bdf5d6") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:45:02.414968 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:02.414897    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:45:02.414968 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:02.414946    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm podName:c82d6a57-745e-47b9-afd5-b5c466bdf5d6 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:06.414930616 +0000 UTC m=+53.038622933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qvc9" (UID: "c82d6a57-745e-47b9-afd5-b5c466bdf5d6") : configmap "kube-rbac-proxy" not found
Sep 08 15:45:02.515829 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:02.515797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jjj7p\" (UID: \"7ff830b7-be75-496c-8e4e-bf04e35912ee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p"
Sep 08 15:45:02.515961 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:02.515942    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:45:02.516040 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:02.516012    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert podName:7ff830b7-be75-496c-8e4e-bf04e35912ee nodeName:}" failed. No retries permitted until 2026-09-08 15:45:06.515992802 +0000 UTC m=+53.139685111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-jjj7p" (UID: "7ff830b7-be75-496c-8e4e-bf04e35912ee") : secret "networking-console-plugin-cert" not found
Sep 08 15:45:02.912559 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:02.912532    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-vdxpx_593f6dab-17b1-48a1-a4bb-a9164f694d90/kube-storage-version-migrator-operator/0.log"
Sep 08 15:45:03.148741 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:03.148710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-q5cth" event={"ID":"a588199a-f0f0-476f-8806-b158e9539913","Type":"ContainerStarted","Data":"c8c72d8d8ce66d038141095cdece2b89f7e7736d89e497bd3654ace1251ae2f1"}
Sep 08 15:45:03.148904 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:03.148763    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-q5cth" event={"ID":"a588199a-f0f0-476f-8806-b158e9539913","Type":"ContainerStarted","Data":"8144813b36005e98ff8fa26f77fc9b056e766f53ee7107c1177a8e2b527cc6c2"}
Sep 08 15:45:03.726847 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:03.726813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:45:03.727232 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:03.726962    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:45:03.727232 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:03.726982    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5f6df75-6fqds: secret "image-registry-tls" not found
Sep 08 15:45:03.727232 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:03.727036    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls podName:38cac5cb-e76e-4799-a54c-eaf3158d7158 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:19.727022446 +0000 UTC m=+66.350714753 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls") pod "image-registry-74f5f6df75-6fqds" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158") : secret "image-registry-tls" not found
Sep 08 15:45:03.828074 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:03.828037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:45:03.828074 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:03.828074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:45:03.828318 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:03.828113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:45:03.828318 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:03.828145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hckv7\" (UID: \"8c9f9660-e1c4-4433-a181-e8eee81bbe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:45:03.828318 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:03.828212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:45:03.828318 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:03.828266    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 15:45:03.828318 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:03.828308    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle podName:73240d9c-1436-48bd-90b2-a665d22d7006 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:19.828286383 +0000 UTC m=+66.451978705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle") pod "router-default-5f6c69889d-c7hpw" (UID: "73240d9c-1436-48bd-90b2-a665d22d7006") : configmap references non-existent config key: service-ca.crt
Sep 08 15:45:03.828600 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:03.828341    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:45:03.828600 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:03.828371    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 15:45:03.828600 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:03.828387    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls podName:8c9f9660-e1c4-4433-a181-e8eee81bbe2e nodeName:}" failed. No retries permitted until 2026-09-08 15:45:19.828369434 +0000 UTC m=+66.452061748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hckv7" (UID: "8c9f9660-e1c4-4433-a181-e8eee81bbe2e") : secret "samples-operator-tls" not found
Sep 08 15:45:03.828600 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:03.828432    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs podName:73240d9c-1436-48bd-90b2-a665d22d7006 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:19.828420006 +0000 UTC m=+66.452112309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs") pod "router-default-5f6c69889d-c7hpw" (UID: "73240d9c-1436-48bd-90b2-a665d22d7006") : secret "router-metrics-certs-default" not found
Sep 08 15:45:03.828600 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:03.828452    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls podName:280e0b1c-6fa9-4d0a-8769-8589179684a3 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:19.828443468 +0000 UTC m=+66.452135777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-n5fbl" (UID: "280e0b1c-6fa9-4d0a-8769-8589179684a3") : secret "cluster-monitoring-operator-tls" not found
Sep 08 15:45:03.828600 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:03.828558    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:45:03.828600 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:03.828600    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls podName:b43463d7-de74-46e8-af40-2db7ab47f8ff nodeName:}" failed. No retries permitted until 2026-09-08 15:45:19.828591404 +0000 UTC m=+66.452283706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls") pod "dns-default-6vlx8" (UID: "b43463d7-de74-46e8-af40-2db7ab47f8ff") : secret "dns-default-metrics-tls" not found
Sep 08 15:45:06.457354 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:06.457319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:45:06.457750 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:06.457417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:45:06.457750 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:06.457585    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:45:06.457750 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:06.457648    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls podName:c82d6a57-745e-47b9-afd5-b5c466bdf5d6 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:14.457629336 +0000 UTC m=+61.081321638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qvc9" (UID: "c82d6a57-745e-47b9-afd5-b5c466bdf5d6") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:45:06.457970 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:06.457947    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:45:06.558704 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:06.558675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jjj7p\" (UID: \"7ff830b7-be75-496c-8e4e-bf04e35912ee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p"
Sep 08 15:45:06.558855 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:06.558836    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:45:06.558915 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:06.558906    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert podName:7ff830b7-be75-496c-8e4e-bf04e35912ee nodeName:}" failed. No retries permitted until 2026-09-08 15:45:14.558888208 +0000 UTC m=+61.182580514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-jjj7p" (UID: "7ff830b7-be75-496c-8e4e-bf04e35912ee") : secret "networking-console-plugin-cert" not found
Sep 08 15:45:08.154205 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:08.154174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:45:08.154658 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:08.154225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:45:08.154658 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:08.154507    2581 scope.go:117] "RemoveContainer" containerID="ef9ea3b00e1aea22d3ca03c4125df6e762d9ed899a9842d30c946bfae270f4de"
Sep 08 15:45:08.154658 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:08.154634    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-fzq5z_openshift-console-operator(a27cac25-7d24-41e2-bb79-a2412491978b)\"" pod="openshift-console-operator/console-operator-85b454874b-fzq5z" podUID="a27cac25-7d24-41e2-bb79-a2412491978b"
Sep 08 15:45:12.052743 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:12.052713    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2wc2"
Sep 08 15:45:12.083191 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:12.083149    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-q5cth" podStartSLOduration=10.600257111 podStartE2EDuration="14.083133438s" totalImagesPullingTime="3.482876327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:03.176230559 +0000 UTC m=+49.799922882" watchObservedRunningTime="2026-09-08 15:45:12.083133438 +0000 UTC m=+58.706825761"
Sep 08 15:45:14.526065 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:14.526029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:45:14.528230 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:14.528205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c82d6a57-745e-47b9-afd5-b5c466bdf5d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qvc9\" (UID: \"c82d6a57-745e-47b9-afd5-b5c466bdf5d6\") " pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:45:14.626933 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:14.626904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jjj7p\" (UID: \"7ff830b7-be75-496c-8e4e-bf04e35912ee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p"
Sep 08 15:45:14.629050 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:14.629028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ff830b7-be75-496c-8e4e-bf04e35912ee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jjj7p\" (UID: \"7ff830b7-be75-496c-8e4e-bf04e35912ee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p"
Sep 08 15:45:14.662667 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:14.662643    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rhb4d\""
Sep 08 15:45:14.671379 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:14.671361    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qvc9"
Sep 08 15:45:14.752971 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:14.752948    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-b4msn\""
Sep 08 15:45:14.760892 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:14.760869    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p"
Sep 08 15:45:14.959297 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:14.959271    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qvc9"]
Sep 08 15:45:14.961746 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:14.961720    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82d6a57_745e_47b9_afd5_b5c466bdf5d6.slice/crio-39003b029f45eab47f1b909d465e1d3b31d211eedc9906fe92768842fd21ccb0 WatchSource:0}: Error finding container 39003b029f45eab47f1b909d465e1d3b31d211eedc9906fe92768842fd21ccb0: Status 404 returned error can't find the container with id 39003b029f45eab47f1b909d465e1d3b31d211eedc9906fe92768842fd21ccb0
Sep 08 15:45:15.047828 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:15.047804    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-jjj7p"]
Sep 08 15:45:15.051731 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:15.051700    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff830b7_be75_496c_8e4e_bf04e35912ee.slice/crio-9b2ca2b4e066e7dc2379e473d9a7c507af67b08f5b6f9c9f9c68e3c421a40583 WatchSource:0}: Error finding container 9b2ca2b4e066e7dc2379e473d9a7c507af67b08f5b6f9c9f9c68e3c421a40583: Status 404 returned error can't find the container with id 9b2ca2b4e066e7dc2379e473d9a7c507af67b08f5b6f9c9f9c68e3c421a40583
Sep 08 15:45:15.187571 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:15.187539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qvc9" event={"ID":"c82d6a57-745e-47b9-afd5-b5c466bdf5d6","Type":"ContainerStarted","Data":"d13695e0cc358b23333b38578b3a9ae4982fa8a275c9f22e4a39003658166767"}
Sep 08 15:45:15.187693 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:15.187598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qvc9" event={"ID":"c82d6a57-745e-47b9-afd5-b5c466bdf5d6","Type":"ContainerStarted","Data":"39003b029f45eab47f1b909d465e1d3b31d211eedc9906fe92768842fd21ccb0"}
Sep 08 15:45:15.188814 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:15.188789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p" event={"ID":"7ff830b7-be75-496c-8e4e-bf04e35912ee","Type":"ContainerStarted","Data":"9b2ca2b4e066e7dc2379e473d9a7c507af67b08f5b6f9c9f9c68e3c421a40583"}
Sep 08 15:45:16.194945 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:16.194907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qvc9" event={"ID":"c82d6a57-745e-47b9-afd5-b5c466bdf5d6","Type":"ContainerStarted","Data":"0e973ed1bdf8f88befd542895c5cbb38f02a4b65c61ba9018970f66d21dd66d2"}
Sep 08 15:45:17.200673 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:17.200637    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p" event={"ID":"7ff830b7-be75-496c-8e4e-bf04e35912ee","Type":"ContainerStarted","Data":"5d5202ed68af1907f7faf3f8f664f08475d17578fca1c3af7770791f15ecb3f3"}
Sep 08 15:45:17.220826 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:17.220770    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-jjj7p" podStartSLOduration=17.835682212000002 podStartE2EDuration="19.220754479s" totalImagesPullingTime="1.385072267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:17.219520764 +0000 UTC m=+63.843213089" watchObservedRunningTime="2026-09-08 15:45:17.220754479 +0000 UTC m=+63.844446804"
Sep 08 15:45:18.205688 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:18.205652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qvc9" event={"ID":"c82d6a57-745e-47b9-afd5-b5c466bdf5d6","Type":"ContainerStarted","Data":"2cc66bb58191f1a736dcde4b2473fa2454564c09fab68e91522e9b9402e5cf34"}
Sep 08 15:45:18.224060 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:18.224016    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7qvc9" podStartSLOduration=17.949021331 podStartE2EDuration="20.224003056s" totalImagesPullingTime="2.274981725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:44:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:18.223456003 +0000 UTC m=+64.847148322" watchObservedRunningTime="2026-09-08 15:45:18.224003056 +0000 UTC m=+64.847695379"
Sep 08 15:45:19.570330 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.570283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs\") pod \"network-metrics-daemon-nkn7w\" (UID: \"996123f7-ff5c-4ffd-8a4d-ebfb862d664e\") " pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:45:19.572900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.572884    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 15:45:19.582935 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.582904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996123f7-ff5c-4ffd-8a4d-ebfb862d664e-metrics-certs\") pod \"network-metrics-daemon-nkn7w\" (UID: \"996123f7-ff5c-4ffd-8a4d-ebfb862d664e\") " pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:45:19.671352 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.671315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:45:19.671514 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.671431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bxz\" (UniqueName: \"kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz\") pod \"network-check-target-8r2bg\" (UID: \"da5fbd85-bda7-46ab-98ea-255e07a0fb82\") " pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:45:19.673897 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.673879    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 15:45:19.673971 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.673879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bxz\" (UniqueName: \"kubernetes.io/projected/da5fbd85-bda7-46ab-98ea-255e07a0fb82-kube-api-access-k8bxz\") pod \"network-check-target-8r2bg\" (UID: \"da5fbd85-bda7-46ab-98ea-255e07a0fb82\") " pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:45:19.683673 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.683652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e92972bf-2b13-43de-bc6c-ccf50050fa24-original-pull-secret\") pod \"global-pull-secret-syncer-xbnhj\" (UID: \"e92972bf-2b13-43de-bc6c-ccf50050fa24\") " pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:45:19.716196 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.716176    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-78hmt\""
Sep 08 15:45:19.724871 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.724856    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkn7w"
Sep 08 15:45:19.772863 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.772833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:45:19.776093 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.776075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls\") pod \"image-registry-74f5f6df75-6fqds\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") " pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:45:19.807124 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.807103    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xkdlq\""
Sep 08 15:45:19.809813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.809792    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbnhj"
Sep 08 15:45:19.815696 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.815684    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:45:19.874484 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.874411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hckv7\" (UID: \"8c9f9660-e1c4-4433-a181-e8eee81bbe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:45:19.874484 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.874479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:45:19.874618 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.874580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:45:19.874618 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.874604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:45:19.874696 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.874635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:45:19.875298 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.875274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73240d9c-1436-48bd-90b2-a665d22d7006-service-ca-bundle\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:45:19.877099 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.877062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/280e0b1c-6fa9-4d0a-8769-8589179684a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-n5fbl\" (UID: \"280e0b1c-6fa9-4d0a-8769-8589179684a3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:45:19.877187 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.877062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c9f9660-e1c4-4433-a181-e8eee81bbe2e-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hckv7\" (UID: \"8c9f9660-e1c4-4433-a181-e8eee81bbe2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:45:19.877187 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.877109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73240d9c-1436-48bd-90b2-a665d22d7006-metrics-certs\") pod \"router-default-5f6c69889d-c7hpw\" (UID: \"73240d9c-1436-48bd-90b2-a665d22d7006\") " pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:45:19.877355 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.877336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b43463d7-de74-46e8-af40-2db7ab47f8ff-metrics-tls\") pod \"dns-default-6vlx8\" (UID: \"b43463d7-de74-46e8-af40-2db7ab47f8ff\") " pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:45:19.895175 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.895156    2581 scope.go:117] "RemoveContainer" containerID="ef9ea3b00e1aea22d3ca03c4125df6e762d9ed899a9842d30c946bfae270f4de"
Sep 08 15:45:19.933148 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.933129    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6lw5f\""
Sep 08 15:45:19.941197 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.941177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:45:19.993559 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.993471    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nkn7w"]
Sep 08 15:45:19.998014 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:19.997933    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p7plt\""
Sep 08 15:45:20.000091 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:19.999944    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996123f7_ff5c_4ffd_8a4d_ebfb862d664e.slice/crio-8a52c395b16d73e4bb4d6c8b2bad394ca55aeb30d0305c288471193748a890d4 WatchSource:0}: Error finding container 8a52c395b16d73e4bb4d6c8b2bad394ca55aeb30d0305c288471193748a890d4: Status 404 returned error can't find the container with id 8a52c395b16d73e4bb4d6c8b2bad394ca55aeb30d0305c288471193748a890d4
Sep 08 15:45:20.006469 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.006444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"
Sep 08 15:45:20.009462 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.009444    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-d8bb6\""
Sep 08 15:45:20.018440 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.018405    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"
Sep 08 15:45:20.036796 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.036749    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-l2fz9\""
Sep 08 15:45:20.045162 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.045019    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:45:20.080658 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.080459    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qhh8b\""
Sep 08 15:45:20.082940 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.082844    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xbnhj"]
Sep 08 15:45:20.086597 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:20.086573    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92972bf_2b13_43de_bc6c_ccf50050fa24.slice/crio-18fc7f4e6d58800fe51cd0c30d44b7f628c456d028582c439c86a30b85c8740e WatchSource:0}: Error finding container 18fc7f4e6d58800fe51cd0c30d44b7f628c456d028582c439c86a30b85c8740e: Status 404 returned error can't find the container with id 18fc7f4e6d58800fe51cd0c30d44b7f628c456d028582c439c86a30b85c8740e
Sep 08 15:45:20.088634 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.088614    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:45:20.103232 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.100446    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8r2bg"]
Sep 08 15:45:20.107700 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:20.107661    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5fbd85_bda7_46ab_98ea_255e07a0fb82.slice/crio-515576a86bdb5a940536522e2c71fe86f8827ef47120f26d9c45933e66961f03 WatchSource:0}: Error finding container 515576a86bdb5a940536522e2c71fe86f8827ef47120f26d9c45933e66961f03: Status 404 returned error can't find the container with id 515576a86bdb5a940536522e2c71fe86f8827ef47120f26d9c45933e66961f03
Sep 08 15:45:20.214061 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.214035    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 15:45:20.214177 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.214147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-fzq5z" event={"ID":"a27cac25-7d24-41e2-bb79-a2412491978b","Type":"ContainerStarted","Data":"0a63377ab5027dffa44b619a6208d8bc5e5abcb80f6e76865237b49c42ad6d58"}
Sep 08 15:45:20.214502 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.214483    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:45:20.215434 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.215414    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xbnhj" event={"ID":"e92972bf-2b13-43de-bc6c-ccf50050fa24","Type":"ContainerStarted","Data":"18fc7f4e6d58800fe51cd0c30d44b7f628c456d028582c439c86a30b85c8740e"}
Sep 08 15:45:20.216893 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.216872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8r2bg" event={"ID":"da5fbd85-bda7-46ab-98ea-255e07a0fb82","Type":"ContainerStarted","Data":"06ccb7ab430ac26bae5b13f00eb66f6177113d5493b795ea918812b71be53fa3"}
Sep 08 15:45:20.216995 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.216910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8r2bg" event={"ID":"da5fbd85-bda7-46ab-98ea-255e07a0fb82","Type":"ContainerStarted","Data":"515576a86bdb5a940536522e2c71fe86f8827ef47120f26d9c45933e66961f03"}
Sep 08 15:45:20.216995 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.216974    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:45:20.218285 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.218144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nkn7w" event={"ID":"996123f7-ff5c-4ffd-8a4d-ebfb862d664e","Type":"ContainerStarted","Data":"8a52c395b16d73e4bb4d6c8b2bad394ca55aeb30d0305c288471193748a890d4"}
Sep 08 15:45:20.219338 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.219321    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74f5f6df75-6fqds"]
Sep 08 15:45:20.222795 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:20.222775    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cac5cb_e76e_4799_a54c_eaf3158d7158.slice/crio-d7b1c4059b8a0d84421a1cd144fcd048baf79aef2f6bccb87e9a4f99d6cc5d29 WatchSource:0}: Error finding container d7b1c4059b8a0d84421a1cd144fcd048baf79aef2f6bccb87e9a4f99d6cc5d29: Status 404 returned error can't find the container with id d7b1c4059b8a0d84421a1cd144fcd048baf79aef2f6bccb87e9a4f99d6cc5d29
Sep 08 15:45:20.233458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.233413    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-fzq5z" podStartSLOduration=42.228392516 podStartE2EDuration="50.233373602s" totalImagesPullingTime="8.004981086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:20.232046648 +0000 UTC m=+66.855738971" watchObservedRunningTime="2026-09-08 15:45:20.233373602 +0000 UTC m=+66.857065928"
Sep 08 15:45:20.246764 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.246724    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8r2bg" podStartSLOduration=65.246706576 podStartE2EDuration="1m5.246706576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:20.246502524 +0000 UTC m=+66.870194848" watchObservedRunningTime="2026-09-08 15:45:20.246706576 +0000 UTC m=+66.870398906"
Sep 08 15:45:20.291258 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.291231    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7"]
Sep 08 15:45:20.314291 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.314268    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl"]
Sep 08 15:45:20.317181 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:20.317148    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280e0b1c_6fa9_4d0a_8769_8589179684a3.slice/crio-8a3541292a0705b0b80f1ce450b74fe042ac566b6ca51a31f46ff394d2240773 WatchSource:0}: Error finding container 8a3541292a0705b0b80f1ce450b74fe042ac566b6ca51a31f46ff394d2240773: Status 404 returned error can't find the container with id 8a3541292a0705b0b80f1ce450b74fe042ac566b6ca51a31f46ff394d2240773
Sep 08 15:45:20.345265 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.345244    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f6c69889d-c7hpw"]
Sep 08 15:45:20.347842 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:20.347821    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73240d9c_1436_48bd_90b2_a665d22d7006.slice/crio-aa96889b2c0e0ba2a2857f04bf746b848c5f2874c5bcfa7772013f4713fa4ec0 WatchSource:0}: Error finding container aa96889b2c0e0ba2a2857f04bf746b848c5f2874c5bcfa7772013f4713fa4ec0: Status 404 returned error can't find the container with id aa96889b2c0e0ba2a2857f04bf746b848c5f2874c5bcfa7772013f4713fa4ec0
Sep 08 15:45:20.380864 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.380843    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-fzq5z"
Sep 08 15:45:20.391267 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:20.391242    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6vlx8"]
Sep 08 15:45:20.394656 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:20.394630    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43463d7_de74_46e8_af40_2db7ab47f8ff.slice/crio-0d25a6eb081328119c7e57a8cdabcfd48ccf2ed13b2df4ef187f938951112b3d WatchSource:0}: Error finding container 0d25a6eb081328119c7e57a8cdabcfd48ccf2ed13b2df4ef187f938951112b3d: Status 404 returned error can't find the container with id 0d25a6eb081328119c7e57a8cdabcfd48ccf2ed13b2df4ef187f938951112b3d
Sep 08 15:45:21.226074 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:21.225994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl" event={"ID":"280e0b1c-6fa9-4d0a-8769-8589179684a3","Type":"ContainerStarted","Data":"8a3541292a0705b0b80f1ce450b74fe042ac566b6ca51a31f46ff394d2240773"}
Sep 08 15:45:21.227845 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:21.227820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7" event={"ID":"8c9f9660-e1c4-4433-a181-e8eee81bbe2e","Type":"ContainerStarted","Data":"d5241c921845610c06dd1a762998e460da0f8bafd6f451b66236d8edfc1057c4"}
Sep 08 15:45:21.230986 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:21.230428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f6c69889d-c7hpw" event={"ID":"73240d9c-1436-48bd-90b2-a665d22d7006","Type":"ContainerStarted","Data":"0415caafce76d729755af4ed2358c2d906f72ac7b6d52b2d4524dd521aba532b"}
Sep 08 15:45:21.230986 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:21.230471    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f6c69889d-c7hpw" event={"ID":"73240d9c-1436-48bd-90b2-a665d22d7006","Type":"ContainerStarted","Data":"aa96889b2c0e0ba2a2857f04bf746b848c5f2874c5bcfa7772013f4713fa4ec0"}
Sep 08 15:45:21.234640 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:21.234617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f5f6df75-6fqds" event={"ID":"38cac5cb-e76e-4799-a54c-eaf3158d7158","Type":"ContainerStarted","Data":"b86fca7eed3457fb0ed238932c80ab52ad7e5852bd00bde4fda8588b7e1656b4"}
Sep 08 15:45:21.234735 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:21.234666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f5f6df75-6fqds" event={"ID":"38cac5cb-e76e-4799-a54c-eaf3158d7158","Type":"ContainerStarted","Data":"d7b1c4059b8a0d84421a1cd144fcd048baf79aef2f6bccb87e9a4f99d6cc5d29"}
Sep 08 15:45:21.234735 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:21.234695    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:45:21.236725 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:21.236656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vlx8" event={"ID":"b43463d7-de74-46e8-af40-2db7ab47f8ff","Type":"ContainerStarted","Data":"0d25a6eb081328119c7e57a8cdabcfd48ccf2ed13b2df4ef187f938951112b3d"}
Sep 08 15:45:21.250849 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:21.250667    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5f6c69889d-c7hpw" podStartSLOduration=51.25065087 podStartE2EDuration="51.25065087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:21.250262161 +0000 UTC m=+67.873954513" watchObservedRunningTime="2026-09-08 15:45:21.25065087 +0000 UTC m=+67.874343195"
Sep 08 15:45:21.271465 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:21.271076    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74f5f6df75-6fqds" podStartSLOduration=66.271059223 podStartE2EDuration="1m6.271059223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:21.269510012 +0000 UTC m=+67.893202338" watchObservedRunningTime="2026-09-08 15:45:21.271059223 +0000 UTC m=+67.894751548"
Sep 08 15:45:22.045386 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:22.045309    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:45:22.048380 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:22.048354    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:45:22.244156 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:22.243538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nkn7w" event={"ID":"996123f7-ff5c-4ffd-8a4d-ebfb862d664e","Type":"ContainerStarted","Data":"28854d491495f6cc93551c170dd513d4f9874bd52ec7e06a3b7ddebd25fb6742"}
Sep 08 15:45:22.244584 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:22.244435    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:45:22.246009 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:22.245893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5f6c69889d-c7hpw"
Sep 08 15:45:24.011879 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.011846    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"]
Sep 08 15:45:24.028533 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.028502    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"]
Sep 08 15:45:24.028655 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.028638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"
Sep 08 15:45:24.047967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.047930    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 15:45:24.049099 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.049074    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 15:45:24.049223 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.049134    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 15:45:24.049345 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.049327    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 15:45:24.050337 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.050319    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc"]
Sep 08 15:45:24.050472 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.050455    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.059215 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.059197    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 15:45:24.065023 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.065006    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 15:45:24.065234 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.065219    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 15:45:24.065296 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.065241    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 15:45:24.066490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.066471    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"]
Sep 08 15:45:24.066577 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.066504    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc"]
Sep 08 15:45:24.066577 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.066515    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"]
Sep 08 15:45:24.066658 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.066604    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc"
Sep 08 15:45:24.085848 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.085832    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xdzw4\""
Sep 08 15:45:24.088999 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.088982    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 08 15:45:24.151296 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.151265    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-95ztd"]
Sep 08 15:45:24.176134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.176108    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-95ztd"
Sep 08 15:45:24.180774 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.180755    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 15:45:24.180866 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.180794    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-tqjp7\""
Sep 08 15:45:24.181277 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.181262    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 15:45:24.190163 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.190141    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-95ztd"]
Sep 08 15:45:24.214435 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.214414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e4cc517-f21f-4655-8e6a-6c30fd9377f4-tmp\") pod \"klusterlet-addon-workmgr-74998dcbb7-wl2tp\" (UID: \"6e4cc517-f21f-4655-8e6a-6c30fd9377f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"
Sep 08 15:45:24.214546 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.214452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/21a82ef8-d614-4ead-9b72-5302ca6391ad-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.214546 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.214483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/21a82ef8-d614-4ead-9b72-5302ca6391ad-hub\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.214657 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.214563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/21a82ef8-d614-4ead-9b72-5302ca6391ad-ca\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.214657 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.214594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a82ef8-d614-4ead-9b72-5302ca6391ad-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.214657 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.214644    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgnjp\" (UniqueName: \"kubernetes.io/projected/e69560cf-bac8-4760-abc6-01498328b7ba-kube-api-access-qgnjp\") pod \"managed-serviceaccount-addon-agent-645857776b-2dvtc\" (UID: \"e69560cf-bac8-4760-abc6-01498328b7ba\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc"
Sep 08 15:45:24.214808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.214698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbj6x\" (UniqueName: \"kubernetes.io/projected/6e4cc517-f21f-4655-8e6a-6c30fd9377f4-kube-api-access-kbj6x\") pod \"klusterlet-addon-workmgr-74998dcbb7-wl2tp\" (UID: \"6e4cc517-f21f-4655-8e6a-6c30fd9377f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"
Sep 08 15:45:24.214808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.214764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t6bk\" (UniqueName: \"kubernetes.io/projected/21a82ef8-d614-4ead-9b72-5302ca6391ad-kube-api-access-6t6bk\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.214808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.214793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e69560cf-bac8-4760-abc6-01498328b7ba-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-645857776b-2dvtc\" (UID: \"e69560cf-bac8-4760-abc6-01498328b7ba\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc"
Sep 08 15:45:24.214941 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.214816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21a82ef8-d614-4ead-9b72-5302ca6391ad-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.214941 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.214896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6e4cc517-f21f-4655-8e6a-6c30fd9377f4-klusterlet-config\") pod \"klusterlet-addon-workmgr-74998dcbb7-wl2tp\" (UID: \"6e4cc517-f21f-4655-8e6a-6c30fd9377f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"
Sep 08 15:45:24.315622 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.315541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnjp\" (UniqueName: \"kubernetes.io/projected/e69560cf-bac8-4760-abc6-01498328b7ba-kube-api-access-qgnjp\") pod \"managed-serviceaccount-addon-agent-645857776b-2dvtc\" (UID: \"e69560cf-bac8-4760-abc6-01498328b7ba\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc"
Sep 08 15:45:24.315622 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.315590    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbj6x\" (UniqueName: \"kubernetes.io/projected/6e4cc517-f21f-4655-8e6a-6c30fd9377f4-kube-api-access-kbj6x\") pod \"klusterlet-addon-workmgr-74998dcbb7-wl2tp\" (UID: \"6e4cc517-f21f-4655-8e6a-6c30fd9377f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"
Sep 08 15:45:24.315834 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.315627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klt86\" (UniqueName: \"kubernetes.io/projected/624ad5f8-22ca-4629-aba5-214f01af0f7d-kube-api-access-klt86\") pod \"downloads-85d98c998f-95ztd\" (UID: \"624ad5f8-22ca-4629-aba5-214f01af0f7d\") " pod="openshift-console/downloads-85d98c998f-95ztd"
Sep 08 15:45:24.315834 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.315661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t6bk\" (UniqueName: \"kubernetes.io/projected/21a82ef8-d614-4ead-9b72-5302ca6391ad-kube-api-access-6t6bk\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.315834 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.315691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e69560cf-bac8-4760-abc6-01498328b7ba-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-645857776b-2dvtc\" (UID: \"e69560cf-bac8-4760-abc6-01498328b7ba\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc"
Sep 08 15:45:24.315834 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.315717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21a82ef8-d614-4ead-9b72-5302ca6391ad-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.315834 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.315758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6e4cc517-f21f-4655-8e6a-6c30fd9377f4-klusterlet-config\") pod \"klusterlet-addon-workmgr-74998dcbb7-wl2tp\" (UID: \"6e4cc517-f21f-4655-8e6a-6c30fd9377f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"
Sep 08 15:45:24.315834 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.315793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e4cc517-f21f-4655-8e6a-6c30fd9377f4-tmp\") pod \"klusterlet-addon-workmgr-74998dcbb7-wl2tp\" (UID: \"6e4cc517-f21f-4655-8e6a-6c30fd9377f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"
Sep 08 15:45:24.315834 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.315830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/21a82ef8-d614-4ead-9b72-5302ca6391ad-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.316175 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.315862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/21a82ef8-d614-4ead-9b72-5302ca6391ad-hub\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.316175 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.315907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/21a82ef8-d614-4ead-9b72-5302ca6391ad-ca\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.316175 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.315939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a82ef8-d614-4ead-9b72-5302ca6391ad-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.316490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.316439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e4cc517-f21f-4655-8e6a-6c30fd9377f4-tmp\") pod \"klusterlet-addon-workmgr-74998dcbb7-wl2tp\" (UID: \"6e4cc517-f21f-4655-8e6a-6c30fd9377f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"
Sep 08 15:45:24.316758 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.316721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a82ef8-d614-4ead-9b72-5302ca6391ad-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.318967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.318941    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/21a82ef8-d614-4ead-9b72-5302ca6391ad-hub\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.319073 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.319038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6e4cc517-f21f-4655-8e6a-6c30fd9377f4-klusterlet-config\") pod \"klusterlet-addon-workmgr-74998dcbb7-wl2tp\" (UID: \"6e4cc517-f21f-4655-8e6a-6c30fd9377f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"
Sep 08 15:45:24.319163 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.319140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21a82ef8-d614-4ead-9b72-5302ca6391ad-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.319261 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.319236    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/21a82ef8-d614-4ead-9b72-5302ca6391ad-ca\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.319365 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.319306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/21a82ef8-d614-4ead-9b72-5302ca6391ad-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.320923 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.320904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e69560cf-bac8-4760-abc6-01498328b7ba-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-645857776b-2dvtc\" (UID: \"e69560cf-bac8-4760-abc6-01498328b7ba\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc"
Sep 08 15:45:24.323645 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.323622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbj6x\" (UniqueName: \"kubernetes.io/projected/6e4cc517-f21f-4655-8e6a-6c30fd9377f4-kube-api-access-kbj6x\") pod \"klusterlet-addon-workmgr-74998dcbb7-wl2tp\" (UID: \"6e4cc517-f21f-4655-8e6a-6c30fd9377f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"
Sep 08 15:45:24.323972 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.323946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t6bk\" (UniqueName: \"kubernetes.io/projected/21a82ef8-d614-4ead-9b72-5302ca6391ad-kube-api-access-6t6bk\") pod \"cluster-proxy-proxy-agent-6678c77956-4fvpq\" (UID: \"21a82ef8-d614-4ead-9b72-5302ca6391ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.326615 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.326596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnjp\" (UniqueName: \"kubernetes.io/projected/e69560cf-bac8-4760-abc6-01498328b7ba-kube-api-access-qgnjp\") pod \"managed-serviceaccount-addon-agent-645857776b-2dvtc\" (UID: \"e69560cf-bac8-4760-abc6-01498328b7ba\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc"
Sep 08 15:45:24.339491 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.339469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"
Sep 08 15:45:24.365838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.365813    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"
Sep 08 15:45:24.373957 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.373941    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc"
Sep 08 15:45:24.416804 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.416782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klt86\" (UniqueName: \"kubernetes.io/projected/624ad5f8-22ca-4629-aba5-214f01af0f7d-kube-api-access-klt86\") pod \"downloads-85d98c998f-95ztd\" (UID: \"624ad5f8-22ca-4629-aba5-214f01af0f7d\") " pod="openshift-console/downloads-85d98c998f-95ztd"
Sep 08 15:45:24.427027 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.427002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klt86\" (UniqueName: \"kubernetes.io/projected/624ad5f8-22ca-4629-aba5-214f01af0f7d-kube-api-access-klt86\") pod \"downloads-85d98c998f-95ztd\" (UID: \"624ad5f8-22ca-4629-aba5-214f01af0f7d\") " pod="openshift-console/downloads-85d98c998f-95ztd"
Sep 08 15:45:24.505883 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.505835    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-95ztd"
Sep 08 15:45:24.875618 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.875590    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58ff85d5d7-hntxc"]
Sep 08 15:45:24.898115 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.898086    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58ff85d5d7-hntxc"]
Sep 08 15:45:24.898263 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.898194    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:24.900768 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.900735    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 15:45:24.900886 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.900782    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p89kc\""
Sep 08 15:45:24.900886 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.900816    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 15:45:24.900886 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.900814    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 15:45:24.901060 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.900888    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 15:45:24.901108 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:24.901054    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 15:45:25.023335 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.023300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-oauth-serving-cert\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.023772 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.023353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-service-ca\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.023772 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.023381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59pk4\" (UniqueName: \"kubernetes.io/projected/d5b09ec7-9a0e-4da3-bed1-15b273e21318-kube-api-access-59pk4\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.023772 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.023479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-oauth-config\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.023772 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.023509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-serving-cert\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.023772 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.023590    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-config\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.124553 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.124518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-oauth-config\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.124719 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.124566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-serving-cert\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.124719 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.124589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-config\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.124719 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.124653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-oauth-serving-cert\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.124719 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.124684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-service-ca\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.124719 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.124703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59pk4\" (UniqueName: \"kubernetes.io/projected/d5b09ec7-9a0e-4da3-bed1-15b273e21318-kube-api-access-59pk4\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.125886 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.125684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-oauth-serving-cert\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.125886 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.125808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-service-ca\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.125886 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.125808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-config\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.127440 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.127415    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-oauth-config\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.128735 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.128711    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-serving-cert\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.132736 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.132711    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59pk4\" (UniqueName: \"kubernetes.io/projected/d5b09ec7-9a0e-4da3-bed1-15b273e21318-kube-api-access-59pk4\") pod \"console-58ff85d5d7-hntxc\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") " pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:25.209970 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:25.209934    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:26.072778 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.072735    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-95ztd"]
Sep 08 15:45:26.087154 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.087127    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58ff85d5d7-hntxc"]
Sep 08 15:45:26.094448 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.094377    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq"]
Sep 08 15:45:26.111017 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.110986    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc"]
Sep 08 15:45:26.131333 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.131304    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"]
Sep 08 15:45:26.147750 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:26.147665    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624ad5f8_22ca_4629_aba5_214f01af0f7d.slice/crio-7497c4166362c916eacf2847decce4eab698c31053c7e41476e7dddb8579f710 WatchSource:0}: Error finding container 7497c4166362c916eacf2847decce4eab698c31053c7e41476e7dddb8579f710: Status 404 returned error can't find the container with id 7497c4166362c916eacf2847decce4eab698c31053c7e41476e7dddb8579f710
Sep 08 15:45:26.152195 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:26.152167    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b09ec7_9a0e_4da3_bed1_15b273e21318.slice/crio-4227a8564aac57b04ff69d95b47fbdd07b6987f29dd7b7abfcd9898e9f90b9fa WatchSource:0}: Error finding container 4227a8564aac57b04ff69d95b47fbdd07b6987f29dd7b7abfcd9898e9f90b9fa: Status 404 returned error can't find the container with id 4227a8564aac57b04ff69d95b47fbdd07b6987f29dd7b7abfcd9898e9f90b9fa
Sep 08 15:45:26.264030 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.263995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58ff85d5d7-hntxc" event={"ID":"d5b09ec7-9a0e-4da3-bed1-15b273e21318","Type":"ContainerStarted","Data":"4227a8564aac57b04ff69d95b47fbdd07b6987f29dd7b7abfcd9898e9f90b9fa"}
Sep 08 15:45:26.266652 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.266606    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc" event={"ID":"e69560cf-bac8-4760-abc6-01498328b7ba","Type":"ContainerStarted","Data":"f72d47ac2244f9c01a1062e49c26153441eee1bbeefc12d91c00e9eadc79bbe1"}
Sep 08 15:45:26.268890 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.268671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7" event={"ID":"8c9f9660-e1c4-4433-a181-e8eee81bbe2e","Type":"ContainerStarted","Data":"44d55269fa0d9bf9a1020ba353db0f18a8ff5bac52fdcf34ee91f6a4fc14c904"}
Sep 08 15:45:26.270449 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.270354    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp" event={"ID":"6e4cc517-f21f-4655-8e6a-6c30fd9377f4","Type":"ContainerStarted","Data":"ecf2fb4022c2078ed89465df072746fdf49d0c258799d2f6a3c543ef76b8bd52"}
Sep 08 15:45:26.272825 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.272765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-95ztd" event={"ID":"624ad5f8-22ca-4629-aba5-214f01af0f7d","Type":"ContainerStarted","Data":"7497c4166362c916eacf2847decce4eab698c31053c7e41476e7dddb8579f710"}
Sep 08 15:45:26.275188 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.275107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vlx8" event={"ID":"b43463d7-de74-46e8-af40-2db7ab47f8ff","Type":"ContainerStarted","Data":"b1609449fed008181714724b182082b566f92d9831f8fe67d1326bad48a5c136"}
Sep 08 15:45:26.278318 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.278107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq" event={"ID":"21a82ef8-d614-4ead-9b72-5302ca6391ad","Type":"ContainerStarted","Data":"82b32a680268e30d01479c19f3282890b26adc341940c4f0dddb04871ee644cc"}
Sep 08 15:45:26.282104 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.281555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nkn7w" event={"ID":"996123f7-ff5c-4ffd-8a4d-ebfb862d664e","Type":"ContainerStarted","Data":"1c33e4a82f90df0026ce22998ae03e8982f4bf0935e5d1c42f19d09f0a6c5175"}
Sep 08 15:45:26.285731 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.285687    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl" event={"ID":"280e0b1c-6fa9-4d0a-8769-8589179684a3","Type":"ContainerStarted","Data":"0aa8213e0beeae7cbf8d93d7d709d39f57a5cbcd221d1b3a0f3dbcc043d6b920"}
Sep 08 15:45:26.301563 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.301332    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nkn7w" podStartSLOduration=69.733443723 podStartE2EDuration="1m11.301315483s" totalImagesPullingTime="1.56787176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:26.300421707 +0000 UTC m=+72.924114036" watchObservedRunningTime="2026-09-08 15:45:26.301315483 +0000 UTC m=+72.925007808"
Sep 08 15:45:26.327569 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:26.327527    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-n5fbl" podStartSLOduration=50.866312262 podStartE2EDuration="56.327512531s" totalImagesPullingTime="5.461200269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:26.326126828 +0000 UTC m=+72.949819171" watchObservedRunningTime="2026-09-08 15:45:26.327512531 +0000 UTC m=+72.951204852"
Sep 08 15:45:27.305852 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:27.305745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7" event={"ID":"8c9f9660-e1c4-4433-a181-e8eee81bbe2e","Type":"ContainerStarted","Data":"84d700f6d2ce723abed58668690f703df85a60ab1165413b976ef4531a959ab2"}
Sep 08 15:45:27.311353 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:27.311322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xbnhj" event={"ID":"e92972bf-2b13-43de-bc6c-ccf50050fa24","Type":"ContainerStarted","Data":"9e178a7305d2119f6829c49606cddd020c19c5b8dca6bf78abdd372b801e7fbe"}
Sep 08 15:45:27.318551 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:27.318425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vlx8" event={"ID":"b43463d7-de74-46e8-af40-2db7ab47f8ff","Type":"ContainerStarted","Data":"524f07017b16b34ff9fe9d47006b0ba9b593ddb4fa6cab31ef201c574ff3d236"}
Sep 08 15:45:27.326418 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:27.326090    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hckv7" podStartSLOduration=51.885836858 podStartE2EDuration="57.326070773s" totalImagesPullingTime="5.440233915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:27.324078367 +0000 UTC m=+73.947770691" watchObservedRunningTime="2026-09-08 15:45:27.326070773 +0000 UTC m=+73.949763099"
Sep 08 15:45:27.344158 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:27.343472    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6vlx8" podStartSLOduration=34.948481566 podStartE2EDuration="40.343455367s" totalImagesPullingTime="5.394973801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:27.341801685 +0000 UTC m=+73.965494011" watchObservedRunningTime="2026-09-08 15:45:27.343455367 +0000 UTC m=+73.967147695"
Sep 08 15:45:27.358734 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:27.357983    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xbnhj" podStartSLOduration=66.260305197 podStartE2EDuration="1m12.357966299s" totalImagesPullingTime="6.097661102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:27.356830864 +0000 UTC m=+73.980523189" watchObservedRunningTime="2026-09-08 15:45:27.357966299 +0000 UTC m=+73.981658625"
Sep 08 15:45:28.328149 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:28.327272    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:45:36.371168 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:36.371124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq" event={"ID":"21a82ef8-d614-4ead-9b72-5302ca6391ad","Type":"ContainerStarted","Data":"06acef4d928745853d5b24c26367ccffe17a10efa0535be581ec7451159bb0eb"}
Sep 08 15:45:36.371670 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:36.371184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq" event={"ID":"21a82ef8-d614-4ead-9b72-5302ca6391ad","Type":"ContainerStarted","Data":"45ee0956c8f2dafd3247b1256b426e4e788ef352cf1f1fe4ca64d4464ef045a0"}
Sep 08 15:45:36.371670 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:36.371203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq" event={"ID":"21a82ef8-d614-4ead-9b72-5302ca6391ad","Type":"ContainerStarted","Data":"0120f9e98217a451ac1946a6b3093c03f670770e2e1f04e44e3ae5beaa567834"}
Sep 08 15:45:36.373188 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:36.373160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58ff85d5d7-hntxc" event={"ID":"d5b09ec7-9a0e-4da3-bed1-15b273e21318","Type":"ContainerStarted","Data":"46dab661b37e9e1a229fe67586ede4c9d9fe21d288c9a5efa8dbb85635101b85"}
Sep 08 15:45:36.374882 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:36.374852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc" event={"ID":"e69560cf-bac8-4760-abc6-01498328b7ba","Type":"ContainerStarted","Data":"47253a7bbc3b55ef23ac228d74ee6a658b8ba120ee78408317fc689919c0abd1"}
Sep 08 15:45:36.376851 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:36.376824    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp" event={"ID":"6e4cc517-f21f-4655-8e6a-6c30fd9377f4","Type":"ContainerStarted","Data":"ad8e022441bc579e7314770a8674042081754333477675d27404cb44d4cdc638"}
Sep 08 15:45:36.377070 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:36.377042    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"
Sep 08 15:45:36.378576 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:36.378555    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp"
Sep 08 15:45:36.392413 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:36.392350    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6678c77956-4fvpq" podStartSLOduration=4.082640714 podStartE2EDuration="13.392335437s" totalImagesPullingTime="9.309694723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:36.390682305 +0000 UTC m=+83.014374629" watchObservedRunningTime="2026-09-08 15:45:36.392335437 +0000 UTC m=+83.016027761"
Sep 08 15:45:36.406259 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:36.406219    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-645857776b-2dvtc" podStartSLOduration=3.117898909 podStartE2EDuration="12.406207369s" totalImagesPullingTime="9.28830846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:36.404907989 +0000 UTC m=+83.028600339" watchObservedRunningTime="2026-09-08 15:45:36.406207369 +0000 UTC m=+83.029899692"
Sep 08 15:45:36.426246 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:36.426175    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58ff85d5d7-hntxc" podStartSLOduration=3.138281676 podStartE2EDuration="12.426159158s" totalImagesPullingTime="9.287877482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:36.425788493 +0000 UTC m=+83.049480817" watchObservedRunningTime="2026-09-08 15:45:36.426159158 +0000 UTC m=+83.049851483"
Sep 08 15:45:36.442748 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:36.442710    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74998dcbb7-wl2tp" podStartSLOduration=4.131576065 podStartE2EDuration="13.442699696s" totalImagesPullingTime="9.311123631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:36.441858916 +0000 UTC m=+83.065551242" watchObservedRunningTime="2026-09-08 15:45:36.442699696 +0000 UTC m=+83.066392020"
Sep 08 15:45:37.915356 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.915317    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r529b"]
Sep 08 15:45:37.919584 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.919561    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:37.922094 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.921942    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 15:45:37.922094 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.921967    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 15:45:37.922094 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.921980    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 15:45:37.922852 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.922832    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 15:45:37.922852 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.922844    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nkrhj\""
Sep 08 15:45:37.941362 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.941250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9573799b-c0da-416e-a64a-6bbcf1344ee4-sys\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:37.941362 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.941296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmssw\" (UniqueName: \"kubernetes.io/projected/9573799b-c0da-416e-a64a-6bbcf1344ee4-kube-api-access-xmssw\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:37.941362 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.941331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-textfile\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:37.941591 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.941428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:37.941591 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.941469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9573799b-c0da-416e-a64a-6bbcf1344ee4-metrics-client-ca\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:37.941591 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.941508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9573799b-c0da-416e-a64a-6bbcf1344ee4-root\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:37.941591 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.941539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-accelerators-collector-config\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:37.941591 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.941570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-wtmp\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:37.941824 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:37.941596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-tls\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.042273 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.042240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-accelerators-collector-config\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.042458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.042286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-wtmp\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.042458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.042317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-tls\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.042458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.042371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9573799b-c0da-416e-a64a-6bbcf1344ee4-sys\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.042458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.042415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmssw\" (UniqueName: \"kubernetes.io/projected/9573799b-c0da-416e-a64a-6bbcf1344ee4-kube-api-access-xmssw\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.042458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.042450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-textfile\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.042740 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.042500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.042740 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.042537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9573799b-c0da-416e-a64a-6bbcf1344ee4-metrics-client-ca\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.042740 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.042573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9573799b-c0da-416e-a64a-6bbcf1344ee4-root\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.042740 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.042660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9573799b-c0da-416e-a64a-6bbcf1344ee4-root\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.043624 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.043248    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-accelerators-collector-config\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.043779 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.043387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-wtmp\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.043779 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:38.043738    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 08 15:45:38.043899 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:45:38.043795    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-tls podName:9573799b-c0da-416e-a64a-6bbcf1344ee4 nodeName:}" failed. No retries permitted until 2026-09-08 15:45:38.543774707 +0000 UTC m=+85.167467015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-tls") pod "node-exporter-r529b" (UID: "9573799b-c0da-416e-a64a-6bbcf1344ee4") : secret "node-exporter-tls" not found
Sep 08 15:45:38.043951 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.043911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-textfile\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.044005 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.043993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9573799b-c0da-416e-a64a-6bbcf1344ee4-sys\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.044516 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.044429    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9573799b-c0da-416e-a64a-6bbcf1344ee4-metrics-client-ca\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.047171 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.047146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.055520 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.055152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmssw\" (UniqueName: \"kubernetes.io/projected/9573799b-c0da-416e-a64a-6bbcf1344ee4-kube-api-access-xmssw\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.336289 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.335655    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6vlx8"
Sep 08 15:45:38.547335 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.547210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-tls\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.549927 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.549900    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9573799b-c0da-416e-a64a-6bbcf1344ee4-node-exporter-tls\") pod \"node-exporter-r529b\" (UID: \"9573799b-c0da-416e-a64a-6bbcf1344ee4\") " pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:38.832523 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:38.832489    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r529b"
Sep 08 15:45:39.946655 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:39.946619    2581 patch_prober.go:28] interesting pod/image-registry-74f5f6df75-6fqds container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 15:45:39.947103 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:39.946683    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-74f5f6df75-6fqds" podUID="38cac5cb-e76e-4799-a54c-eaf3158d7158" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 15:45:42.248727 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:42.248696    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:45:43.495466 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:43.495429    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58ff85d5d7-hntxc"]
Sep 08 15:45:45.202526 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:45.202484    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9573799b_c0da_416e_a64a_6bbcf1344ee4.slice/crio-a3ab6c69ca77034dab1044b492c1fc79f18c97efc77ed5cf0f0562582d777850 WatchSource:0}: Error finding container a3ab6c69ca77034dab1044b492c1fc79f18c97efc77ed5cf0f0562582d777850: Status 404 returned error can't find the container with id a3ab6c69ca77034dab1044b492c1fc79f18c97efc77ed5cf0f0562582d777850
Sep 08 15:45:45.210358 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:45.210335    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:45:45.429478 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:45.429439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r529b" event={"ID":"9573799b-c0da-416e-a64a-6bbcf1344ee4","Type":"ContainerStarted","Data":"a3ab6c69ca77034dab1044b492c1fc79f18c97efc77ed5cf0f0562582d777850"}
Sep 08 15:45:46.436761 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:46.436719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-95ztd" event={"ID":"624ad5f8-22ca-4629-aba5-214f01af0f7d","Type":"ContainerStarted","Data":"bf4428bdffb88a1acbadbfab652b0bb630a7e318609db982f34999c6a3a18f3d"}
Sep 08 15:45:46.437224 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:46.436877    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-95ztd"
Sep 08 15:45:46.453496 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:46.453431    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-95ztd" podStartSLOduration=3.017058169 podStartE2EDuration="22.45341094s" totalImagesPullingTime="19.436352771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:46.453227359 +0000 UTC m=+93.076919683" watchObservedRunningTime="2026-09-08 15:45:46.45341094 +0000 UTC m=+93.077103281"
Sep 08 15:45:46.454473 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:46.454445    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-95ztd"
Sep 08 15:45:47.444025 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.443987    2581 generic.go:356] "Generic (PLEG): container finished" podID="9573799b-c0da-416e-a64a-6bbcf1344ee4" containerID="81beb8064a48e2f9f2ff2f22bd5b1f0c725a35ca0ce61b95cefb330be0fe196a" exitCode=0
Sep 08 15:45:47.444534 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.444082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r529b" event={"ID":"9573799b-c0da-416e-a64a-6bbcf1344ee4","Type":"ContainerDied","Data":"81beb8064a48e2f9f2ff2f22bd5b1f0c725a35ca0ce61b95cefb330be0fe196a"}
Sep 08 15:45:47.657490 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.657454    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74f5f6df75-6fqds"]
Sep 08 15:45:47.736891 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.736817    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-785c579579-8mtc8"]
Sep 08 15:45:47.759269 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.759189    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.762161 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.762133    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-785c579579-8mtc8"]
Sep 08 15:45:47.768705 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.768681    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 15:45:47.832764 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.832733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-serving-cert\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.832764 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.832771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-oauth-config\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.832998 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.832914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-trusted-ca-bundle\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.832998 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.832946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkzff\" (UniqueName: \"kubernetes.io/projected/88a70cbf-8314-414d-81a9-f4bea4326cfc-kube-api-access-pkzff\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.832998 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.832995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-config\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.833136 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.833034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-service-ca\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.833136 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.833097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-oauth-serving-cert\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.933698 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.933663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-trusted-ca-bundle\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.933871 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.933703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkzff\" (UniqueName: \"kubernetes.io/projected/88a70cbf-8314-414d-81a9-f4bea4326cfc-kube-api-access-pkzff\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.933871 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.933831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-config\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.933985 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.933873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-service-ca\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.933985 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.933910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-oauth-serving-cert\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.934087 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.933981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-serving-cert\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.934087 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.934014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-oauth-config\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.934657 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.934628    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-service-ca\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.934784 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.934675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-trusted-ca-bundle\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.934944 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.934913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-oauth-serving-cert\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.935242 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.935219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-config\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.937199 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.937180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-oauth-config\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.937488 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.937457    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-serving-cert\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:47.956304 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:47.956278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkzff\" (UniqueName: \"kubernetes.io/projected/88a70cbf-8314-414d-81a9-f4bea4326cfc-kube-api-access-pkzff\") pod \"console-785c579579-8mtc8\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") " pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:48.072330 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:48.072239    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:48.397567 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:48.397490    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-785c579579-8mtc8"]
Sep 08 15:45:48.400124 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:45:48.400088    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a70cbf_8314_414d_81a9_f4bea4326cfc.slice/crio-63f2245aac64409a6940c3e6c39e9054578b616e9c9885cc688ba29ffd6088ad WatchSource:0}: Error finding container 63f2245aac64409a6940c3e6c39e9054578b616e9c9885cc688ba29ffd6088ad: Status 404 returned error can't find the container with id 63f2245aac64409a6940c3e6c39e9054578b616e9c9885cc688ba29ffd6088ad
Sep 08 15:45:48.453481 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:48.453436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r529b" event={"ID":"9573799b-c0da-416e-a64a-6bbcf1344ee4","Type":"ContainerStarted","Data":"a14e8dce19487c43c26be05905b507357959fc63617d3768087bc7bd519e13e8"}
Sep 08 15:45:48.453865 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:48.453512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r529b" event={"ID":"9573799b-c0da-416e-a64a-6bbcf1344ee4","Type":"ContainerStarted","Data":"e57409c5c7d8de3e256224c5f2998818ccac2ba25130dedf66c70ef2f25cb3e7"}
Sep 08 15:45:48.456109 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:48.456086    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-785c579579-8mtc8" event={"ID":"88a70cbf-8314-414d-81a9-f4bea4326cfc","Type":"ContainerStarted","Data":"63f2245aac64409a6940c3e6c39e9054578b616e9c9885cc688ba29ffd6088ad"}
Sep 08 15:45:48.480652 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:48.480605    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r529b" podStartSLOduration=10.031837493 podStartE2EDuration="11.480587181s" totalImagesPullingTime="1.364510311s" totalInitContainerRuntime="84.239377ms" isStatefulPod=true podCreationTimestamp="2026-09-08 15:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:48.478200277 +0000 UTC m=+95.101892620" watchObservedRunningTime="2026-09-08 15:45:48.480587181 +0000 UTC m=+95.104279506"
Sep 08 15:45:49.462790 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:49.462748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-785c579579-8mtc8" event={"ID":"88a70cbf-8314-414d-81a9-f4bea4326cfc","Type":"ContainerStarted","Data":"c8017a793bc42ccaceeaf15afcbc915340ad9d7f7dc70cad7175ca94df0a09bf"}
Sep 08 15:45:51.239466 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:51.239438    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8r2bg"
Sep 08 15:45:51.264407 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:51.264339    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-785c579579-8mtc8" podStartSLOduration=4.264324175 podStartE2EDuration="4.264324175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:49.480234175 +0000 UTC m=+96.103926522" watchObservedRunningTime="2026-09-08 15:45:51.264324175 +0000 UTC m=+97.888016532"
Sep 08 15:45:58.073122 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:58.073087    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:58.073122 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:58.073137    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:58.077879 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:58.077856    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:45:58.510497 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:45:58.510472    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:46:08.518009 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.517968    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58ff85d5d7-hntxc" podUID="d5b09ec7-9a0e-4da3-bed1-15b273e21318" containerName="console" containerID="cri-o://46dab661b37e9e1a229fe67586ede4c9d9fe21d288c9a5efa8dbb85635101b85" gracePeriod=15
Sep 08 15:46:08.548173 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.548143    2581 generic.go:356] "Generic (PLEG): container finished" podID="d818a8bd-49ec-4d08-9602-78e859ddf43e" containerID="2b00d570ce009527d5f71c4c5ab8b88b1100ea427755a2333ea6060d024c0605" exitCode=0
Sep 08 15:46:08.548255 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.548207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2" event={"ID":"d818a8bd-49ec-4d08-9602-78e859ddf43e","Type":"ContainerDied","Data":"2b00d570ce009527d5f71c4c5ab8b88b1100ea427755a2333ea6060d024c0605"}
Sep 08 15:46:08.548531 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.548515    2581 scope.go:117] "RemoveContainer" containerID="2b00d570ce009527d5f71c4c5ab8b88b1100ea427755a2333ea6060d024c0605"
Sep 08 15:46:08.829698 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.829671    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:46:08.902986 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.902959    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-serving-cert\") pod \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") "
Sep 08 15:46:08.903130 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.903014    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-oauth-serving-cert\") pod \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") "
Sep 08 15:46:08.903130 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.903043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59pk4\" (UniqueName: \"kubernetes.io/projected/d5b09ec7-9a0e-4da3-bed1-15b273e21318-kube-api-access-59pk4\") pod \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") "
Sep 08 15:46:08.903130 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.903072    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-config\") pod \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") "
Sep 08 15:46:08.903130 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.903105    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-service-ca\") pod \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") "
Sep 08 15:46:08.903380 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.903141    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-oauth-config\") pod \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\" (UID: \"d5b09ec7-9a0e-4da3-bed1-15b273e21318\") "
Sep 08 15:46:08.903532 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.903501    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d5b09ec7-9a0e-4da3-bed1-15b273e21318" (UID: "d5b09ec7-9a0e-4da3-bed1-15b273e21318"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:46:08.903532 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.903519    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-config" (OuterVolumeSpecName: "console-config") pod "d5b09ec7-9a0e-4da3-bed1-15b273e21318" (UID: "d5b09ec7-9a0e-4da3-bed1-15b273e21318"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:46:08.903698 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.903516    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-service-ca" (OuterVolumeSpecName: "service-ca") pod "d5b09ec7-9a0e-4da3-bed1-15b273e21318" (UID: "d5b09ec7-9a0e-4da3-bed1-15b273e21318"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:46:08.905468 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.905442    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b09ec7-9a0e-4da3-bed1-15b273e21318-kube-api-access-59pk4" (OuterVolumeSpecName: "kube-api-access-59pk4") pod "d5b09ec7-9a0e-4da3-bed1-15b273e21318" (UID: "d5b09ec7-9a0e-4da3-bed1-15b273e21318"). InnerVolumeSpecName "kube-api-access-59pk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:46:08.905750 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.905724    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d5b09ec7-9a0e-4da3-bed1-15b273e21318" (UID: "d5b09ec7-9a0e-4da3-bed1-15b273e21318"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:46:08.905844 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:08.905774    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d5b09ec7-9a0e-4da3-bed1-15b273e21318" (UID: "d5b09ec7-9a0e-4da3-bed1-15b273e21318"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:46:09.003988 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.003966    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-oauth-config\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:09.003988 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.003987    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-serving-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:09.004159 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.003996    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-oauth-serving-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:09.004159 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.004007    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59pk4\" (UniqueName: \"kubernetes.io/projected/d5b09ec7-9a0e-4da3-bed1-15b273e21318-kube-api-access-59pk4\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:09.004159 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.004016    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-console-config\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:09.004159 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.004025    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b09ec7-9a0e-4da3-bed1-15b273e21318-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:09.554807 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.554775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xk8g2" event={"ID":"d818a8bd-49ec-4d08-9602-78e859ddf43e","Type":"ContainerStarted","Data":"743a63ce7679f3d9f33a101ad06d8a2331e7f0939ea85c386812fa97729e926b"}
Sep 08 15:46:09.556115 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.556090    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5b09ec7-9a0e-4da3-bed1-15b273e21318" containerID="46dab661b37e9e1a229fe67586ede4c9d9fe21d288c9a5efa8dbb85635101b85" exitCode=0
Sep 08 15:46:09.556228 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.556124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58ff85d5d7-hntxc" event={"ID":"d5b09ec7-9a0e-4da3-bed1-15b273e21318","Type":"ContainerDied","Data":"46dab661b37e9e1a229fe67586ede4c9d9fe21d288c9a5efa8dbb85635101b85"}
Sep 08 15:46:09.556228 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.556154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58ff85d5d7-hntxc" event={"ID":"d5b09ec7-9a0e-4da3-bed1-15b273e21318","Type":"ContainerDied","Data":"4227a8564aac57b04ff69d95b47fbdd07b6987f29dd7b7abfcd9898e9f90b9fa"}
Sep 08 15:46:09.556228 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.556126    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58ff85d5d7-hntxc"
Sep 08 15:46:09.556228 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.556169    2581 scope.go:117] "RemoveContainer" containerID="46dab661b37e9e1a229fe67586ede4c9d9fe21d288c9a5efa8dbb85635101b85"
Sep 08 15:46:09.565046 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.565028    2581 scope.go:117] "RemoveContainer" containerID="46dab661b37e9e1a229fe67586ede4c9d9fe21d288c9a5efa8dbb85635101b85"
Sep 08 15:46:09.565308 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:46:09.565290    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46dab661b37e9e1a229fe67586ede4c9d9fe21d288c9a5efa8dbb85635101b85\": container with ID starting with 46dab661b37e9e1a229fe67586ede4c9d9fe21d288c9a5efa8dbb85635101b85 not found: ID does not exist" containerID="46dab661b37e9e1a229fe67586ede4c9d9fe21d288c9a5efa8dbb85635101b85"
Sep 08 15:46:09.565348 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.565316    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46dab661b37e9e1a229fe67586ede4c9d9fe21d288c9a5efa8dbb85635101b85"} err="failed to get container status \"46dab661b37e9e1a229fe67586ede4c9d9fe21d288c9a5efa8dbb85635101b85\": rpc error: code = NotFound desc = could not find container \"46dab661b37e9e1a229fe67586ede4c9d9fe21d288c9a5efa8dbb85635101b85\": container with ID starting with 46dab661b37e9e1a229fe67586ede4c9d9fe21d288c9a5efa8dbb85635101b85 not found: ID does not exist"
Sep 08 15:46:09.584656 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.584633    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58ff85d5d7-hntxc"]
Sep 08 15:46:09.600630 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.598260    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58ff85d5d7-hntxc"]
Sep 08 15:46:09.899785 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:09.899750    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5b09ec7-9a0e-4da3-bed1-15b273e21318" path="/var/lib/kubelet/pods/d5b09ec7-9a0e-4da3-bed1-15b273e21318/volumes"
Sep 08 15:46:12.681350 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:12.681311    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74f5f6df75-6fqds" podUID="38cac5cb-e76e-4799-a54c-eaf3158d7158" containerName="registry" containerID="cri-o://b86fca7eed3457fb0ed238932c80ab52ad7e5852bd00bde4fda8588b7e1656b4" gracePeriod=30
Sep 08 15:46:12.955234 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:12.955170    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:46:13.037762 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.037732    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-bound-sa-token\") pod \"38cac5cb-e76e-4799-a54c-eaf3158d7158\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") "
Sep 08 15:46:13.037910 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.037774    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/38cac5cb-e76e-4799-a54c-eaf3158d7158-image-registry-private-configuration\") pod \"38cac5cb-e76e-4799-a54c-eaf3158d7158\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") "
Sep 08 15:46:13.037910 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.037801    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls\") pod \"38cac5cb-e76e-4799-a54c-eaf3158d7158\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") "
Sep 08 15:46:13.037910 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.037824    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38cac5cb-e76e-4799-a54c-eaf3158d7158-trusted-ca\") pod \"38cac5cb-e76e-4799-a54c-eaf3158d7158\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") "
Sep 08 15:46:13.037910 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.037854    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-certificates\") pod \"38cac5cb-e76e-4799-a54c-eaf3158d7158\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") "
Sep 08 15:46:13.038134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.037937    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/38cac5cb-e76e-4799-a54c-eaf3158d7158-ca-trust-extracted\") pod \"38cac5cb-e76e-4799-a54c-eaf3158d7158\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") "
Sep 08 15:46:13.038134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.037967    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/38cac5cb-e76e-4799-a54c-eaf3158d7158-installation-pull-secrets\") pod \"38cac5cb-e76e-4799-a54c-eaf3158d7158\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") "
Sep 08 15:46:13.038134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.037993    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq4qg\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-kube-api-access-cq4qg\") pod \"38cac5cb-e76e-4799-a54c-eaf3158d7158\" (UID: \"38cac5cb-e76e-4799-a54c-eaf3158d7158\") "
Sep 08 15:46:13.038343 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.038306    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38cac5cb-e76e-4799-a54c-eaf3158d7158-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "38cac5cb-e76e-4799-a54c-eaf3158d7158" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:46:13.038430 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.038352    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "38cac5cb-e76e-4799-a54c-eaf3158d7158" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:46:13.040556 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.040521    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38cac5cb-e76e-4799-a54c-eaf3158d7158-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "38cac5cb-e76e-4799-a54c-eaf3158d7158" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:46:13.040556 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.040525    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "38cac5cb-e76e-4799-a54c-eaf3158d7158" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:46:13.040687 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.040605    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "38cac5cb-e76e-4799-a54c-eaf3158d7158" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:46:13.040876 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.040852    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38cac5cb-e76e-4799-a54c-eaf3158d7158-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "38cac5cb-e76e-4799-a54c-eaf3158d7158" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:46:13.040916 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.040874    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-kube-api-access-cq4qg" (OuterVolumeSpecName: "kube-api-access-cq4qg") pod "38cac5cb-e76e-4799-a54c-eaf3158d7158" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158"). InnerVolumeSpecName "kube-api-access-cq4qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:46:13.046979 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.046948    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cac5cb-e76e-4799-a54c-eaf3158d7158-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "38cac5cb-e76e-4799-a54c-eaf3158d7158" (UID: "38cac5cb-e76e-4799-a54c-eaf3158d7158"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:46:13.139024 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.139000    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/38cac5cb-e76e-4799-a54c-eaf3158d7158-installation-pull-secrets\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:13.139024 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.139022    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq4qg\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-kube-api-access-cq4qg\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:13.139140 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.139033    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-bound-sa-token\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:13.139140 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.139048    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/38cac5cb-e76e-4799-a54c-eaf3158d7158-image-registry-private-configuration\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:13.139140 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.139058    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-tls\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:13.139140 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.139066    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38cac5cb-e76e-4799-a54c-eaf3158d7158-trusted-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:13.139140 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.139073    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/38cac5cb-e76e-4799-a54c-eaf3158d7158-registry-certificates\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:13.139140 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.139081    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/38cac5cb-e76e-4799-a54c-eaf3158d7158-ca-trust-extracted\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:46:13.574299 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.574267    2581 generic.go:356] "Generic (PLEG): container finished" podID="38cac5cb-e76e-4799-a54c-eaf3158d7158" containerID="b86fca7eed3457fb0ed238932c80ab52ad7e5852bd00bde4fda8588b7e1656b4" exitCode=0
Sep 08 15:46:13.574436 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.574309    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f5f6df75-6fqds"
Sep 08 15:46:13.574436 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.574351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f5f6df75-6fqds" event={"ID":"38cac5cb-e76e-4799-a54c-eaf3158d7158","Type":"ContainerDied","Data":"b86fca7eed3457fb0ed238932c80ab52ad7e5852bd00bde4fda8588b7e1656b4"}
Sep 08 15:46:13.574436 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.574417    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f5f6df75-6fqds" event={"ID":"38cac5cb-e76e-4799-a54c-eaf3158d7158","Type":"ContainerDied","Data":"d7b1c4059b8a0d84421a1cd144fcd048baf79aef2f6bccb87e9a4f99d6cc5d29"}
Sep 08 15:46:13.574561 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.574440    2581 scope.go:117] "RemoveContainer" containerID="b86fca7eed3457fb0ed238932c80ab52ad7e5852bd00bde4fda8588b7e1656b4"
Sep 08 15:46:13.585122 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.583550    2581 scope.go:117] "RemoveContainer" containerID="b86fca7eed3457fb0ed238932c80ab52ad7e5852bd00bde4fda8588b7e1656b4"
Sep 08 15:46:13.585122 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:46:13.583870    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86fca7eed3457fb0ed238932c80ab52ad7e5852bd00bde4fda8588b7e1656b4\": container with ID starting with b86fca7eed3457fb0ed238932c80ab52ad7e5852bd00bde4fda8588b7e1656b4 not found: ID does not exist" containerID="b86fca7eed3457fb0ed238932c80ab52ad7e5852bd00bde4fda8588b7e1656b4"
Sep 08 15:46:13.585122 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.583897    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86fca7eed3457fb0ed238932c80ab52ad7e5852bd00bde4fda8588b7e1656b4"} err="failed to get container status \"b86fca7eed3457fb0ed238932c80ab52ad7e5852bd00bde4fda8588b7e1656b4\": rpc error: code = NotFound desc = could not find container \"b86fca7eed3457fb0ed238932c80ab52ad7e5852bd00bde4fda8588b7e1656b4\": container with ID starting with b86fca7eed3457fb0ed238932c80ab52ad7e5852bd00bde4fda8588b7e1656b4 not found: ID does not exist"
Sep 08 15:46:13.595748 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.595728    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74f5f6df75-6fqds"]
Sep 08 15:46:13.605187 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.605168    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74f5f6df75-6fqds"]
Sep 08 15:46:13.898760 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:13.898734    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38cac5cb-e76e-4799-a54c-eaf3158d7158" path="/var/lib/kubelet/pods/38cac5cb-e76e-4799-a54c-eaf3158d7158/volumes"
Sep 08 15:46:18.595043 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:18.595009    2581 generic.go:356] "Generic (PLEG): container finished" podID="b6a25233-dc78-40bb-9a74-25f3aae52b3c" containerID="76b04771ce6c458646b46922ccc8d8a34f3c465bce0d83fb4aee236dd960b314" exitCode=0
Sep 08 15:46:18.595533 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:18.595097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p" event={"ID":"b6a25233-dc78-40bb-9a74-25f3aae52b3c","Type":"ContainerDied","Data":"76b04771ce6c458646b46922ccc8d8a34f3c465bce0d83fb4aee236dd960b314"}
Sep 08 15:46:18.595533 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:18.595496    2581 scope.go:117] "RemoveContainer" containerID="76b04771ce6c458646b46922ccc8d8a34f3c465bce0d83fb4aee236dd960b314"
Sep 08 15:46:18.596665 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:18.596645    2581 generic.go:356] "Generic (PLEG): container finished" podID="593f6dab-17b1-48a1-a4bb-a9164f694d90" containerID="f428acbe143abea5aff4c8e8fa5f666fb0b797ffb71c9d3c1e931c3fceb7d069" exitCode=0
Sep 08 15:46:18.596762 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:18.596719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx" event={"ID":"593f6dab-17b1-48a1-a4bb-a9164f694d90","Type":"ContainerDied","Data":"f428acbe143abea5aff4c8e8fa5f666fb0b797ffb71c9d3c1e931c3fceb7d069"}
Sep 08 15:46:18.596981 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:18.596968    2581 scope.go:117] "RemoveContainer" containerID="f428acbe143abea5aff4c8e8fa5f666fb0b797ffb71c9d3c1e931c3fceb7d069"
Sep 08 15:46:19.602449 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:19.602416    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-nxt9p" event={"ID":"b6a25233-dc78-40bb-9a74-25f3aae52b3c","Type":"ContainerStarted","Data":"f3428af27fe5fc9947e5b7ce320f3522ffcc32b218f81b2b33acf3984ce37b4e"}
Sep 08 15:46:19.604268 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:19.604243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-vdxpx" event={"ID":"593f6dab-17b1-48a1-a4bb-a9164f694d90","Type":"ContainerStarted","Data":"f4c9bd138f3746590275feb49ff2c978625f46e9e80d1b8691a51b054a395353"}
Sep 08 15:46:35.940561 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:46:35.940525    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-785c579579-8mtc8"]
Sep 08 15:47:00.968885 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:00.968836    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-785c579579-8mtc8" podUID="88a70cbf-8314-414d-81a9-f4bea4326cfc" containerName="console" containerID="cri-o://c8017a793bc42ccaceeaf15afcbc915340ad9d7f7dc70cad7175ca94df0a09bf" gracePeriod=15
Sep 08 15:47:01.217307 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.217285    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:47:01.314687 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.314626    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-serving-cert\") pod \"88a70cbf-8314-414d-81a9-f4bea4326cfc\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") "
Sep 08 15:47:01.314817 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.314688    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-oauth-serving-cert\") pod \"88a70cbf-8314-414d-81a9-f4bea4326cfc\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") "
Sep 08 15:47:01.314817 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.314716    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-oauth-config\") pod \"88a70cbf-8314-414d-81a9-f4bea4326cfc\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") "
Sep 08 15:47:01.314817 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.314746    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkzff\" (UniqueName: \"kubernetes.io/projected/88a70cbf-8314-414d-81a9-f4bea4326cfc-kube-api-access-pkzff\") pod \"88a70cbf-8314-414d-81a9-f4bea4326cfc\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") "
Sep 08 15:47:01.314817 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.314764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-service-ca\") pod \"88a70cbf-8314-414d-81a9-f4bea4326cfc\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") "
Sep 08 15:47:01.314817 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.314789    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-config\") pod \"88a70cbf-8314-414d-81a9-f4bea4326cfc\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") "
Sep 08 15:47:01.315099 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.314821    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-trusted-ca-bundle\") pod \"88a70cbf-8314-414d-81a9-f4bea4326cfc\" (UID: \"88a70cbf-8314-414d-81a9-f4bea4326cfc\") "
Sep 08 15:47:01.315229 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.315183    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "88a70cbf-8314-414d-81a9-f4bea4326cfc" (UID: "88a70cbf-8314-414d-81a9-f4bea4326cfc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:47:01.315229 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.315209    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-service-ca" (OuterVolumeSpecName: "service-ca") pod "88a70cbf-8314-414d-81a9-f4bea4326cfc" (UID: "88a70cbf-8314-414d-81a9-f4bea4326cfc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:47:01.315332 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.315226    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-config" (OuterVolumeSpecName: "console-config") pod "88a70cbf-8314-414d-81a9-f4bea4326cfc" (UID: "88a70cbf-8314-414d-81a9-f4bea4326cfc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:47:01.315474 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.315452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "88a70cbf-8314-414d-81a9-f4bea4326cfc" (UID: "88a70cbf-8314-414d-81a9-f4bea4326cfc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:47:01.316865 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.316830    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a70cbf-8314-414d-81a9-f4bea4326cfc-kube-api-access-pkzff" (OuterVolumeSpecName: "kube-api-access-pkzff") pod "88a70cbf-8314-414d-81a9-f4bea4326cfc" (UID: "88a70cbf-8314-414d-81a9-f4bea4326cfc"). InnerVolumeSpecName "kube-api-access-pkzff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:47:01.316972 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.316869    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "88a70cbf-8314-414d-81a9-f4bea4326cfc" (UID: "88a70cbf-8314-414d-81a9-f4bea4326cfc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:47:01.316972 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.316892    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "88a70cbf-8314-414d-81a9-f4bea4326cfc" (UID: "88a70cbf-8314-414d-81a9-f4bea4326cfc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:47:01.415301 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.415279    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-oauth-config\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:47:01.415301 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.415300    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkzff\" (UniqueName: \"kubernetes.io/projected/88a70cbf-8314-414d-81a9-f4bea4326cfc-kube-api-access-pkzff\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:47:01.415458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.415310    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:47:01.415458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.415319    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-config\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:47:01.415458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.415326    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-trusted-ca-bundle\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:47:01.415458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.415334    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a70cbf-8314-414d-81a9-f4bea4326cfc-console-serving-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:47:01.415458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.415342    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a70cbf-8314-414d-81a9-f4bea4326cfc-oauth-serving-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:47:01.771521 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.771490    2581 generic.go:356] "Generic (PLEG): container finished" podID="88a70cbf-8314-414d-81a9-f4bea4326cfc" containerID="c8017a793bc42ccaceeaf15afcbc915340ad9d7f7dc70cad7175ca94df0a09bf" exitCode=0
Sep 08 15:47:01.771702 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.771536    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-785c579579-8mtc8"
Sep 08 15:47:01.771702 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.771583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-785c579579-8mtc8" event={"ID":"88a70cbf-8314-414d-81a9-f4bea4326cfc","Type":"ContainerDied","Data":"c8017a793bc42ccaceeaf15afcbc915340ad9d7f7dc70cad7175ca94df0a09bf"}
Sep 08 15:47:01.771702 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.771672    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-785c579579-8mtc8" event={"ID":"88a70cbf-8314-414d-81a9-f4bea4326cfc","Type":"ContainerDied","Data":"63f2245aac64409a6940c3e6c39e9054578b616e9c9885cc688ba29ffd6088ad"}
Sep 08 15:47:01.771702 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.771701    2581 scope.go:117] "RemoveContainer" containerID="c8017a793bc42ccaceeaf15afcbc915340ad9d7f7dc70cad7175ca94df0a09bf"
Sep 08 15:47:01.780233 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.780218    2581 scope.go:117] "RemoveContainer" containerID="c8017a793bc42ccaceeaf15afcbc915340ad9d7f7dc70cad7175ca94df0a09bf"
Sep 08 15:47:01.780500 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:47:01.780480    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8017a793bc42ccaceeaf15afcbc915340ad9d7f7dc70cad7175ca94df0a09bf\": container with ID starting with c8017a793bc42ccaceeaf15afcbc915340ad9d7f7dc70cad7175ca94df0a09bf not found: ID does not exist" containerID="c8017a793bc42ccaceeaf15afcbc915340ad9d7f7dc70cad7175ca94df0a09bf"
Sep 08 15:47:01.780555 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.780508    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8017a793bc42ccaceeaf15afcbc915340ad9d7f7dc70cad7175ca94df0a09bf"} err="failed to get container status \"c8017a793bc42ccaceeaf15afcbc915340ad9d7f7dc70cad7175ca94df0a09bf\": rpc error: code = NotFound desc = could not find container \"c8017a793bc42ccaceeaf15afcbc915340ad9d7f7dc70cad7175ca94df0a09bf\": container with ID starting with c8017a793bc42ccaceeaf15afcbc915340ad9d7f7dc70cad7175ca94df0a09bf not found: ID does not exist"
Sep 08 15:47:01.791866 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.791844    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-785c579579-8mtc8"]
Sep 08 15:47:01.795689 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.795671    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-785c579579-8mtc8"]
Sep 08 15:47:01.899280 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:47:01.899257    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88a70cbf-8314-414d-81a9-f4bea4326cfc" path="/var/lib/kubelet/pods/88a70cbf-8314-414d-81a9-f4bea4326cfc/volumes"
Sep 08 15:48:41.289138 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.289054    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"]
Sep 08 15:48:41.289656 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.289348    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b09ec7-9a0e-4da3-bed1-15b273e21318" containerName="console"
Sep 08 15:48:41.289656 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.289359    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b09ec7-9a0e-4da3-bed1-15b273e21318" containerName="console"
Sep 08 15:48:41.289656 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.289370    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a70cbf-8314-414d-81a9-f4bea4326cfc" containerName="console"
Sep 08 15:48:41.289656 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.289376    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a70cbf-8314-414d-81a9-f4bea4326cfc" containerName="console"
Sep 08 15:48:41.289656 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.289404    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38cac5cb-e76e-4799-a54c-eaf3158d7158" containerName="registry"
Sep 08 15:48:41.289656 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.289410    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38cac5cb-e76e-4799-a54c-eaf3158d7158" containerName="registry"
Sep 08 15:48:41.289656 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.289460    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5b09ec7-9a0e-4da3-bed1-15b273e21318" containerName="console"
Sep 08 15:48:41.289656 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.289467    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a70cbf-8314-414d-81a9-f4bea4326cfc" containerName="console"
Sep 08 15:48:41.289656 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.289477    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="38cac5cb-e76e-4799-a54c-eaf3158d7158" containerName="registry"
Sep 08 15:48:41.292372 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.292356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.294776 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.294755    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-89p79\""
Sep 08 15:48:41.295045 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.295030    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 15:48:41.295512 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.295490    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 15:48:41.295601 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.295510    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 15:48:41.295601 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.295590    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 15:48:41.307365 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.307344    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"]
Sep 08 15:48:41.415475 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.415422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2b2fc001-999d-4c56-8fde-828543a20324-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.415475 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.415482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2b2fc001-999d-4c56-8fde-828543a20324-manifests\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.415689 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.415534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b2fc001-999d-4c56-8fde-828543a20324-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.415689 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.415583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b2fc001-999d-4c56-8fde-828543a20324-tmp\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.415689 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.415623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjqql\" (UniqueName: \"kubernetes.io/projected/2b2fc001-999d-4c56-8fde-828543a20324-kube-api-access-bjqql\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.516286 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.516232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b2fc001-999d-4c56-8fde-828543a20324-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.516286 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.516288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b2fc001-999d-4c56-8fde-828543a20324-tmp\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.516561 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.516319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjqql\" (UniqueName: \"kubernetes.io/projected/2b2fc001-999d-4c56-8fde-828543a20324-kube-api-access-bjqql\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.516561 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.516369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2b2fc001-999d-4c56-8fde-828543a20324-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.516561 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.516414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2b2fc001-999d-4c56-8fde-828543a20324-manifests\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.516863 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.516833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2b2fc001-999d-4c56-8fde-828543a20324-manifests\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.518805 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.518783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b2fc001-999d-4c56-8fde-828543a20324-tmp\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.518927 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.518909    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2b2fc001-999d-4c56-8fde-828543a20324-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.518968 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.518940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b2fc001-999d-4c56-8fde-828543a20324-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.524895 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.524873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjqql\" (UniqueName: \"kubernetes.io/projected/2b2fc001-999d-4c56-8fde-828543a20324-kube-api-access-bjqql\") pod \"opendatahub-operator-controller-manager-5d4b5866d6-b5dpm\" (UID: \"2b2fc001-999d-4c56-8fde-828543a20324\") " pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.602616 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.602530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:41.886409 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:41.886367    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"]
Sep 08 15:48:41.889000 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:48:41.888973    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2fc001_999d_4c56_8fde_828543a20324.slice/crio-57d2ddcf57b3704f23a74a8c21b4db16ea1731ae847d0e6179208173a7e420c1 WatchSource:0}: Error finding container 57d2ddcf57b3704f23a74a8c21b4db16ea1731ae847d0e6179208173a7e420c1: Status 404 returned error can't find the container with id 57d2ddcf57b3704f23a74a8c21b4db16ea1731ae847d0e6179208173a7e420c1
Sep 08 15:48:42.138591 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:42.138558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm" event={"ID":"2b2fc001-999d-4c56-8fde-828543a20324","Type":"ContainerStarted","Data":"57d2ddcf57b3704f23a74a8c21b4db16ea1731ae847d0e6179208173a7e420c1"}
Sep 08 15:48:46.158203 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:46.158105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm" event={"ID":"2b2fc001-999d-4c56-8fde-828543a20324","Type":"ContainerStarted","Data":"eb7af06f0f6583fdd96e51fdb3c9719ce12a2b512c26da766115dbc1cfd533c1"}
Sep 08 15:48:47.163514 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:47.163482    2581 generic.go:356] "Generic (PLEG): container finished" podID="2b2fc001-999d-4c56-8fde-828543a20324" containerID="eb7af06f0f6583fdd96e51fdb3c9719ce12a2b512c26da766115dbc1cfd533c1" exitCode=0
Sep 08 15:48:47.163945 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:47.163570    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm" event={"ID":"2b2fc001-999d-4c56-8fde-828543a20324","Type":"ContainerDied","Data":"eb7af06f0f6583fdd96e51fdb3c9719ce12a2b512c26da766115dbc1cfd533c1"}
Sep 08 15:48:48.170007 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:48.169969    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm" event={"ID":"2b2fc001-999d-4c56-8fde-828543a20324","Type":"ContainerStarted","Data":"c76c231d9900ee3f7564a4c41c5ad1fb8e8013d71e0e619aba3df7d89901e5e9"}
Sep 08 15:48:48.170421 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:48.170055    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:48:48.193737 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:48.193685    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm" podStartSLOduration=1.8103048830000001 podStartE2EDuration="7.193670192s" totalImagesPullingTime="4.360204562s" totalInitContainerRuntime="1.023160747s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:48:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:48:48.192622515 +0000 UTC m=+274.816314855" watchObservedRunningTime="2026-09-08 15:48:48.193670192 +0000 UTC m=+274.817362516"
Sep 08 15:48:59.179673 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:48:59.179638    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d4b5866d6-b5dpm"
Sep 08 15:49:13.805621 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:13.805592    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 15:49:13.816515 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:13.816489    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 15:49:13.818325 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:13.818295    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 15:49:18.500456 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.500417    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"]
Sep 08 15:49:18.503660 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.503643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:18.506716 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.506469    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 15:49:18.506716 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.506473    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 15:49:18.506939 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.506788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6e2519b-a925-4dbb-9383-2ab9454866bb-manager-config\") pod \"lws-controller-manager-578d6cf975-h7ppl\" (UID: \"d6e2519b-a925-4dbb-9383-2ab9454866bb\") " pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:18.506939 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.506855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swscb\" (UniqueName: \"kubernetes.io/projected/d6e2519b-a925-4dbb-9383-2ab9454866bb-kube-api-access-swscb\") pod \"lws-controller-manager-578d6cf975-h7ppl\" (UID: \"d6e2519b-a925-4dbb-9383-2ab9454866bb\") " pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:18.506939 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.506883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6e2519b-a925-4dbb-9383-2ab9454866bb-cert\") pod \"lws-controller-manager-578d6cf975-h7ppl\" (UID: \"d6e2519b-a925-4dbb-9383-2ab9454866bb\") " pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:18.506939 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.506924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6e2519b-a925-4dbb-9383-2ab9454866bb-metrics-cert\") pod \"lws-controller-manager-578d6cf975-h7ppl\" (UID: \"d6e2519b-a925-4dbb-9383-2ab9454866bb\") " pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:18.507191 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.506946    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:49:18.507191 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.507138    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 15:49:18.507270 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.507193    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 15:49:18.507367 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.507352    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lpf5t\""
Sep 08 15:49:18.514163 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.514140    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"]
Sep 08 15:49:18.607765 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.607730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6e2519b-a925-4dbb-9383-2ab9454866bb-manager-config\") pod \"lws-controller-manager-578d6cf975-h7ppl\" (UID: \"d6e2519b-a925-4dbb-9383-2ab9454866bb\") " pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:18.607935 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.607786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swscb\" (UniqueName: \"kubernetes.io/projected/d6e2519b-a925-4dbb-9383-2ab9454866bb-kube-api-access-swscb\") pod \"lws-controller-manager-578d6cf975-h7ppl\" (UID: \"d6e2519b-a925-4dbb-9383-2ab9454866bb\") " pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:18.607935 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.607808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6e2519b-a925-4dbb-9383-2ab9454866bb-cert\") pod \"lws-controller-manager-578d6cf975-h7ppl\" (UID: \"d6e2519b-a925-4dbb-9383-2ab9454866bb\") " pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:18.607935 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.607843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6e2519b-a925-4dbb-9383-2ab9454866bb-metrics-cert\") pod \"lws-controller-manager-578d6cf975-h7ppl\" (UID: \"d6e2519b-a925-4dbb-9383-2ab9454866bb\") " pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:18.608485 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.608463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6e2519b-a925-4dbb-9383-2ab9454866bb-manager-config\") pod \"lws-controller-manager-578d6cf975-h7ppl\" (UID: \"d6e2519b-a925-4dbb-9383-2ab9454866bb\") " pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:18.610315 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.610291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6e2519b-a925-4dbb-9383-2ab9454866bb-cert\") pod \"lws-controller-manager-578d6cf975-h7ppl\" (UID: \"d6e2519b-a925-4dbb-9383-2ab9454866bb\") " pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:18.610416 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.610334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6e2519b-a925-4dbb-9383-2ab9454866bb-metrics-cert\") pod \"lws-controller-manager-578d6cf975-h7ppl\" (UID: \"d6e2519b-a925-4dbb-9383-2ab9454866bb\") " pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:18.615522 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.615504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swscb\" (UniqueName: \"kubernetes.io/projected/d6e2519b-a925-4dbb-9383-2ab9454866bb-kube-api-access-swscb\") pod \"lws-controller-manager-578d6cf975-h7ppl\" (UID: \"d6e2519b-a925-4dbb-9383-2ab9454866bb\") " pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:18.813300 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:18.813205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:19.102045 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:19.101957    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"]
Sep 08 15:49:19.106190 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:49:19.106158    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e2519b_a925_4dbb_9383_2ab9454866bb.slice/crio-0bfc88a2686de18d43d4407972c2451ec635c8909b7e3661203b63375ffa4b3c WatchSource:0}: Error finding container 0bfc88a2686de18d43d4407972c2451ec635c8909b7e3661203b63375ffa4b3c: Status 404 returned error can't find the container with id 0bfc88a2686de18d43d4407972c2451ec635c8909b7e3661203b63375ffa4b3c
Sep 08 15:49:19.108062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:19.108043    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:49:19.304873 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:19.304832    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl" event={"ID":"d6e2519b-a925-4dbb-9383-2ab9454866bb","Type":"ContainerStarted","Data":"0bfc88a2686de18d43d4407972c2451ec635c8909b7e3661203b63375ffa4b3c"}
Sep 08 15:49:33.366002 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:33.365962    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl" event={"ID":"d6e2519b-a925-4dbb-9383-2ab9454866bb","Type":"ContainerStarted","Data":"46eef0744ce7aad5ad9c6dc7c00439d1b7af74f3c7531e960beb735cf2f7c4a4"}
Sep 08 15:49:33.366468 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:33.366042    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:49:33.383881 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:33.383828    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl" podStartSLOduration=2.097922554 podStartE2EDuration="15.383811954s" totalImagesPullingTime="13.2858894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:49:33.381992151 +0000 UTC m=+320.005684475" watchObservedRunningTime="2026-09-08 15:49:33.383811954 +0000 UTC m=+320.007504352"
Sep 08 15:49:44.372234 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:49:44.372197    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-578d6cf975-h7ppl"
Sep 08 15:50:30.092000 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.091968    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66f7cf5496-w8l6k"]
Sep 08 15:50:30.096101 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.096079    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.099504 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.099472    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p89kc\""
Sep 08 15:50:30.099504 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.099472    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 15:50:30.099707 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.099575    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 15:50:30.099707 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.099474    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 15:50:30.099707 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.099609    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 15:50:30.099707 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.099475    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 15:50:30.104452 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.104433    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 15:50:30.106923 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.106903    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f7cf5496-w8l6k"]
Sep 08 15:50:30.204872 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.204835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qrgs\" (UniqueName: \"kubernetes.io/projected/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-kube-api-access-7qrgs\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.204872 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.204875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-service-ca\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.205134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.204904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-trusted-ca-bundle\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.205134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.205001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-console-config\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.205134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.205042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-oauth-serving-cert\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.205134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.205073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-console-serving-cert\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.205134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.205100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-console-oauth-config\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.306122 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.306080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qrgs\" (UniqueName: \"kubernetes.io/projected/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-kube-api-access-7qrgs\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.306328 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.306128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-service-ca\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.306328 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.306171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-trusted-ca-bundle\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.306328 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.306219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-console-config\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.306328 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.306245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-oauth-serving-cert\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.306328 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.306270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-console-serving-cert\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.306328 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.306295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-console-oauth-config\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.307071 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.307039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-console-config\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.307212 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.307045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-service-ca\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.307212 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.307102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-oauth-serving-cert\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.307212 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.307188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-trusted-ca-bundle\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.308745 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.308714    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-console-oauth-config\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.308899 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.308882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-console-serving-cert\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.313541 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.313524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qrgs\" (UniqueName: \"kubernetes.io/projected/026b9f7a-72c6-4eb7-b8fb-e50d2d72788e-kube-api-access-7qrgs\") pod \"console-66f7cf5496-w8l6k\" (UID: \"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e\") " pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.406797 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.406762    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:30.680720 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:30.680503    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f7cf5496-w8l6k"]
Sep 08 15:50:30.683615 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:50:30.683587    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026b9f7a_72c6_4eb7_b8fb_e50d2d72788e.slice/crio-cb31a3da23f40de6e020cfe438e4a1c24fa88a66a018a3421ac698c60db29652 WatchSource:0}: Error finding container cb31a3da23f40de6e020cfe438e4a1c24fa88a66a018a3421ac698c60db29652: Status 404 returned error can't find the container with id cb31a3da23f40de6e020cfe438e4a1c24fa88a66a018a3421ac698c60db29652
Sep 08 15:50:31.613502 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:31.613464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f7cf5496-w8l6k" event={"ID":"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e","Type":"ContainerStarted","Data":"4b833e3fb0b3c217196ecb1ffa847461fcae239babb837b265f25a8a8bec2530"}
Sep 08 15:50:31.613889 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:31.613518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f7cf5496-w8l6k" event={"ID":"026b9f7a-72c6-4eb7-b8fb-e50d2d72788e","Type":"ContainerStarted","Data":"cb31a3da23f40de6e020cfe438e4a1c24fa88a66a018a3421ac698c60db29652"}
Sep 08 15:50:31.632067 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:31.632021    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66f7cf5496-w8l6k" podStartSLOduration=1.632005979 podStartE2EDuration="1.632005979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:50:31.630531049 +0000 UTC m=+378.254223371" watchObservedRunningTime="2026-09-08 15:50:31.632005979 +0000 UTC m=+378.255698303"
Sep 08 15:50:40.407096 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:40.407063    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:40.407568 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:40.407323    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:40.412101 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:40.412078    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:40.655371 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:40.655345    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66f7cf5496-w8l6k"
Sep 08 15:50:44.164475 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:44.164435    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"]
Sep 08 15:50:44.167805 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:44.167782    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:50:44.170639 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:44.170615    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tl8hx\""
Sep 08 15:50:44.170900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:44.170881    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 15:50:44.171543 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:44.171528    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 15:50:44.183505 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:44.183485    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"]
Sep 08 15:50:44.315265 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:44.315240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pglfr\" (UniqueName: \"kubernetes.io/projected/772cff16-1a92-437f-b747-e59fa7bf82bd-kube-api-access-pglfr\") pod \"limitador-operator-controller-manager-6c549b6d85-2wgfs\" (UID: \"772cff16-1a92-437f-b747-e59fa7bf82bd\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:50:44.416079 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:44.415994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pglfr\" (UniqueName: \"kubernetes.io/projected/772cff16-1a92-437f-b747-e59fa7bf82bd-kube-api-access-pglfr\") pod \"limitador-operator-controller-manager-6c549b6d85-2wgfs\" (UID: \"772cff16-1a92-437f-b747-e59fa7bf82bd\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:50:44.434872 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:44.434847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pglfr\" (UniqueName: \"kubernetes.io/projected/772cff16-1a92-437f-b747-e59fa7bf82bd-kube-api-access-pglfr\") pod \"limitador-operator-controller-manager-6c549b6d85-2wgfs\" (UID: \"772cff16-1a92-437f-b747-e59fa7bf82bd\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:50:44.478173 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:44.478145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:50:44.754459 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:44.754380    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"]
Sep 08 15:50:44.758059 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:50:44.758032    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772cff16_1a92_437f_b747_e59fa7bf82bd.slice/crio-4221aa4537fc533f08620ee4a70a19f4fecc2c1e782b95795bc9d36c71562817 WatchSource:0}: Error finding container 4221aa4537fc533f08620ee4a70a19f4fecc2c1e782b95795bc9d36c71562817: Status 404 returned error can't find the container with id 4221aa4537fc533f08620ee4a70a19f4fecc2c1e782b95795bc9d36c71562817
Sep 08 15:50:45.673612 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:45.673579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs" event={"ID":"772cff16-1a92-437f-b747-e59fa7bf82bd","Type":"ContainerStarted","Data":"4221aa4537fc533f08620ee4a70a19f4fecc2c1e782b95795bc9d36c71562817"}
Sep 08 15:50:46.342837 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:46.342802    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"]
Sep 08 15:50:46.346463 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:46.346440    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"
Sep 08 15:50:46.348745 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:46.348726    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 08 15:50:46.348831 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:46.348756    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ftfb4\""
Sep 08 15:50:46.359642 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:46.359617    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"]
Sep 08 15:50:46.433691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:46.433661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68whp\" (UniqueName: \"kubernetes.io/projected/59d7537e-e948-4fde-930c-dd50424db8cc-kube-api-access-68whp\") pod \"dns-operator-controller-manager-5985564776-vxpps\" (UID: \"59d7537e-e948-4fde-930c-dd50424db8cc\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"
Sep 08 15:50:46.534714 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:46.534671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68whp\" (UniqueName: \"kubernetes.io/projected/59d7537e-e948-4fde-930c-dd50424db8cc-kube-api-access-68whp\") pod \"dns-operator-controller-manager-5985564776-vxpps\" (UID: \"59d7537e-e948-4fde-930c-dd50424db8cc\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"
Sep 08 15:50:46.545963 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:46.545935    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68whp\" (UniqueName: \"kubernetes.io/projected/59d7537e-e948-4fde-930c-dd50424db8cc-kube-api-access-68whp\") pod \"dns-operator-controller-manager-5985564776-vxpps\" (UID: \"59d7537e-e948-4fde-930c-dd50424db8cc\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"
Sep 08 15:50:46.657306 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:46.657274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"
Sep 08 15:50:47.617683 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:47.617577    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"]
Sep 08 15:50:47.620719 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:50:47.620679    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d7537e_e948_4fde_930c_dd50424db8cc.slice/crio-0159e275a94179af8b0010fedd43048561ecb155c24e890c66c7304558026237 WatchSource:0}: Error finding container 0159e275a94179af8b0010fedd43048561ecb155c24e890c66c7304558026237: Status 404 returned error can't find the container with id 0159e275a94179af8b0010fedd43048561ecb155c24e890c66c7304558026237
Sep 08 15:50:47.684231 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:47.684189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs" event={"ID":"772cff16-1a92-437f-b747-e59fa7bf82bd","Type":"ContainerStarted","Data":"c7fcf800bdae7d184d9d66382e5b8117c385e9307bc17ea74c75edf62a944d8f"}
Sep 08 15:50:47.684373 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:47.684298    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:50:47.685803 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:47.685778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps" event={"ID":"59d7537e-e948-4fde-930c-dd50424db8cc","Type":"ContainerStarted","Data":"0159e275a94179af8b0010fedd43048561ecb155c24e890c66c7304558026237"}
Sep 08 15:50:47.707883 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:47.707840    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs" podStartSLOduration=1.062664197 podStartE2EDuration="3.70782538s" totalImagesPullingTime="2.645161183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:50:47.706720269 +0000 UTC m=+394.330412594" watchObservedRunningTime="2026-09-08 15:50:47.70782538 +0000 UTC m=+394.331517703"
Sep 08 15:50:49.699118 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:49.699081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps" event={"ID":"59d7537e-e948-4fde-930c-dd50424db8cc","Type":"ContainerStarted","Data":"ca9b6740eeb7e8bd2424163cdb37e926f184f075fb37fe50ecdd6042d27029ec"}
Sep 08 15:50:49.699553 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:49.699169    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"
Sep 08 15:50:49.718378 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:49.718331    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps" podStartSLOduration=2.19459954 podStartE2EDuration="3.718314815s" totalImagesPullingTime="1.523715275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:50:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:50:49.717150662 +0000 UTC m=+396.340842986" watchObservedRunningTime="2026-09-08 15:50:49.718314815 +0000 UTC m=+396.342007138"
Sep 08 15:50:58.693790 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:50:58.693758    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:51:00.706139 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:00.706107    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"
Sep 08 15:51:08.281368 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.281333    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"]
Sep 08 15:51:08.284643 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.284625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"
Sep 08 15:51:08.286832 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.286804    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-t8mht\""
Sep 08 15:51:08.297492 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.297463    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"]
Sep 08 15:51:08.393901 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.393872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4vjz\" (UniqueName: \"kubernetes.io/projected/46ca2186-e3f0-4d8e-889d-853ffa7de247-kube-api-access-q4vjz\") pod \"kuadrant-operator-controller-manager-96db98c47-wx8dh\" (UID: \"46ca2186-e3f0-4d8e-889d-853ffa7de247\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"
Sep 08 15:51:08.394021 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.393909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/46ca2186-e3f0-4d8e-889d-853ffa7de247-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-wx8dh\" (UID: \"46ca2186-e3f0-4d8e-889d-853ffa7de247\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"
Sep 08 15:51:08.411885 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.411861    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"]
Sep 08 15:51:08.412041 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:51:08.412023    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-q4vjz], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh" podUID="46ca2186-e3f0-4d8e-889d-853ffa7de247"
Sep 08 15:51:08.495125 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.495097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4vjz\" (UniqueName: \"kubernetes.io/projected/46ca2186-e3f0-4d8e-889d-853ffa7de247-kube-api-access-q4vjz\") pod \"kuadrant-operator-controller-manager-96db98c47-wx8dh\" (UID: \"46ca2186-e3f0-4d8e-889d-853ffa7de247\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"
Sep 08 15:51:08.495209 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.495136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/46ca2186-e3f0-4d8e-889d-853ffa7de247-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-wx8dh\" (UID: \"46ca2186-e3f0-4d8e-889d-853ffa7de247\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"
Sep 08 15:51:08.495513 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.495497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/46ca2186-e3f0-4d8e-889d-853ffa7de247-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-wx8dh\" (UID: \"46ca2186-e3f0-4d8e-889d-853ffa7de247\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"
Sep 08 15:51:08.514843 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.514814    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4vjz\" (UniqueName: \"kubernetes.io/projected/46ca2186-e3f0-4d8e-889d-853ffa7de247-kube-api-access-q4vjz\") pod \"kuadrant-operator-controller-manager-96db98c47-wx8dh\" (UID: \"46ca2186-e3f0-4d8e-889d-853ffa7de247\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"
Sep 08 15:51:08.784407 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.784358    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"
Sep 08 15:51:08.791595 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.791572    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"
Sep 08 15:51:08.897027 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.897006    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4vjz\" (UniqueName: \"kubernetes.io/projected/46ca2186-e3f0-4d8e-889d-853ffa7de247-kube-api-access-q4vjz\") pod \"46ca2186-e3f0-4d8e-889d-853ffa7de247\" (UID: \"46ca2186-e3f0-4d8e-889d-853ffa7de247\") "
Sep 08 15:51:08.897156 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.897043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/46ca2186-e3f0-4d8e-889d-853ffa7de247-extensions-socket-volume\") pod \"46ca2186-e3f0-4d8e-889d-853ffa7de247\" (UID: \"46ca2186-e3f0-4d8e-889d-853ffa7de247\") "
Sep 08 15:51:08.897332 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.897306    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ca2186-e3f0-4d8e-889d-853ffa7de247-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "46ca2186-e3f0-4d8e-889d-853ffa7de247" (UID: "46ca2186-e3f0-4d8e-889d-853ffa7de247"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:51:08.899113 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.899096    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ca2186-e3f0-4d8e-889d-853ffa7de247-kube-api-access-q4vjz" (OuterVolumeSpecName: "kube-api-access-q4vjz") pod "46ca2186-e3f0-4d8e-889d-853ffa7de247" (UID: "46ca2186-e3f0-4d8e-889d-853ffa7de247"). InnerVolumeSpecName "kube-api-access-q4vjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:51:08.998001 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.997977    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4vjz\" (UniqueName: \"kubernetes.io/projected/46ca2186-e3f0-4d8e-889d-853ffa7de247-kube-api-access-q4vjz\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:51:08.998001 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:08.997999    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/46ca2186-e3f0-4d8e-889d-853ffa7de247-extensions-socket-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:51:09.321033 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.320999    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"]
Sep 08 15:51:09.325828 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.325797    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"]
Sep 08 15:51:09.337598 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.337573    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"]
Sep 08 15:51:09.337780 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.337760    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs" podUID="772cff16-1a92-437f-b747-e59fa7bf82bd" containerName="manager" containerID="cri-o://c7fcf800bdae7d184d9d66382e5b8117c385e9307bc17ea74c75edf62a944d8f" gracePeriod=2
Sep 08 15:51:09.346760 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.346740    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"]
Sep 08 15:51:09.351115 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:51:09.351067    2581 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"772cff16-1a92-437f-b747-e59fa7bf82bd\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-6c549b6d85-2wgfs\": pods \"limitador-operator-controller-manager-6c549b6d85-2wgfs\" not found" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:51:09.366322 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.366295    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"]
Sep 08 15:51:09.366693 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.366677    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772cff16-1a92-437f-b747-e59fa7bf82bd" containerName="manager"
Sep 08 15:51:09.366693 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.366694    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="772cff16-1a92-437f-b747-e59fa7bf82bd" containerName="manager"
Sep 08 15:51:09.366827 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.366767    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="772cff16-1a92-437f-b747-e59fa7bf82bd" containerName="manager"
Sep 08 15:51:09.371829 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.371809    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"
Sep 08 15:51:09.380149 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.380126    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"]
Sep 08 15:51:09.396355 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:51:09.396327    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-2wgfs\" is forbidden: User \"system:node:ip-10-0-134-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-107.ec2.internal' and this object" podUID="772cff16-1a92-437f-b747-e59fa7bf82bd" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:51:09.502112 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.502067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2rs\" (UniqueName: \"kubernetes.io/projected/a780cf4c-9615-4eb9-adc5-802b0f2372ae-kube-api-access-2m2rs\") pod \"limitador-operator-controller-manager-6c549b6d85-rgkn5\" (UID: \"a780cf4c-9615-4eb9-adc5-802b0f2372ae\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"
Sep 08 15:51:09.574616 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.574567    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:51:09.576683 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:51:09.576659    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-2wgfs\" is forbidden: User \"system:node:ip-10-0-134-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-107.ec2.internal' and this object" podUID="772cff16-1a92-437f-b747-e59fa7bf82bd" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:51:09.603096 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.603071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2rs\" (UniqueName: \"kubernetes.io/projected/a780cf4c-9615-4eb9-adc5-802b0f2372ae-kube-api-access-2m2rs\") pod \"limitador-operator-controller-manager-6c549b6d85-rgkn5\" (UID: \"a780cf4c-9615-4eb9-adc5-802b0f2372ae\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"
Sep 08 15:51:09.613714 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.613685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2rs\" (UniqueName: \"kubernetes.io/projected/a780cf4c-9615-4eb9-adc5-802b0f2372ae-kube-api-access-2m2rs\") pod \"limitador-operator-controller-manager-6c549b6d85-rgkn5\" (UID: \"a780cf4c-9615-4eb9-adc5-802b0f2372ae\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"
Sep 08 15:51:09.703861 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.703837    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pglfr\" (UniqueName: \"kubernetes.io/projected/772cff16-1a92-437f-b747-e59fa7bf82bd-kube-api-access-pglfr\") pod \"772cff16-1a92-437f-b747-e59fa7bf82bd\" (UID: \"772cff16-1a92-437f-b747-e59fa7bf82bd\") "
Sep 08 15:51:09.705805 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.705783    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772cff16-1a92-437f-b747-e59fa7bf82bd-kube-api-access-pglfr" (OuterVolumeSpecName: "kube-api-access-pglfr") pod "772cff16-1a92-437f-b747-e59fa7bf82bd" (UID: "772cff16-1a92-437f-b747-e59fa7bf82bd"). InnerVolumeSpecName "kube-api-access-pglfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:51:09.726064 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.726043    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"
Sep 08 15:51:09.792108 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.791960    2581 generic.go:356] "Generic (PLEG): container finished" podID="772cff16-1a92-437f-b747-e59fa7bf82bd" containerID="c7fcf800bdae7d184d9d66382e5b8117c385e9307bc17ea74c75edf62a944d8f" exitCode=0
Sep 08 15:51:09.792108 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.792023    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:51:09.792108 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.792052    2581 scope.go:117] "RemoveContainer" containerID="c7fcf800bdae7d184d9d66382e5b8117c385e9307bc17ea74c75edf62a944d8f"
Sep 08 15:51:09.792352 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.792314    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"
Sep 08 15:51:09.795143 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:51:09.795005    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-2wgfs\" is forbidden: User \"system:node:ip-10-0-134-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-107.ec2.internal' and this object" podUID="772cff16-1a92-437f-b747-e59fa7bf82bd" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:51:09.796831 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:51:09.796802    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-wx8dh\" is forbidden: User \"system:node:ip-10-0-134-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-107.ec2.internal' and this object" podUID="46ca2186-e3f0-4d8e-889d-853ffa7de247" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"
Sep 08 15:51:09.798592 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:51:09.798566    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-2wgfs\" is forbidden: User \"system:node:ip-10-0-134-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-107.ec2.internal' and this object" podUID="772cff16-1a92-437f-b747-e59fa7bf82bd" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:51:09.802124 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.802069    2581 scope.go:117] "RemoveContainer" containerID="c7fcf800bdae7d184d9d66382e5b8117c385e9307bc17ea74c75edf62a944d8f"
Sep 08 15:51:09.802536 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:51:09.802427    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fcf800bdae7d184d9d66382e5b8117c385e9307bc17ea74c75edf62a944d8f\": container with ID starting with c7fcf800bdae7d184d9d66382e5b8117c385e9307bc17ea74c75edf62a944d8f not found: ID does not exist" containerID="c7fcf800bdae7d184d9d66382e5b8117c385e9307bc17ea74c75edf62a944d8f"
Sep 08 15:51:09.802536 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.802463    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fcf800bdae7d184d9d66382e5b8117c385e9307bc17ea74c75edf62a944d8f"} err="failed to get container status \"c7fcf800bdae7d184d9d66382e5b8117c385e9307bc17ea74c75edf62a944d8f\": rpc error: code = NotFound desc = could not find container \"c7fcf800bdae7d184d9d66382e5b8117c385e9307bc17ea74c75edf62a944d8f\": container with ID starting with c7fcf800bdae7d184d9d66382e5b8117c385e9307bc17ea74c75edf62a944d8f not found: ID does not exist"
Sep 08 15:51:09.804551 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.804521    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pglfr\" (UniqueName: \"kubernetes.io/projected/772cff16-1a92-437f-b747-e59fa7bf82bd-kube-api-access-pglfr\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:51:09.806740 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:51:09.806708    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-2wgfs\" is forbidden: User \"system:node:ip-10-0-134-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-107.ec2.internal' and this object" podUID="772cff16-1a92-437f-b747-e59fa7bf82bd" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:51:09.808562 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:51:09.808538    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-wx8dh\" is forbidden: User \"system:node:ip-10-0-134-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-107.ec2.internal' and this object" podUID="46ca2186-e3f0-4d8e-889d-853ffa7de247" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"
Sep 08 15:51:09.810571 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:51:09.810543    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-wx8dh\" is forbidden: User \"system:node:ip-10-0-134-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-107.ec2.internal' and this object" podUID="46ca2186-e3f0-4d8e-889d-853ffa7de247" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-wx8dh"
Sep 08 15:51:09.812315 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:51:09.812260    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-2wgfs\" is forbidden: User \"system:node:ip-10-0-134-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-107.ec2.internal' and this object" podUID="772cff16-1a92-437f-b747-e59fa7bf82bd" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2wgfs"
Sep 08 15:51:09.899836 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.899811    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46ca2186-e3f0-4d8e-889d-853ffa7de247" path="/var/lib/kubelet/pods/46ca2186-e3f0-4d8e-889d-853ffa7de247/volumes"
Sep 08 15:51:09.900079 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:09.900062    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="772cff16-1a92-437f-b747-e59fa7bf82bd" path="/var/lib/kubelet/pods/772cff16-1a92-437f-b747-e59fa7bf82bd/volumes"
Sep 08 15:51:10.004244 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:10.004207    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"]
Sep 08 15:51:10.007696 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:51:10.007671    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda780cf4c_9615_4eb9_adc5_802b0f2372ae.slice/crio-13a6b45dfd73761def3997742e2963d9dd85232305026cb82892f31340f3a730 WatchSource:0}: Error finding container 13a6b45dfd73761def3997742e2963d9dd85232305026cb82892f31340f3a730: Status 404 returned error can't find the container with id 13a6b45dfd73761def3997742e2963d9dd85232305026cb82892f31340f3a730
Sep 08 15:51:10.799283 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:10.799249    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5" event={"ID":"a780cf4c-9615-4eb9-adc5-802b0f2372ae","Type":"ContainerStarted","Data":"d026a2acad661ae06e3bb527e42a1c8dfb4232760312509bfb8d44658a81ff5f"}
Sep 08 15:51:10.799639 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:10.799303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5" event={"ID":"a780cf4c-9615-4eb9-adc5-802b0f2372ae","Type":"ContainerStarted","Data":"13a6b45dfd73761def3997742e2963d9dd85232305026cb82892f31340f3a730"}
Sep 08 15:51:10.799639 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:10.799333    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"
Sep 08 15:51:10.818811 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:10.818753    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5" podStartSLOduration=1.818734551 podStartE2EDuration="1.818734551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:51:10.818661704 +0000 UTC m=+417.442354027" watchObservedRunningTime="2026-09-08 15:51:10.818734551 +0000 UTC m=+417.442426889"
Sep 08 15:51:21.805551 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:21.805519    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"
Sep 08 15:51:51.651450 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:51.651414    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:51:51.654949 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:51.654930    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"
Sep 08 15:51:51.657087 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:51.657060    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 15:51:51.657201 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:51.657089    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9sb96\""
Sep 08 15:51:51.665012 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:51.664991    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:51:51.691474 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:51.691445    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:51:51.716034 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:51.716010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b4d0763-376e-46c3-8600-792f83a05a50-config-file\") pod \"limitador-limitador-85d9db4d5d-lrcmj\" (UID: \"6b4d0763-376e-46c3-8600-792f83a05a50\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"
Sep 08 15:51:51.716142 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:51.716049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g74cf\" (UniqueName: \"kubernetes.io/projected/6b4d0763-376e-46c3-8600-792f83a05a50-kube-api-access-g74cf\") pod \"limitador-limitador-85d9db4d5d-lrcmj\" (UID: \"6b4d0763-376e-46c3-8600-792f83a05a50\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"
Sep 08 15:51:51.816691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:51.816664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b4d0763-376e-46c3-8600-792f83a05a50-config-file\") pod \"limitador-limitador-85d9db4d5d-lrcmj\" (UID: \"6b4d0763-376e-46c3-8600-792f83a05a50\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"
Sep 08 15:51:51.816815 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:51.816700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g74cf\" (UniqueName: \"kubernetes.io/projected/6b4d0763-376e-46c3-8600-792f83a05a50-kube-api-access-g74cf\") pod \"limitador-limitador-85d9db4d5d-lrcmj\" (UID: \"6b4d0763-376e-46c3-8600-792f83a05a50\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"
Sep 08 15:51:51.817285 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:51.817259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b4d0763-376e-46c3-8600-792f83a05a50-config-file\") pod \"limitador-limitador-85d9db4d5d-lrcmj\" (UID: \"6b4d0763-376e-46c3-8600-792f83a05a50\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"
Sep 08 15:51:51.823999 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:51.823979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g74cf\" (UniqueName: \"kubernetes.io/projected/6b4d0763-376e-46c3-8600-792f83a05a50-kube-api-access-g74cf\") pod \"limitador-limitador-85d9db4d5d-lrcmj\" (UID: \"6b4d0763-376e-46c3-8600-792f83a05a50\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"
Sep 08 15:51:51.965906 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:51.965855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"
Sep 08 15:51:52.072667 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:52.072636    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-7zzj6"]
Sep 08 15:51:52.076100 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:52.076084    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-7zzj6"
Sep 08 15:51:52.078244 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:52.078222    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2znf2\""
Sep 08 15:51:52.082805 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:52.082784    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-7zzj6"]
Sep 08 15:51:52.120100 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:52.120077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5tdx\" (UniqueName: \"kubernetes.io/projected/78d68427-6c87-49c5-978b-dba153f9e71b-kube-api-access-k5tdx\") pod \"authorino-b54df94b4-7zzj6\" (UID: \"78d68427-6c87-49c5-978b-dba153f9e71b\") " pod="kuadrant-system/authorino-b54df94b4-7zzj6"
Sep 08 15:51:52.221180 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:52.221119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5tdx\" (UniqueName: \"kubernetes.io/projected/78d68427-6c87-49c5-978b-dba153f9e71b-kube-api-access-k5tdx\") pod \"authorino-b54df94b4-7zzj6\" (UID: \"78d68427-6c87-49c5-978b-dba153f9e71b\") " pod="kuadrant-system/authorino-b54df94b4-7zzj6"
Sep 08 15:51:52.230711 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:52.230687    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5tdx\" (UniqueName: \"kubernetes.io/projected/78d68427-6c87-49c5-978b-dba153f9e71b-kube-api-access-k5tdx\") pod \"authorino-b54df94b4-7zzj6\" (UID: \"78d68427-6c87-49c5-978b-dba153f9e71b\") " pod="kuadrant-system/authorino-b54df94b4-7zzj6"
Sep 08 15:51:52.233472 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:52.233446    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:51:52.237256 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:51:52.237231    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4d0763_376e_46c3_8600_792f83a05a50.slice/crio-720dee4d0bfe9fb5e4e3b22b32324da00332fe78b775870ca0fca5370af07a69 WatchSource:0}: Error finding container 720dee4d0bfe9fb5e4e3b22b32324da00332fe78b775870ca0fca5370af07a69: Status 404 returned error can't find the container with id 720dee4d0bfe9fb5e4e3b22b32324da00332fe78b775870ca0fca5370af07a69
Sep 08 15:51:52.386349 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:52.386327    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-7zzj6"
Sep 08 15:51:52.655234 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:52.655209    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-7zzj6"]
Sep 08 15:51:52.657643 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:51:52.657617    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78d68427_6c87_49c5_978b_dba153f9e71b.slice/crio-1e92b7f034d35fd24962dc9c5cd70d55bbe005fca0040e925fd358e85872f513 WatchSource:0}: Error finding container 1e92b7f034d35fd24962dc9c5cd70d55bbe005fca0040e925fd358e85872f513: Status 404 returned error can't find the container with id 1e92b7f034d35fd24962dc9c5cd70d55bbe005fca0040e925fd358e85872f513
Sep 08 15:51:52.984515 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:52.984443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-7zzj6" event={"ID":"78d68427-6c87-49c5-978b-dba153f9e71b","Type":"ContainerStarted","Data":"1e92b7f034d35fd24962dc9c5cd70d55bbe005fca0040e925fd358e85872f513"}
Sep 08 15:51:52.985679 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:52.985653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj" event={"ID":"6b4d0763-376e-46c3-8600-792f83a05a50","Type":"ContainerStarted","Data":"720dee4d0bfe9fb5e4e3b22b32324da00332fe78b775870ca0fca5370af07a69"}
Sep 08 15:51:53.992244 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:53.992203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj" event={"ID":"6b4d0763-376e-46c3-8600-792f83a05a50","Type":"ContainerStarted","Data":"4f289f9d178e0f2cb343d3283385e810c623a177c5a672288b9d343586380766"}
Sep 08 15:51:53.992610 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:53.992321    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"
Sep 08 15:51:54.012514 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:54.012329    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj" podStartSLOduration=2.023345229 podStartE2EDuration="3.012309645s" totalImagesPullingTime="988.964416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:51:54.01103712 +0000 UTC m=+460.634729458" watchObservedRunningTime="2026-09-08 15:51:54.012309645 +0000 UTC m=+460.636001974"
Sep 08 15:51:54.998159 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:54.998122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-7zzj6" event={"ID":"78d68427-6c87-49c5-978b-dba153f9e71b","Type":"ContainerStarted","Data":"8f62309740f68263deda843c3f1205d995872714781bf5b0cfbf666dd2f5c0a0"}
Sep 08 15:51:55.013538 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:51:55.013487    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-7zzj6" podStartSLOduration=1.247725457 podStartE2EDuration="3.013473195s" totalImagesPullingTime="1.765747738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:51:55.012964898 +0000 UTC m=+461.636657221" watchObservedRunningTime="2026-09-08 15:51:55.013473195 +0000 UTC m=+461.637165519"
Sep 08 15:52:04.998991 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:04.998959    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"
Sep 08 15:52:06.024084 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:06.024051    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-7zzj6"]
Sep 08 15:52:06.024523 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:06.024247    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-7zzj6" podUID="78d68427-6c87-49c5-978b-dba153f9e71b" containerName="authorino" containerID="cri-o://8f62309740f68263deda843c3f1205d995872714781bf5b0cfbf666dd2f5c0a0" gracePeriod=30
Sep 08 15:52:06.278062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:06.278008    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-7zzj6"
Sep 08 15:52:06.430133 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:06.430102    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5tdx\" (UniqueName: \"kubernetes.io/projected/78d68427-6c87-49c5-978b-dba153f9e71b-kube-api-access-k5tdx\") pod \"78d68427-6c87-49c5-978b-dba153f9e71b\" (UID: \"78d68427-6c87-49c5-978b-dba153f9e71b\") "
Sep 08 15:52:06.432114 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:06.432091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d68427-6c87-49c5-978b-dba153f9e71b-kube-api-access-k5tdx" (OuterVolumeSpecName: "kube-api-access-k5tdx") pod "78d68427-6c87-49c5-978b-dba153f9e71b" (UID: "78d68427-6c87-49c5-978b-dba153f9e71b"). InnerVolumeSpecName "kube-api-access-k5tdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:52:06.530883 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:06.530826    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5tdx\" (UniqueName: \"kubernetes.io/projected/78d68427-6c87-49c5-978b-dba153f9e71b-kube-api-access-k5tdx\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:52:07.057237 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:07.057185    2581 generic.go:356] "Generic (PLEG): container finished" podID="78d68427-6c87-49c5-978b-dba153f9e71b" containerID="8f62309740f68263deda843c3f1205d995872714781bf5b0cfbf666dd2f5c0a0" exitCode=0
Sep 08 15:52:07.057237 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:07.057227    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-7zzj6"
Sep 08 15:52:07.057722 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:07.057254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-7zzj6" event={"ID":"78d68427-6c87-49c5-978b-dba153f9e71b","Type":"ContainerDied","Data":"8f62309740f68263deda843c3f1205d995872714781bf5b0cfbf666dd2f5c0a0"}
Sep 08 15:52:07.057722 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:07.057287    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-7zzj6" event={"ID":"78d68427-6c87-49c5-978b-dba153f9e71b","Type":"ContainerDied","Data":"1e92b7f034d35fd24962dc9c5cd70d55bbe005fca0040e925fd358e85872f513"}
Sep 08 15:52:07.057722 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:07.057305    2581 scope.go:117] "RemoveContainer" containerID="8f62309740f68263deda843c3f1205d995872714781bf5b0cfbf666dd2f5c0a0"
Sep 08 15:52:07.067042 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:07.067022    2581 scope.go:117] "RemoveContainer" containerID="8f62309740f68263deda843c3f1205d995872714781bf5b0cfbf666dd2f5c0a0"
Sep 08 15:52:07.067328 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:52:07.067303    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f62309740f68263deda843c3f1205d995872714781bf5b0cfbf666dd2f5c0a0\": container with ID starting with 8f62309740f68263deda843c3f1205d995872714781bf5b0cfbf666dd2f5c0a0 not found: ID does not exist" containerID="8f62309740f68263deda843c3f1205d995872714781bf5b0cfbf666dd2f5c0a0"
Sep 08 15:52:07.067386 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:07.067337    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f62309740f68263deda843c3f1205d995872714781bf5b0cfbf666dd2f5c0a0"} err="failed to get container status \"8f62309740f68263deda843c3f1205d995872714781bf5b0cfbf666dd2f5c0a0\": rpc error: code = NotFound desc = could not find container \"8f62309740f68263deda843c3f1205d995872714781bf5b0cfbf666dd2f5c0a0\": container with ID starting with 8f62309740f68263deda843c3f1205d995872714781bf5b0cfbf666dd2f5c0a0 not found: ID does not exist"
Sep 08 15:52:07.076991 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:07.076970    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-7zzj6"]
Sep 08 15:52:07.080860 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:07.080840    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-7zzj6"]
Sep 08 15:52:07.900539 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:07.900508    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78d68427-6c87-49c5-978b-dba153f9e71b" path="/var/lib/kubelet/pods/78d68427-6c87-49c5-978b-dba153f9e71b/volumes"
Sep 08 15:52:19.244892 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.244859    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-x2m2q"]
Sep 08 15:52:19.245277 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.245230    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78d68427-6c87-49c5-978b-dba153f9e71b" containerName="authorino"
Sep 08 15:52:19.245277 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.245253    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="78d68427-6c87-49c5-978b-dba153f9e71b" containerName="authorino"
Sep 08 15:52:19.245350 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.245320    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="78d68427-6c87-49c5-978b-dba153f9e71b" containerName="authorino"
Sep 08 15:52:19.249399 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.249375    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-x2m2q"
Sep 08 15:52:19.252450 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.252426    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 08 15:52:19.252575 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.252451    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2znf2\""
Sep 08 15:52:19.255405 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.255361    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-x2m2q"]
Sep 08 15:52:19.326482 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.326453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7cf26809-e791-42bd-9940-d912d96b6b6d-tls-cert\") pod \"authorino-687bd45585-x2m2q\" (UID: \"7cf26809-e791-42bd-9940-d912d96b6b6d\") " pod="kuadrant-system/authorino-687bd45585-x2m2q"
Sep 08 15:52:19.326597 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.326517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnzth\" (UniqueName: \"kubernetes.io/projected/7cf26809-e791-42bd-9940-d912d96b6b6d-kube-api-access-xnzth\") pod \"authorino-687bd45585-x2m2q\" (UID: \"7cf26809-e791-42bd-9940-d912d96b6b6d\") " pod="kuadrant-system/authorino-687bd45585-x2m2q"
Sep 08 15:52:19.381164 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.381138    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-x2m2q"]
Sep 08 15:52:19.381353 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:52:19.381334    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xnzth tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-x2m2q" podUID="7cf26809-e791-42bd-9940-d912d96b6b6d"
Sep 08 15:52:19.409267 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.409241    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-775b68f4f4-jwlgs"]
Sep 08 15:52:19.412339 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.412322    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775b68f4f4-jwlgs"
Sep 08 15:52:19.419602 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.419576    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-775b68f4f4-jwlgs"]
Sep 08 15:52:19.427183 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.427157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7cf26809-e791-42bd-9940-d912d96b6b6d-tls-cert\") pod \"authorino-687bd45585-x2m2q\" (UID: \"7cf26809-e791-42bd-9940-d912d96b6b6d\") " pod="kuadrant-system/authorino-687bd45585-x2m2q"
Sep 08 15:52:19.427275 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.427257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnzth\" (UniqueName: \"kubernetes.io/projected/7cf26809-e791-42bd-9940-d912d96b6b6d-kube-api-access-xnzth\") pod \"authorino-687bd45585-x2m2q\" (UID: \"7cf26809-e791-42bd-9940-d912d96b6b6d\") " pod="kuadrant-system/authorino-687bd45585-x2m2q"
Sep 08 15:52:19.429786 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.429764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7cf26809-e791-42bd-9940-d912d96b6b6d-tls-cert\") pod \"authorino-687bd45585-x2m2q\" (UID: \"7cf26809-e791-42bd-9940-d912d96b6b6d\") " pod="kuadrant-system/authorino-687bd45585-x2m2q"
Sep 08 15:52:19.434342 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.434311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnzth\" (UniqueName: \"kubernetes.io/projected/7cf26809-e791-42bd-9940-d912d96b6b6d-kube-api-access-xnzth\") pod \"authorino-687bd45585-x2m2q\" (UID: \"7cf26809-e791-42bd-9940-d912d96b6b6d\") " pod="kuadrant-system/authorino-687bd45585-x2m2q"
Sep 08 15:52:19.527960 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.527891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf97bca0-c209-45b4-8d83-93460656a8eb-tls-cert\") pod \"authorino-775b68f4f4-jwlgs\" (UID: \"bf97bca0-c209-45b4-8d83-93460656a8eb\") " pod="kuadrant-system/authorino-775b68f4f4-jwlgs"
Sep 08 15:52:19.527960 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.527930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c585f\" (UniqueName: \"kubernetes.io/projected/bf97bca0-c209-45b4-8d83-93460656a8eb-kube-api-access-c585f\") pod \"authorino-775b68f4f4-jwlgs\" (UID: \"bf97bca0-c209-45b4-8d83-93460656a8eb\") " pod="kuadrant-system/authorino-775b68f4f4-jwlgs"
Sep 08 15:52:19.628731 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.628697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf97bca0-c209-45b4-8d83-93460656a8eb-tls-cert\") pod \"authorino-775b68f4f4-jwlgs\" (UID: \"bf97bca0-c209-45b4-8d83-93460656a8eb\") " pod="kuadrant-system/authorino-775b68f4f4-jwlgs"
Sep 08 15:52:19.628837 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.628739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c585f\" (UniqueName: \"kubernetes.io/projected/bf97bca0-c209-45b4-8d83-93460656a8eb-kube-api-access-c585f\") pod \"authorino-775b68f4f4-jwlgs\" (UID: \"bf97bca0-c209-45b4-8d83-93460656a8eb\") " pod="kuadrant-system/authorino-775b68f4f4-jwlgs"
Sep 08 15:52:19.631104 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.631082    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf97bca0-c209-45b4-8d83-93460656a8eb-tls-cert\") pod \"authorino-775b68f4f4-jwlgs\" (UID: \"bf97bca0-c209-45b4-8d83-93460656a8eb\") " pod="kuadrant-system/authorino-775b68f4f4-jwlgs"
Sep 08 15:52:19.636258 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.636236    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c585f\" (UniqueName: \"kubernetes.io/projected/bf97bca0-c209-45b4-8d83-93460656a8eb-kube-api-access-c585f\") pod \"authorino-775b68f4f4-jwlgs\" (UID: \"bf97bca0-c209-45b4-8d83-93460656a8eb\") " pod="kuadrant-system/authorino-775b68f4f4-jwlgs"
Sep 08 15:52:19.721145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.721124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775b68f4f4-jwlgs"
Sep 08 15:52:19.991003 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:19.990978    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-775b68f4f4-jwlgs"]
Sep 08 15:52:19.993474 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:52:19.993447    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf97bca0_c209_45b4_8d83_93460656a8eb.slice/crio-f4cccecd93767ab646d25866774b355fa45d37d509b514acf4b3ea27d2f47d86 WatchSource:0}: Error finding container f4cccecd93767ab646d25866774b355fa45d37d509b514acf4b3ea27d2f47d86: Status 404 returned error can't find the container with id f4cccecd93767ab646d25866774b355fa45d37d509b514acf4b3ea27d2f47d86
Sep 08 15:52:20.122287 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:20.122257    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775b68f4f4-jwlgs" event={"ID":"bf97bca0-c209-45b4-8d83-93460656a8eb","Type":"ContainerStarted","Data":"f4cccecd93767ab646d25866774b355fa45d37d509b514acf4b3ea27d2f47d86"}
Sep 08 15:52:20.122287 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:20.122280    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-x2m2q"
Sep 08 15:52:20.130516 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:20.130492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-x2m2q"
Sep 08 15:52:20.233411 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:20.233360    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7cf26809-e791-42bd-9940-d912d96b6b6d-tls-cert\") pod \"7cf26809-e791-42bd-9940-d912d96b6b6d\" (UID: \"7cf26809-e791-42bd-9940-d912d96b6b6d\") "
Sep 08 15:52:20.233562 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:20.233417    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnzth\" (UniqueName: \"kubernetes.io/projected/7cf26809-e791-42bd-9940-d912d96b6b6d-kube-api-access-xnzth\") pod \"7cf26809-e791-42bd-9940-d912d96b6b6d\" (UID: \"7cf26809-e791-42bd-9940-d912d96b6b6d\") "
Sep 08 15:52:20.235308 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:20.235283    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf26809-e791-42bd-9940-d912d96b6b6d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7cf26809-e791-42bd-9940-d912d96b6b6d" (UID: "7cf26809-e791-42bd-9940-d912d96b6b6d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:52:20.235428 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:20.235309    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf26809-e791-42bd-9940-d912d96b6b6d-kube-api-access-xnzth" (OuterVolumeSpecName: "kube-api-access-xnzth") pod "7cf26809-e791-42bd-9940-d912d96b6b6d" (UID: "7cf26809-e791-42bd-9940-d912d96b6b6d"). InnerVolumeSpecName "kube-api-access-xnzth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:52:20.334147 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:20.334089    2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7cf26809-e791-42bd-9940-d912d96b6b6d-tls-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:52:20.334147 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:20.334115    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnzth\" (UniqueName: \"kubernetes.io/projected/7cf26809-e791-42bd-9940-d912d96b6b6d-kube-api-access-xnzth\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:52:21.127857 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:21.127818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775b68f4f4-jwlgs" event={"ID":"bf97bca0-c209-45b4-8d83-93460656a8eb","Type":"ContainerStarted","Data":"15d2ecbe211092d948b81845af9d15e2d2c5efd55aa21877ef3360902c9bacbb"}
Sep 08 15:52:21.128010 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:21.127871    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-x2m2q"
Sep 08 15:52:21.147014 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:21.146970    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-775b68f4f4-jwlgs" podStartSLOduration=1.203015296 podStartE2EDuration="2.146956175s" totalImagesPullingTime="943.940879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:52:21.145627775 +0000 UTC m=+487.769320100" watchObservedRunningTime="2026-09-08 15:52:21.146956175 +0000 UTC m=+487.770648501"
Sep 08 15:52:21.168562 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:21.168536    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-x2m2q"]
Sep 08 15:52:21.174571 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:21.174549    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-x2m2q"]
Sep 08 15:52:21.900311 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:52:21.900278    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cf26809-e791-42bd-9940-d912d96b6b6d" path="/var/lib/kubelet/pods/7cf26809-e791-42bd-9940-d912d96b6b6d/volumes"
Sep 08 15:53:02.059484 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.059443    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57"]
Sep 08 15:53:02.063148 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.063126    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57"
Sep 08 15:53:02.065740 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.065707    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 08 15:53:02.066040 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.066013    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6dgjp\""
Sep 08 15:53:02.081983 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.081956    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57"]
Sep 08 15:53:02.122616 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.122582    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:53:02.135328 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.135290    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac0e430-ceb6-4fd8-aac3-db1b5911ba65-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6qm57\" (UID: \"3ac0e430-ceb6-4fd8-aac3-db1b5911ba65\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57"
Sep 08 15:53:02.135503 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.135335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x8mp\" (UniqueName: \"kubernetes.io/projected/3ac0e430-ceb6-4fd8-aac3-db1b5911ba65-kube-api-access-5x8mp\") pod \"payload-pre-processing-59fbc47bf8-6qm57\" (UID: \"3ac0e430-ceb6-4fd8-aac3-db1b5911ba65\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57"
Sep 08 15:53:02.236348 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.236303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x8mp\" (UniqueName: \"kubernetes.io/projected/3ac0e430-ceb6-4fd8-aac3-db1b5911ba65-kube-api-access-5x8mp\") pod \"payload-pre-processing-59fbc47bf8-6qm57\" (UID: \"3ac0e430-ceb6-4fd8-aac3-db1b5911ba65\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57"
Sep 08 15:53:02.236556 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.236488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac0e430-ceb6-4fd8-aac3-db1b5911ba65-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6qm57\" (UID: \"3ac0e430-ceb6-4fd8-aac3-db1b5911ba65\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57"
Sep 08 15:53:02.237350 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.237317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac0e430-ceb6-4fd8-aac3-db1b5911ba65-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6qm57\" (UID: \"3ac0e430-ceb6-4fd8-aac3-db1b5911ba65\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57"
Sep 08 15:53:02.247155 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.247126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x8mp\" (UniqueName: \"kubernetes.io/projected/3ac0e430-ceb6-4fd8-aac3-db1b5911ba65-kube-api-access-5x8mp\") pod \"payload-pre-processing-59fbc47bf8-6qm57\" (UID: \"3ac0e430-ceb6-4fd8-aac3-db1b5911ba65\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57"
Sep 08 15:53:02.375250 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.375165    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57"
Sep 08 15:53:02.646813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:02.646780    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57"]
Sep 08 15:53:02.650477 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:53:02.650448    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac0e430_ceb6_4fd8_aac3_db1b5911ba65.slice/crio-dfb4f201c501473ba54828c9ccf3377982faa115306bc644f48fc2d2ed219741 WatchSource:0}: Error finding container dfb4f201c501473ba54828c9ccf3377982faa115306bc644f48fc2d2ed219741: Status 404 returned error can't find the container with id dfb4f201c501473ba54828c9ccf3377982faa115306bc644f48fc2d2ed219741
Sep 08 15:53:03.320698 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:03.320652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57" event={"ID":"3ac0e430-ceb6-4fd8-aac3-db1b5911ba65","Type":"ContainerStarted","Data":"dfb4f201c501473ba54828c9ccf3377982faa115306bc644f48fc2d2ed219741"}
Sep 08 15:53:05.332191 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:05.332153    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57" event={"ID":"3ac0e430-ceb6-4fd8-aac3-db1b5911ba65","Type":"ContainerStarted","Data":"382c675bd8050bcc1d0f5d4bb5a95378bf399219e64c1fabb03fb402d1334072"}
Sep 08 15:53:05.332608 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:05.332249    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57"
Sep 08 15:53:05.350117 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:05.350061    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57" podStartSLOduration=0.87159307 podStartE2EDuration="3.350046294s" totalImagesPullingTime="2.478453224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:53:05.348215046 +0000 UTC m=+531.971907369" watchObservedRunningTime="2026-09-08 15:53:05.350046294 +0000 UTC m=+531.973738618"
Sep 08 15:53:16.338570 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:16.338538    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6qm57"
Sep 08 15:53:21.701519 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:21.701481    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:53:25.283818 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:25.283778    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:53:35.555182 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.555148    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"]
Sep 08 15:53:35.558837 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.558822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.562130 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.562103    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 08 15:53:35.562327 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.562312    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-668bh\""
Sep 08 15:53:35.562399 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.562324    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 15:53:35.562399 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.562331    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 15:53:35.569411 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.569367    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"]
Sep 08 15:53:35.717541 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.717506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.717794 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.717550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.717794 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.717591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.717794 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.717676    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.717794 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.717752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6vnv\" (UniqueName: \"kubernetes.io/projected/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-kube-api-access-j6vnv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.717794 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.717783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.818676 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.818590    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.818676 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.818628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.818882 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.818780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.818882 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.818844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6vnv\" (UniqueName: \"kubernetes.io/projected/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-kube-api-access-j6vnv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.818882 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.818876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.819028 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.818947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.819028 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.818998    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.819133 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.819029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.819187 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.819159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.820989 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.820970    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.821378 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.821356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.826900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.826876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6vnv\" (UniqueName: \"kubernetes.io/projected/2c052f9f-6efe-4788-88e5-9ed2318a3cbb-kube-api-access-j6vnv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h\" (UID: \"2c052f9f-6efe-4788-88e5-9ed2318a3cbb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:35.869910 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:35.869873    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:36.145781 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:36.145746    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"]
Sep 08 15:53:36.149790 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:53:36.149766    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c052f9f_6efe_4788_88e5_9ed2318a3cbb.slice/crio-bda3967bd91abdab5f62038b190d3bb89175d2ac0764c2e769959910b6101338 WatchSource:0}: Error finding container bda3967bd91abdab5f62038b190d3bb89175d2ac0764c2e769959910b6101338: Status 404 returned error can't find the container with id bda3967bd91abdab5f62038b190d3bb89175d2ac0764c2e769959910b6101338
Sep 08 15:53:36.480842 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:36.480751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h" event={"ID":"2c052f9f-6efe-4788-88e5-9ed2318a3cbb","Type":"ContainerStarted","Data":"bda3967bd91abdab5f62038b190d3bb89175d2ac0764c2e769959910b6101338"}
Sep 08 15:53:36.569230 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:36.569190    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:53:37.487074 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:37.487029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h" event={"ID":"2c052f9f-6efe-4788-88e5-9ed2318a3cbb","Type":"ContainerStarted","Data":"c4d7a172c6af67c98ba3b5cf73849b0d6e9ce622c680fadf494a41389b5e50e0"}
Sep 08 15:53:37.507487 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:37.507430    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h" podStartSLOduration=1.3929467739999999 podStartE2EDuration="2.507413729s" totalImagesPullingTime="1.114466955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:53:37.506137783 +0000 UTC m=+564.129830119" watchObservedRunningTime="2026-09-08 15:53:37.507413729 +0000 UTC m=+564.131106048"
Sep 08 15:53:45.870513 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:45.870476    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:45.870962 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:45.870709    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:45.884747 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:45.884724    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:46.544794 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:46.544767    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h"
Sep 08 15:53:49.371683 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:53:49.371640    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:54:01.576648 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:01.576607    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:54:07.476454 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.476414    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"]
Sep 08 15:54:07.483302 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.483280    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.487040 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.487014    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 08 15:54:07.489711 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.489684    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"]
Sep 08 15:54:07.598999 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.598969    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e599441e-6561-4eb4-a54b-6f49fccd80c2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.599144 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.599009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e599441e-6561-4eb4-a54b-6f49fccd80c2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.599144 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.599029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e599441e-6561-4eb4-a54b-6f49fccd80c2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.599144 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.599107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e599441e-6561-4eb4-a54b-6f49fccd80c2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.599144 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.599138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e599441e-6561-4eb4-a54b-6f49fccd80c2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.599280 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.599161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4hdv\" (UniqueName: \"kubernetes.io/projected/e599441e-6561-4eb4-a54b-6f49fccd80c2-kube-api-access-p4hdv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.699648 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.699621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e599441e-6561-4eb4-a54b-6f49fccd80c2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.699788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.699653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e599441e-6561-4eb4-a54b-6f49fccd80c2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.699788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.699673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4hdv\" (UniqueName: \"kubernetes.io/projected/e599441e-6561-4eb4-a54b-6f49fccd80c2-kube-api-access-p4hdv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.699788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.699713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e599441e-6561-4eb4-a54b-6f49fccd80c2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.699788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.699738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e599441e-6561-4eb4-a54b-6f49fccd80c2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.699788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.699756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e599441e-6561-4eb4-a54b-6f49fccd80c2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.700045 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.699992    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e599441e-6561-4eb4-a54b-6f49fccd80c2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.700101 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.700074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e599441e-6561-4eb4-a54b-6f49fccd80c2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.700193 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.700175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e599441e-6561-4eb4-a54b-6f49fccd80c2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.701941 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.701920    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e599441e-6561-4eb4-a54b-6f49fccd80c2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.702109 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.702092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e599441e-6561-4eb4-a54b-6f49fccd80c2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.709160 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.709130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4hdv\" (UniqueName: \"kubernetes.io/projected/e599441e-6561-4eb4-a54b-6f49fccd80c2-kube-api-access-p4hdv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmv6x\" (UID: \"e599441e-6561-4eb4-a54b-6f49fccd80c2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:07.797892 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:07.797837    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:08.069320 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:08.069148    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"]
Sep 08 15:54:08.071486 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:54:08.071456    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode599441e_6561_4eb4_a54b_6f49fccd80c2.slice/crio-d81fc3f7b4cd1ae3629b59721166c6634778d9324d691bb0f6f096bf990f72d2 WatchSource:0}: Error finding container d81fc3f7b4cd1ae3629b59721166c6634778d9324d691bb0f6f096bf990f72d2: Status 404 returned error can't find the container with id d81fc3f7b4cd1ae3629b59721166c6634778d9324d691bb0f6f096bf990f72d2
Sep 08 15:54:08.636807 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:08.636774    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x" event={"ID":"e599441e-6561-4eb4-a54b-6f49fccd80c2","Type":"ContainerStarted","Data":"7969b4c243e8c8011a3a1da7878da6730ddfbd4cbdfb814b7587c348adc2e6e4"}
Sep 08 15:54:08.637182 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:08.636824    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x" event={"ID":"e599441e-6561-4eb4-a54b-6f49fccd80c2","Type":"ContainerStarted","Data":"d81fc3f7b4cd1ae3629b59721166c6634778d9324d691bb0f6f096bf990f72d2"}
Sep 08 15:54:08.653107 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:08.653060    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x" podStartSLOduration=1.468369654 podStartE2EDuration="1.653045346s" totalImagesPullingTime="184.675692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:54:08.651833242 +0000 UTC m=+595.275525569" watchObservedRunningTime="2026-09-08 15:54:08.653045346 +0000 UTC m=+595.276737670"
Sep 08 15:54:09.883936 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:09.883899    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:54:13.838152 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:13.838126    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 15:54:13.844667 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:13.844642    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 15:54:17.798497 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:17.798458    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:17.798934 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:17.798524    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:17.811427 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:17.811384    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:18.698433 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:18.698388    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmv6x"
Sep 08 15:54:21.369884 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:54:21.369854    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:55:14.601853 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:14.601819    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-m279r"]
Sep 08 15:55:14.605177 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:14.605159    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-m279r"
Sep 08 15:55:14.607639 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:14.607622    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-lrbxw\""
Sep 08 15:55:14.614406 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:14.614366    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-m279r"]
Sep 08 15:55:14.738183 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:14.738152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbfq\" (UniqueName: \"kubernetes.io/projected/451adf51-dc13-41b2-b0cd-bbce32e8a2d2-kube-api-access-mbbfq\") pod \"authorino-operator-76b478bd78-m279r\" (UID: \"451adf51-dc13-41b2-b0cd-bbce32e8a2d2\") " pod="kuadrant-system/authorino-operator-76b478bd78-m279r"
Sep 08 15:55:14.838700 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:14.838672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbfq\" (UniqueName: \"kubernetes.io/projected/451adf51-dc13-41b2-b0cd-bbce32e8a2d2-kube-api-access-mbbfq\") pod \"authorino-operator-76b478bd78-m279r\" (UID: \"451adf51-dc13-41b2-b0cd-bbce32e8a2d2\") " pod="kuadrant-system/authorino-operator-76b478bd78-m279r"
Sep 08 15:55:14.848450 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:14.848427    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbfq\" (UniqueName: \"kubernetes.io/projected/451adf51-dc13-41b2-b0cd-bbce32e8a2d2-kube-api-access-mbbfq\") pod \"authorino-operator-76b478bd78-m279r\" (UID: \"451adf51-dc13-41b2-b0cd-bbce32e8a2d2\") " pod="kuadrant-system/authorino-operator-76b478bd78-m279r"
Sep 08 15:55:14.916366 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:14.916345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-m279r"
Sep 08 15:55:15.197270 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:15.197206    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-m279r"]
Sep 08 15:55:15.200673 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:55:15.200645    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451adf51_dc13_41b2_b0cd_bbce32e8a2d2.slice/crio-32f20e2e33054c50b0827ead3e53cc7647772151d128058542f9a6d3fae0ed46 WatchSource:0}: Error finding container 32f20e2e33054c50b0827ead3e53cc7647772151d128058542f9a6d3fae0ed46: Status 404 returned error can't find the container with id 32f20e2e33054c50b0827ead3e53cc7647772151d128058542f9a6d3fae0ed46
Sep 08 15:55:15.202830 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:15.202812    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:55:15.958598 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:15.958552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-m279r" event={"ID":"451adf51-dc13-41b2-b0cd-bbce32e8a2d2","Type":"ContainerStarted","Data":"32f20e2e33054c50b0827ead3e53cc7647772151d128058542f9a6d3fae0ed46"}
Sep 08 15:55:17.970923 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:17.970882    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-m279r" event={"ID":"451adf51-dc13-41b2-b0cd-bbce32e8a2d2","Type":"ContainerStarted","Data":"575bead6416d537a06f7ec9096eb488657085e75123a29ab0715af58af55a052"}
Sep 08 15:55:17.971362 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:17.970987    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-m279r"
Sep 08 15:55:18.004676 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:18.004612    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-m279r" podStartSLOduration=1.850801491 podStartE2EDuration="4.004598106s" totalImagesPullingTime="2.153796615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:55:18.003438356 +0000 UTC m=+664.627130677" watchObservedRunningTime="2026-09-08 15:55:18.004598106 +0000 UTC m=+664.628290490"
Sep 08 15:55:25.674650 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:25.674613    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"]
Sep 08 15:55:25.675179 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:25.674807    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5" podUID="a780cf4c-9615-4eb9-adc5-802b0f2372ae" containerName="manager" containerID="cri-o://d026a2acad661ae06e3bb527e42a1c8dfb4232760312509bfb8d44658a81ff5f" gracePeriod=10
Sep 08 15:55:25.926239 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:25.926190    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"
Sep 08 15:55:26.009075 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.009043    2581 generic.go:356] "Generic (PLEG): container finished" podID="a780cf4c-9615-4eb9-adc5-802b0f2372ae" containerID="d026a2acad661ae06e3bb527e42a1c8dfb4232760312509bfb8d44658a81ff5f" exitCode=0
Sep 08 15:55:26.009225 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.009088    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"
Sep 08 15:55:26.009225 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.009092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5" event={"ID":"a780cf4c-9615-4eb9-adc5-802b0f2372ae","Type":"ContainerDied","Data":"d026a2acad661ae06e3bb527e42a1c8dfb4232760312509bfb8d44658a81ff5f"}
Sep 08 15:55:26.009225 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.009214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5" event={"ID":"a780cf4c-9615-4eb9-adc5-802b0f2372ae","Type":"ContainerDied","Data":"13a6b45dfd73761def3997742e2963d9dd85232305026cb82892f31340f3a730"}
Sep 08 15:55:26.009381 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.009244    2581 scope.go:117] "RemoveContainer" containerID="d026a2acad661ae06e3bb527e42a1c8dfb4232760312509bfb8d44658a81ff5f"
Sep 08 15:55:26.017208 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.017192    2581 scope.go:117] "RemoveContainer" containerID="d026a2acad661ae06e3bb527e42a1c8dfb4232760312509bfb8d44658a81ff5f"
Sep 08 15:55:26.017489 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:55:26.017471    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d026a2acad661ae06e3bb527e42a1c8dfb4232760312509bfb8d44658a81ff5f\": container with ID starting with d026a2acad661ae06e3bb527e42a1c8dfb4232760312509bfb8d44658a81ff5f not found: ID does not exist" containerID="d026a2acad661ae06e3bb527e42a1c8dfb4232760312509bfb8d44658a81ff5f"
Sep 08 15:55:26.017534 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.017497    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d026a2acad661ae06e3bb527e42a1c8dfb4232760312509bfb8d44658a81ff5f"} err="failed to get container status \"d026a2acad661ae06e3bb527e42a1c8dfb4232760312509bfb8d44658a81ff5f\": rpc error: code = NotFound desc = could not find container \"d026a2acad661ae06e3bb527e42a1c8dfb4232760312509bfb8d44658a81ff5f\": container with ID starting with d026a2acad661ae06e3bb527e42a1c8dfb4232760312509bfb8d44658a81ff5f not found: ID does not exist"
Sep 08 15:55:26.030811 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.030793    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m2rs\" (UniqueName: \"kubernetes.io/projected/a780cf4c-9615-4eb9-adc5-802b0f2372ae-kube-api-access-2m2rs\") pod \"a780cf4c-9615-4eb9-adc5-802b0f2372ae\" (UID: \"a780cf4c-9615-4eb9-adc5-802b0f2372ae\") "
Sep 08 15:55:26.032586 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.032557    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a780cf4c-9615-4eb9-adc5-802b0f2372ae-kube-api-access-2m2rs" (OuterVolumeSpecName: "kube-api-access-2m2rs") pod "a780cf4c-9615-4eb9-adc5-802b0f2372ae" (UID: "a780cf4c-9615-4eb9-adc5-802b0f2372ae"). InnerVolumeSpecName "kube-api-access-2m2rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:55:26.132119 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.132096    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m2rs\" (UniqueName: \"kubernetes.io/projected/a780cf4c-9615-4eb9-adc5-802b0f2372ae-kube-api-access-2m2rs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:55:26.342183 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.342154    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"]
Sep 08 15:55:26.345878 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.345853    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-rgkn5"]
Sep 08 15:55:26.661163 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.661137    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcb58cd9f-6bx5d"]
Sep 08 15:55:26.661525 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.661512    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a780cf4c-9615-4eb9-adc5-802b0f2372ae" containerName="manager"
Sep 08 15:55:26.661578 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.661528    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a780cf4c-9615-4eb9-adc5-802b0f2372ae" containerName="manager"
Sep 08 15:55:26.661616 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.661585    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a780cf4c-9615-4eb9-adc5-802b0f2372ae" containerName="manager"
Sep 08 15:55:26.665646 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.665630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d"
Sep 08 15:55:26.676546 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.676524    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcb58cd9f-6bx5d"]
Sep 08 15:55:26.735832 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.735813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8698\" (UniqueName: \"kubernetes.io/projected/932644db-ca94-4804-ac21-f1981aa21e23-kube-api-access-b8698\") pod \"authorino-7bcb58cd9f-6bx5d\" (UID: \"932644db-ca94-4804-ac21-f1981aa21e23\") " pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d"
Sep 08 15:55:26.735933 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.735867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/932644db-ca94-4804-ac21-f1981aa21e23-tls-cert\") pod \"authorino-7bcb58cd9f-6bx5d\" (UID: \"932644db-ca94-4804-ac21-f1981aa21e23\") " pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d"
Sep 08 15:55:26.837180 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.837152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/932644db-ca94-4804-ac21-f1981aa21e23-tls-cert\") pod \"authorino-7bcb58cd9f-6bx5d\" (UID: \"932644db-ca94-4804-ac21-f1981aa21e23\") " pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d"
Sep 08 15:55:26.837292 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.837212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8698\" (UniqueName: \"kubernetes.io/projected/932644db-ca94-4804-ac21-f1981aa21e23-kube-api-access-b8698\") pod \"authorino-7bcb58cd9f-6bx5d\" (UID: \"932644db-ca94-4804-ac21-f1981aa21e23\") " pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d"
Sep 08 15:55:26.839547 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.839526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/932644db-ca94-4804-ac21-f1981aa21e23-tls-cert\") pod \"authorino-7bcb58cd9f-6bx5d\" (UID: \"932644db-ca94-4804-ac21-f1981aa21e23\") " pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d"
Sep 08 15:55:26.845576 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.845548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8698\" (UniqueName: \"kubernetes.io/projected/932644db-ca94-4804-ac21-f1981aa21e23-kube-api-access-b8698\") pod \"authorino-7bcb58cd9f-6bx5d\" (UID: \"932644db-ca94-4804-ac21-f1981aa21e23\") " pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d"
Sep 08 15:55:26.974253 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:26.974200    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d"
Sep 08 15:55:27.244488 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:27.244427    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcb58cd9f-6bx5d"]
Sep 08 15:55:27.246993 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:55:27.246968    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod932644db_ca94_4804_ac21_f1981aa21e23.slice/crio-14193d0da2543badf14d638b3a70160c75b0001609e00ad31d6c16961d39dac8 WatchSource:0}: Error finding container 14193d0da2543badf14d638b3a70160c75b0001609e00ad31d6c16961d39dac8: Status 404 returned error can't find the container with id 14193d0da2543badf14d638b3a70160c75b0001609e00ad31d6c16961d39dac8
Sep 08 15:55:27.907757 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:27.907722    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a780cf4c-9615-4eb9-adc5-802b0f2372ae" path="/var/lib/kubelet/pods/a780cf4c-9615-4eb9-adc5-802b0f2372ae/volumes"
Sep 08 15:55:28.024498 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:28.024458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d" event={"ID":"932644db-ca94-4804-ac21-f1981aa21e23","Type":"ContainerStarted","Data":"b7de8cfa4ac535af3c02e735e0285b6dddb7bac9a48886017965bdac1a46aaf6"}
Sep 08 15:55:28.024678 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:28.024509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d" event={"ID":"932644db-ca94-4804-ac21-f1981aa21e23","Type":"ContainerStarted","Data":"14193d0da2543badf14d638b3a70160c75b0001609e00ad31d6c16961d39dac8"}
Sep 08 15:55:28.041311 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:28.041258    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d" podStartSLOduration=1.384676097 podStartE2EDuration="2.041242946s" totalImagesPullingTime="656.566849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:55:28.04001358 +0000 UTC m=+674.663705904" watchObservedRunningTime="2026-09-08 15:55:28.041242946 +0000 UTC m=+674.664935297"
Sep 08 15:55:28.072495 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:28.072461    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-775b68f4f4-jwlgs"]
Sep 08 15:55:28.072699 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:28.072668    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-775b68f4f4-jwlgs" podUID="bf97bca0-c209-45b4-8d83-93460656a8eb" containerName="authorino" containerID="cri-o://15d2ecbe211092d948b81845af9d15e2d2c5efd55aa21877ef3360902c9bacbb" gracePeriod=30
Sep 08 15:55:28.317241 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:28.317218    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775b68f4f4-jwlgs"
Sep 08 15:55:28.451447 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:28.451338    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf97bca0-c209-45b4-8d83-93460656a8eb-tls-cert\") pod \"bf97bca0-c209-45b4-8d83-93460656a8eb\" (UID: \"bf97bca0-c209-45b4-8d83-93460656a8eb\") "
Sep 08 15:55:28.451607 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:28.451501    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c585f\" (UniqueName: \"kubernetes.io/projected/bf97bca0-c209-45b4-8d83-93460656a8eb-kube-api-access-c585f\") pod \"bf97bca0-c209-45b4-8d83-93460656a8eb\" (UID: \"bf97bca0-c209-45b4-8d83-93460656a8eb\") "
Sep 08 15:55:28.453779 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:28.453748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf97bca0-c209-45b4-8d83-93460656a8eb-kube-api-access-c585f" (OuterVolumeSpecName: "kube-api-access-c585f") pod "bf97bca0-c209-45b4-8d83-93460656a8eb" (UID: "bf97bca0-c209-45b4-8d83-93460656a8eb"). InnerVolumeSpecName "kube-api-access-c585f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:55:28.462351 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:28.462319    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf97bca0-c209-45b4-8d83-93460656a8eb-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "bf97bca0-c209-45b4-8d83-93460656a8eb" (UID: "bf97bca0-c209-45b4-8d83-93460656a8eb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:55:28.552868 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:28.552829    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c585f\" (UniqueName: \"kubernetes.io/projected/bf97bca0-c209-45b4-8d83-93460656a8eb-kube-api-access-c585f\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:55:28.552868 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:28.552862    2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf97bca0-c209-45b4-8d83-93460656a8eb-tls-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:55:28.978693 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:28.978660    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-m279r"
Sep 08 15:55:29.030865 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:29.030831    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf97bca0-c209-45b4-8d83-93460656a8eb" containerID="15d2ecbe211092d948b81845af9d15e2d2c5efd55aa21877ef3360902c9bacbb" exitCode=0
Sep 08 15:55:29.031011 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:29.030876    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775b68f4f4-jwlgs"
Sep 08 15:55:29.031011 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:29.030871    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775b68f4f4-jwlgs" event={"ID":"bf97bca0-c209-45b4-8d83-93460656a8eb","Type":"ContainerDied","Data":"15d2ecbe211092d948b81845af9d15e2d2c5efd55aa21877ef3360902c9bacbb"}
Sep 08 15:55:29.031011 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:29.030922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775b68f4f4-jwlgs" event={"ID":"bf97bca0-c209-45b4-8d83-93460656a8eb","Type":"ContainerDied","Data":"f4cccecd93767ab646d25866774b355fa45d37d509b514acf4b3ea27d2f47d86"}
Sep 08 15:55:29.031011 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:29.030944    2581 scope.go:117] "RemoveContainer" containerID="15d2ecbe211092d948b81845af9d15e2d2c5efd55aa21877ef3360902c9bacbb"
Sep 08 15:55:29.041305 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:29.041286    2581 scope.go:117] "RemoveContainer" containerID="15d2ecbe211092d948b81845af9d15e2d2c5efd55aa21877ef3360902c9bacbb"
Sep 08 15:55:29.041719 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:55:29.041690    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d2ecbe211092d948b81845af9d15e2d2c5efd55aa21877ef3360902c9bacbb\": container with ID starting with 15d2ecbe211092d948b81845af9d15e2d2c5efd55aa21877ef3360902c9bacbb not found: ID does not exist" containerID="15d2ecbe211092d948b81845af9d15e2d2c5efd55aa21877ef3360902c9bacbb"
Sep 08 15:55:29.041830 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:29.041729    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d2ecbe211092d948b81845af9d15e2d2c5efd55aa21877ef3360902c9bacbb"} err="failed to get container status \"15d2ecbe211092d948b81845af9d15e2d2c5efd55aa21877ef3360902c9bacbb\": rpc error: code = NotFound desc = could not find container \"15d2ecbe211092d948b81845af9d15e2d2c5efd55aa21877ef3360902c9bacbb\": container with ID starting with 15d2ecbe211092d948b81845af9d15e2d2c5efd55aa21877ef3360902c9bacbb not found: ID does not exist"
Sep 08 15:55:29.070487 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:29.070454    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-775b68f4f4-jwlgs"]
Sep 08 15:55:29.089844 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:29.089813    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-775b68f4f4-jwlgs"]
Sep 08 15:55:29.901774 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:29.901741    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf97bca0-c209-45b4-8d83-93460656a8eb" path="/var/lib/kubelet/pods/bf97bca0-c209-45b4-8d83-93460656a8eb/volumes"
Sep 08 15:55:29.973572 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:29.973538    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"]
Sep 08 15:55:29.973737 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:29.973718    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps" podUID="59d7537e-e948-4fde-930c-dd50424db8cc" containerName="manager" containerID="cri-o://ca9b6740eeb7e8bd2424163cdb37e926f184f075fb37fe50ecdd6042d27029ec" gracePeriod=10
Sep 08 15:55:30.214101 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:30.214072    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"
Sep 08 15:55:30.369743 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:30.369710    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68whp\" (UniqueName: \"kubernetes.io/projected/59d7537e-e948-4fde-930c-dd50424db8cc-kube-api-access-68whp\") pod \"59d7537e-e948-4fde-930c-dd50424db8cc\" (UID: \"59d7537e-e948-4fde-930c-dd50424db8cc\") "
Sep 08 15:55:30.371758 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:30.371724    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d7537e-e948-4fde-930c-dd50424db8cc-kube-api-access-68whp" (OuterVolumeSpecName: "kube-api-access-68whp") pod "59d7537e-e948-4fde-930c-dd50424db8cc" (UID: "59d7537e-e948-4fde-930c-dd50424db8cc"). InnerVolumeSpecName "kube-api-access-68whp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:55:30.470908 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:30.470826    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68whp\" (UniqueName: \"kubernetes.io/projected/59d7537e-e948-4fde-930c-dd50424db8cc-kube-api-access-68whp\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:55:31.044715 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:31.044677    2581 generic.go:356] "Generic (PLEG): container finished" podID="59d7537e-e948-4fde-930c-dd50424db8cc" containerID="ca9b6740eeb7e8bd2424163cdb37e926f184f075fb37fe50ecdd6042d27029ec" exitCode=0
Sep 08 15:55:31.044875 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:31.044722    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"
Sep 08 15:55:31.044875 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:31.044769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps" event={"ID":"59d7537e-e948-4fde-930c-dd50424db8cc","Type":"ContainerDied","Data":"ca9b6740eeb7e8bd2424163cdb37e926f184f075fb37fe50ecdd6042d27029ec"}
Sep 08 15:55:31.044875 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:31.044834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vxpps" event={"ID":"59d7537e-e948-4fde-930c-dd50424db8cc","Type":"ContainerDied","Data":"0159e275a94179af8b0010fedd43048561ecb155c24e890c66c7304558026237"}
Sep 08 15:55:31.044875 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:31.044871    2581 scope.go:117] "RemoveContainer" containerID="ca9b6740eeb7e8bd2424163cdb37e926f184f075fb37fe50ecdd6042d27029ec"
Sep 08 15:55:31.053829 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:31.053812    2581 scope.go:117] "RemoveContainer" containerID="ca9b6740eeb7e8bd2424163cdb37e926f184f075fb37fe50ecdd6042d27029ec"
Sep 08 15:55:31.054105 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:55:31.054089    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9b6740eeb7e8bd2424163cdb37e926f184f075fb37fe50ecdd6042d27029ec\": container with ID starting with ca9b6740eeb7e8bd2424163cdb37e926f184f075fb37fe50ecdd6042d27029ec not found: ID does not exist" containerID="ca9b6740eeb7e8bd2424163cdb37e926f184f075fb37fe50ecdd6042d27029ec"
Sep 08 15:55:31.054143 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:31.054112    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9b6740eeb7e8bd2424163cdb37e926f184f075fb37fe50ecdd6042d27029ec"} err="failed to get container status \"ca9b6740eeb7e8bd2424163cdb37e926f184f075fb37fe50ecdd6042d27029ec\": rpc error: code = NotFound desc = could not find container \"ca9b6740eeb7e8bd2424163cdb37e926f184f075fb37fe50ecdd6042d27029ec\": container with ID starting with ca9b6740eeb7e8bd2424163cdb37e926f184f075fb37fe50ecdd6042d27029ec not found: ID does not exist"
Sep 08 15:55:31.070209 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:31.070186    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"]
Sep 08 15:55:31.073921 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:31.073903    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-vxpps"]
Sep 08 15:55:31.900174 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:31.900137    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59d7537e-e948-4fde-930c-dd50424db8cc" path="/var/lib/kubelet/pods/59d7537e-e948-4fde-930c-dd50424db8cc/volumes"
Sep 08 15:55:46.868354 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:46.868323    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b89456cc-lj4m5"]
Sep 08 15:55:46.868788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:46.868666    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf97bca0-c209-45b4-8d83-93460656a8eb" containerName="authorino"
Sep 08 15:55:46.868788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:46.868681    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf97bca0-c209-45b4-8d83-93460656a8eb" containerName="authorino"
Sep 08 15:55:46.868788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:46.868704    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d7537e-e948-4fde-930c-dd50424db8cc" containerName="manager"
Sep 08 15:55:46.868788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:46.868710    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d7537e-e948-4fde-930c-dd50424db8cc" containerName="manager"
Sep 08 15:55:46.868788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:46.868770    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="59d7537e-e948-4fde-930c-dd50424db8cc" containerName="manager"
Sep 08 15:55:46.868788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:46.868781    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf97bca0-c209-45b4-8d83-93460656a8eb" containerName="authorino"
Sep 08 15:55:46.873759 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:46.873739    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b89456cc-lj4m5"
Sep 08 15:55:46.879817 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:46.879789    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b89456cc-lj4m5"]
Sep 08 15:55:46.905105 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:46.905078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9293816e-2555-4268-9e94-d87394144cd3-tls-cert\") pod \"authorino-7b89456cc-lj4m5\" (UID: \"9293816e-2555-4268-9e94-d87394144cd3\") " pod="kuadrant-system/authorino-7b89456cc-lj4m5"
Sep 08 15:55:46.905231 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:46.905129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ntq7\" (UniqueName: \"kubernetes.io/projected/9293816e-2555-4268-9e94-d87394144cd3-kube-api-access-7ntq7\") pod \"authorino-7b89456cc-lj4m5\" (UID: \"9293816e-2555-4268-9e94-d87394144cd3\") " pod="kuadrant-system/authorino-7b89456cc-lj4m5"
Sep 08 15:55:47.005930 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:47.005892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ntq7\" (UniqueName: \"kubernetes.io/projected/9293816e-2555-4268-9e94-d87394144cd3-kube-api-access-7ntq7\") pod \"authorino-7b89456cc-lj4m5\" (UID: \"9293816e-2555-4268-9e94-d87394144cd3\") " pod="kuadrant-system/authorino-7b89456cc-lj4m5"
Sep 08 15:55:47.006082 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:47.005999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9293816e-2555-4268-9e94-d87394144cd3-tls-cert\") pod \"authorino-7b89456cc-lj4m5\" (UID: \"9293816e-2555-4268-9e94-d87394144cd3\") " pod="kuadrant-system/authorino-7b89456cc-lj4m5"
Sep 08 15:55:47.008525 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:47.008498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9293816e-2555-4268-9e94-d87394144cd3-tls-cert\") pod \"authorino-7b89456cc-lj4m5\" (UID: \"9293816e-2555-4268-9e94-d87394144cd3\") " pod="kuadrant-system/authorino-7b89456cc-lj4m5"
Sep 08 15:55:47.016851 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:47.016829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ntq7\" (UniqueName: \"kubernetes.io/projected/9293816e-2555-4268-9e94-d87394144cd3-kube-api-access-7ntq7\") pod \"authorino-7b89456cc-lj4m5\" (UID: \"9293816e-2555-4268-9e94-d87394144cd3\") " pod="kuadrant-system/authorino-7b89456cc-lj4m5"
Sep 08 15:55:47.184167 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:47.184142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b89456cc-lj4m5"
Sep 08 15:55:47.454659 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:47.454590    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b89456cc-lj4m5"]
Sep 08 15:55:47.457701 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:55:47.457664    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9293816e_2555_4268_9e94_d87394144cd3.slice/crio-08f5a3d7329e9248655b77fff50e12f92025b4d3efb5ea3fc6be58c8e5598930 WatchSource:0}: Error finding container 08f5a3d7329e9248655b77fff50e12f92025b4d3efb5ea3fc6be58c8e5598930: Status 404 returned error can't find the container with id 08f5a3d7329e9248655b77fff50e12f92025b4d3efb5ea3fc6be58c8e5598930
Sep 08 15:55:48.130827 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:48.130789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b89456cc-lj4m5" event={"ID":"9293816e-2555-4268-9e94-d87394144cd3","Type":"ContainerStarted","Data":"08f5a3d7329e9248655b77fff50e12f92025b4d3efb5ea3fc6be58c8e5598930"}
Sep 08 15:55:50.146801 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:50.146753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b89456cc-lj4m5" event={"ID":"9293816e-2555-4268-9e94-d87394144cd3","Type":"ContainerStarted","Data":"47b72ac896456cce62223537ab31de665b715fefe599b7d21d067e88e3dba438"}
Sep 08 15:55:50.192873 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:50.192814    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b89456cc-lj4m5" podStartSLOduration=2.001268255 podStartE2EDuration="4.19279343s" totalImagesPullingTime="2.191525175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:55:50.19141607 +0000 UTC m=+696.815108391" watchObservedRunningTime="2026-09-08 15:55:50.19279343 +0000 UTC m=+696.816485745"
Sep 08 15:55:50.265739 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:50.265693    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcb58cd9f-6bx5d"]
Sep 08 15:55:50.265956 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:50.265924    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d" podUID="932644db-ca94-4804-ac21-f1981aa21e23" containerName="authorino" containerID="cri-o://b7de8cfa4ac535af3c02e735e0285b6dddb7bac9a48886017965bdac1a46aaf6" gracePeriod=30
Sep 08 15:55:50.521683 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:50.521659    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d"
Sep 08 15:55:50.538937 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:50.538902    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8698\" (UniqueName: \"kubernetes.io/projected/932644db-ca94-4804-ac21-f1981aa21e23-kube-api-access-b8698\") pod \"932644db-ca94-4804-ac21-f1981aa21e23\" (UID: \"932644db-ca94-4804-ac21-f1981aa21e23\") "
Sep 08 15:55:50.539098 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:50.539009    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/932644db-ca94-4804-ac21-f1981aa21e23-tls-cert\") pod \"932644db-ca94-4804-ac21-f1981aa21e23\" (UID: \"932644db-ca94-4804-ac21-f1981aa21e23\") "
Sep 08 15:55:50.541631 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:50.541596    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932644db-ca94-4804-ac21-f1981aa21e23-kube-api-access-b8698" (OuterVolumeSpecName: "kube-api-access-b8698") pod "932644db-ca94-4804-ac21-f1981aa21e23" (UID: "932644db-ca94-4804-ac21-f1981aa21e23"). InnerVolumeSpecName "kube-api-access-b8698". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:55:50.552141 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:50.552110    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/932644db-ca94-4804-ac21-f1981aa21e23-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "932644db-ca94-4804-ac21-f1981aa21e23" (UID: "932644db-ca94-4804-ac21-f1981aa21e23"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:55:50.642414 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:50.641855    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8698\" (UniqueName: \"kubernetes.io/projected/932644db-ca94-4804-ac21-f1981aa21e23-kube-api-access-b8698\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:55:50.642414 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:50.641894    2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/932644db-ca94-4804-ac21-f1981aa21e23-tls-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:55:51.157855 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.157811    2581 generic.go:356] "Generic (PLEG): container finished" podID="932644db-ca94-4804-ac21-f1981aa21e23" containerID="b7de8cfa4ac535af3c02e735e0285b6dddb7bac9a48886017965bdac1a46aaf6" exitCode=0
Sep 08 15:55:51.158785 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.158747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d" event={"ID":"932644db-ca94-4804-ac21-f1981aa21e23","Type":"ContainerDied","Data":"b7de8cfa4ac535af3c02e735e0285b6dddb7bac9a48886017965bdac1a46aaf6"}
Sep 08 15:55:51.158785 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.158784    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d"
Sep 08 15:55:51.158977 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.158800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcb58cd9f-6bx5d" event={"ID":"932644db-ca94-4804-ac21-f1981aa21e23","Type":"ContainerDied","Data":"14193d0da2543badf14d638b3a70160c75b0001609e00ad31d6c16961d39dac8"}
Sep 08 15:55:51.158977 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.158837    2581 scope.go:117] "RemoveContainer" containerID="b7de8cfa4ac535af3c02e735e0285b6dddb7bac9a48886017965bdac1a46aaf6"
Sep 08 15:55:51.162837 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.162812    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7959-140182330835888"]
Sep 08 15:55:51.163230 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.163210    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932644db-ca94-4804-ac21-f1981aa21e23" containerName="authorino"
Sep 08 15:55:51.163311 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.163233    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="932644db-ca94-4804-ac21-f1981aa21e23" containerName="authorino"
Sep 08 15:55:51.163370 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.163345    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="932644db-ca94-4804-ac21-f1981aa21e23" containerName="authorino"
Sep 08 15:55:51.166620 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.166598    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7959-140182330835888"
Sep 08 15:55:51.169627 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.169589    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5nm68\""
Sep 08 15:55:51.170358 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.170334    2581 scope.go:117] "RemoveContainer" containerID="b7de8cfa4ac535af3c02e735e0285b6dddb7bac9a48886017965bdac1a46aaf6"
Sep 08 15:55:51.170969 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:55:51.170944    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7de8cfa4ac535af3c02e735e0285b6dddb7bac9a48886017965bdac1a46aaf6\": container with ID starting with b7de8cfa4ac535af3c02e735e0285b6dddb7bac9a48886017965bdac1a46aaf6 not found: ID does not exist" containerID="b7de8cfa4ac535af3c02e735e0285b6dddb7bac9a48886017965bdac1a46aaf6"
Sep 08 15:55:51.171073 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.170978    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7de8cfa4ac535af3c02e735e0285b6dddb7bac9a48886017965bdac1a46aaf6"} err="failed to get container status \"b7de8cfa4ac535af3c02e735e0285b6dddb7bac9a48886017965bdac1a46aaf6\": rpc error: code = NotFound desc = could not find container \"b7de8cfa4ac535af3c02e735e0285b6dddb7bac9a48886017965bdac1a46aaf6\": container with ID starting with b7de8cfa4ac535af3c02e735e0285b6dddb7bac9a48886017965bdac1a46aaf6 not found: ID does not exist"
Sep 08 15:55:51.171489 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.171468    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7959-140182330835888"]
Sep 08 15:55:51.192616 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.192584    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcb58cd9f-6bx5d"]
Sep 08 15:55:51.198499 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.198475    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcb58cd9f-6bx5d"]
Sep 08 15:55:51.247733 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.247703    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4z2\" (UniqueName: \"kubernetes.io/projected/d07daeb1-538f-43d6-b886-e3151374251a-kube-api-access-tz4z2\") pod \"test-curl-7959-140182330835888\" (UID: \"d07daeb1-538f-43d6-b886-e3151374251a\") " pod="opendatahub/test-curl-7959-140182330835888"
Sep 08 15:55:51.348454 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.348407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4z2\" (UniqueName: \"kubernetes.io/projected/d07daeb1-538f-43d6-b886-e3151374251a-kube-api-access-tz4z2\") pod \"test-curl-7959-140182330835888\" (UID: \"d07daeb1-538f-43d6-b886-e3151374251a\") " pod="opendatahub/test-curl-7959-140182330835888"
Sep 08 15:55:51.356906 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.356870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4z2\" (UniqueName: \"kubernetes.io/projected/d07daeb1-538f-43d6-b886-e3151374251a-kube-api-access-tz4z2\") pod \"test-curl-7959-140182330835888\" (UID: \"d07daeb1-538f-43d6-b886-e3151374251a\") " pod="opendatahub/test-curl-7959-140182330835888"
Sep 08 15:55:51.484338 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.484261    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7959-140182330835888"
Sep 08 15:55:51.755543 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.755518    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7959-140182330835888"]
Sep 08 15:55:51.757115 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:55:51.757086    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd07daeb1_538f_43d6_b886_e3151374251a.slice/crio-c45b819578530681a58d4582d74f2b8b5ae9c5530c2d7e67b8c4ff444199400d WatchSource:0}: Error finding container c45b819578530681a58d4582d74f2b8b5ae9c5530c2d7e67b8c4ff444199400d: Status 404 returned error can't find the container with id c45b819578530681a58d4582d74f2b8b5ae9c5530c2d7e67b8c4ff444199400d
Sep 08 15:55:51.900775 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:51.900744    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="932644db-ca94-4804-ac21-f1981aa21e23" path="/var/lib/kubelet/pods/932644db-ca94-4804-ac21-f1981aa21e23/volumes"
Sep 08 15:55:52.165861 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:52.165829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7959-140182330835888" event={"ID":"d07daeb1-538f-43d6-b886-e3151374251a","Type":"ContainerStarted","Data":"c45b819578530681a58d4582d74f2b8b5ae9c5530c2d7e67b8c4ff444199400d"}
Sep 08 15:55:53.172176 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:53.172140    2581 generic.go:356] "Generic (PLEG): container finished" podID="d07daeb1-538f-43d6-b886-e3151374251a" containerID="f040af2ea3e41123133d5031b6e47043acb9d501559a74e2a3fcac471fbee8f0" exitCode=0
Sep 08 15:55:53.172552 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:53.172225    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7959-140182330835888" event={"ID":"d07daeb1-538f-43d6-b886-e3151374251a","Type":"ContainerDied","Data":"f040af2ea3e41123133d5031b6e47043acb9d501559a74e2a3fcac471fbee8f0"}
Sep 08 15:55:54.315385 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.315361    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7959-140182330835888"
Sep 08 15:55:54.353721 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.353688    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7959-140182330835888_d07daeb1-538f-43d6-b886-e3151374251a/test-curl-7959-140182330835888/0.log"
Sep 08 15:55:54.375202 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.375176    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz4z2\" (UniqueName: \"kubernetes.io/projected/d07daeb1-538f-43d6-b886-e3151374251a-kube-api-access-tz4z2\") pod \"d07daeb1-538f-43d6-b886-e3151374251a\" (UID: \"d07daeb1-538f-43d6-b886-e3151374251a\") "
Sep 08 15:55:54.377418 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.377378    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07daeb1-538f-43d6-b886-e3151374251a-kube-api-access-tz4z2" (OuterVolumeSpecName: "kube-api-access-tz4z2") pod "d07daeb1-538f-43d6-b886-e3151374251a" (UID: "d07daeb1-538f-43d6-b886-e3151374251a"). InnerVolumeSpecName "kube-api-access-tz4z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:55:54.388128 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.388102    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7959-140182330835888"]
Sep 08 15:55:54.391361 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.391342    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7959-140182330835888"]
Sep 08 15:55:54.475884 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.475832    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz4z2\" (UniqueName: \"kubernetes.io/projected/d07daeb1-538f-43d6-b886-e3151374251a-kube-api-access-tz4z2\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:55:54.932487 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.932451    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"]
Sep 08 15:55:54.932812 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.932798    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d07daeb1-538f-43d6-b886-e3151374251a" containerName="test-curl-7959-140182330835888"
Sep 08 15:55:54.932812 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.932812    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d07daeb1-538f-43d6-b886-e3151374251a" containerName="test-curl-7959-140182330835888"
Sep 08 15:55:54.932917 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.932877    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d07daeb1-538f-43d6-b886-e3151374251a" containerName="test-curl-7959-140182330835888"
Sep 08 15:55:54.935373 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.935357    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:54.938533 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.938512    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 15:55:54.938633 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.938517    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cb6c353e-openshift-default-dockercfg-8lhg9\""
Sep 08 15:55:54.955540 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.955520    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"]
Sep 08 15:55:54.979139 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.979118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-workload-socket\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:54.979256 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.979145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-istio-data\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:54.979256 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.979167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e161569-3090-47f5-935c-798e9888494c-istio-token\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:54.979377 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.979288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e161569-3090-47f5-935c-798e9888494c-istiod-ca-cert\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:54.979377 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.979367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-istio-envoy\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:54.979497 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.979427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85c24\" (UniqueName: \"kubernetes.io/projected/9e161569-3090-47f5-935c-798e9888494c-kube-api-access-85c24\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:54.979551 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.979533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-workload-certs\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:54.979605 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.979576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-credential-socket\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:54.979661 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:54.979632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e161569-3090-47f5-935c-798e9888494c-istio-podinfo\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.080383 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.080355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-credential-socket\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.080383 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.080398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e161569-3090-47f5-935c-798e9888494c-istio-podinfo\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.080576 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.080428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-workload-socket\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.080576 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.080443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-istio-data\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.080576 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.080467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e161569-3090-47f5-935c-798e9888494c-istio-token\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.080576 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.080495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e161569-3090-47f5-935c-798e9888494c-istiod-ca-cert\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.080576 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.080537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-istio-envoy\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.080576 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.080564    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85c24\" (UniqueName: \"kubernetes.io/projected/9e161569-3090-47f5-935c-798e9888494c-kube-api-access-85c24\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.080853 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.080614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-workload-certs\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.080853 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.080753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-credential-socket\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.080853 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.080797    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-workload-socket\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.080853 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.080828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-istio-data\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.081061 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.080913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-workload-certs\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.081374 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.081346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e161569-3090-47f5-935c-798e9888494c-istiod-ca-cert\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.083072 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.083052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-istio-envoy\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.083342 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.083324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e161569-3090-47f5-935c-798e9888494c-istio-podinfo\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.088904 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.088873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e161569-3090-47f5-935c-798e9888494c-istio-token\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.089522 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.089500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85c24\" (UniqueName: \"kubernetes.io/projected/9e161569-3090-47f5-935c-798e9888494c-kube-api-access-85c24\") pod \"e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") " pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.184471 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.184378    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7959-140182330835888"
Sep 08 15:55:55.184471 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.184443    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c45b819578530681a58d4582d74f2b8b5ae9c5530c2d7e67b8c4ff444199400d"
Sep 08 15:55:55.194738 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:55:55.194715    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7959-140182330835888\" is forbidden: User \"system:node:ip-10-0-134-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-107.ec2.internal' and this object" podUID="d07daeb1-538f-43d6-b886-e3151374251a" pod="opendatahub/test-curl-7959-140182330835888"
Sep 08 15:55:55.245208 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.245180    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:55.535579 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.535455    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"]
Sep 08 15:55:55.537974 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:55:55.537949    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e161569_3090_47f5_935c_798e9888494c.slice/crio-47d177e1b8de40f4912b2d68023ab2de3d2aef47d55ace11c6edd4da05ab02d7 WatchSource:0}: Error finding container 47d177e1b8de40f4912b2d68023ab2de3d2aef47d55ace11c6edd4da05ab02d7: Status 404 returned error can't find the container with id 47d177e1b8de40f4912b2d68023ab2de3d2aef47d55ace11c6edd4da05ab02d7
Sep 08 15:55:55.901926 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:55.901894    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d07daeb1-538f-43d6-b886-e3151374251a" path="/var/lib/kubelet/pods/d07daeb1-538f-43d6-b886-e3151374251a/volumes"
Sep 08 15:55:56.191663 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.191575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" event={"ID":"9e161569-3090-47f5-935c-798e9888494c","Type":"ContainerStarted","Data":"47d177e1b8de40f4912b2d68023ab2de3d2aef47d55ace11c6edd4da05ab02d7"}
Sep 08 15:55:56.551673 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.551596    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"]
Sep 08 15:55:56.555483 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.555457    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"
Sep 08 15:55:56.558547 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.558516    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ef1190\""
Sep 08 15:55:56.558675 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.558553    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ef1190-dockercfg-glgx7\""
Sep 08 15:55:56.569954 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.569929    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"]
Sep 08 15:55:56.572238 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.572209    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"]
Sep 08 15:55:56.576053 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.576030    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"
Sep 08 15:55:56.583752 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.583729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"]
Sep 08 15:55:56.595684 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.595659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsb2z\" (UniqueName: \"kubernetes.io/projected/31151fb8-a8eb-4ef7-9da9-f268a1dd4aad-kube-api-access-lsb2z\") pod \"payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j\" (UID: \"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"
Sep 08 15:55:56.595791 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.595699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3568e2a-0e61-4004-a2bb-1686463cf0cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9\" (UID: \"a3568e2a-0e61-4004-a2bb-1686463cf0cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"
Sep 08 15:55:56.595791 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.595783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6tw8\" (UniqueName: \"kubernetes.io/projected/a3568e2a-0e61-4004-a2bb-1686463cf0cd-kube-api-access-h6tw8\") pod \"payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9\" (UID: \"a3568e2a-0e61-4004-a2bb-1686463cf0cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"
Sep 08 15:55:56.595909 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.595855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31151fb8-a8eb-4ef7-9da9-f268a1dd4aad-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j\" (UID: \"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"
Sep 08 15:55:56.697283 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.697206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tw8\" (UniqueName: \"kubernetes.io/projected/a3568e2a-0e61-4004-a2bb-1686463cf0cd-kube-api-access-h6tw8\") pod \"payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9\" (UID: \"a3568e2a-0e61-4004-a2bb-1686463cf0cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"
Sep 08 15:55:56.697480 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.697299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31151fb8-a8eb-4ef7-9da9-f268a1dd4aad-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j\" (UID: \"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"
Sep 08 15:55:56.697480 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.697368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsb2z\" (UniqueName: \"kubernetes.io/projected/31151fb8-a8eb-4ef7-9da9-f268a1dd4aad-kube-api-access-lsb2z\") pod \"payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j\" (UID: \"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"
Sep 08 15:55:56.697480 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.697409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3568e2a-0e61-4004-a2bb-1686463cf0cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9\" (UID: \"a3568e2a-0e61-4004-a2bb-1686463cf0cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"
Sep 08 15:55:56.698175 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.698152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31151fb8-a8eb-4ef7-9da9-f268a1dd4aad-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j\" (UID: \"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"
Sep 08 15:55:56.698253 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.698211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3568e2a-0e61-4004-a2bb-1686463cf0cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9\" (UID: \"a3568e2a-0e61-4004-a2bb-1686463cf0cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"
Sep 08 15:55:56.705488 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.705460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6tw8\" (UniqueName: \"kubernetes.io/projected/a3568e2a-0e61-4004-a2bb-1686463cf0cd-kube-api-access-h6tw8\") pod \"payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9\" (UID: \"a3568e2a-0e61-4004-a2bb-1686463cf0cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"
Sep 08 15:55:56.705595 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.705546    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsb2z\" (UniqueName: \"kubernetes.io/projected/31151fb8-a8eb-4ef7-9da9-f268a1dd4aad-kube-api-access-lsb2z\") pod \"payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j\" (UID: \"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"
Sep 08 15:55:56.869540 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.869468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"
Sep 08 15:55:56.889149 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:56.889115    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"
Sep 08 15:55:57.858486 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:57.857010    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:55:57.858486 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:57.857268    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj" podUID="6b4d0763-376e-46c3-8600-792f83a05a50" containerName="limitador" containerID="cri-o://4f289f9d178e0f2cb343d3283385e810c623a177c5a672288b9d343586380766" gracePeriod=30
Sep 08 15:55:57.883431 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:57.883373    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:55:57.883509 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:57.883463    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:55:57.883509 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:57.883498    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:55:58.076577 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.076541    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"]
Sep 08 15:55:58.080366 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.080339    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"]
Sep 08 15:55:58.085022 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:55:58.084995    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3568e2a_0e61_4004_a2bb_1686463cf0cd.slice/crio-5ac67ece6877921ea39564e413deb2bcd2ebcbe3a8a18cb4219ed2ed468d39f7 WatchSource:0}: Error finding container 5ac67ece6877921ea39564e413deb2bcd2ebcbe3a8a18cb4219ed2ed468d39f7: Status 404 returned error can't find the container with id 5ac67ece6877921ea39564e413deb2bcd2ebcbe3a8a18cb4219ed2ed468d39f7
Sep 08 15:55:58.209198 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.209161    2581 generic.go:356] "Generic (PLEG): container finished" podID="6b4d0763-376e-46c3-8600-792f83a05a50" containerID="4f289f9d178e0f2cb343d3283385e810c623a177c5a672288b9d343586380766" exitCode=0
Sep 08 15:55:58.209348 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.209262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj" event={"ID":"6b4d0763-376e-46c3-8600-792f83a05a50","Type":"ContainerDied","Data":"4f289f9d178e0f2cb343d3283385e810c623a177c5a672288b9d343586380766"}
Sep 08 15:55:58.211274 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.211246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" event={"ID":"9e161569-3090-47f5-935c-798e9888494c","Type":"ContainerStarted","Data":"12c7521909ec0bf938af7f670a830214a9083be6170b0b7b3fd97ebdeac14639"}
Sep 08 15:55:58.216580 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.216549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j" event={"ID":"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad","Type":"ContainerStarted","Data":"c156ce399056ec098b3fd0e4032b46eb0d4586960b6ccca657ead7945a9a2571"}
Sep 08 15:55:58.216708 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.216611    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j" event={"ID":"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad","Type":"ContainerStarted","Data":"e572db8c878bb4bdfa112634983de805c8c0321aa05665b164269e6545acc37a"}
Sep 08 15:55:58.216929 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.216868    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"
Sep 08 15:55:58.218812 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.218780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9" event={"ID":"a3568e2a-0e61-4004-a2bb-1686463cf0cd","Type":"ContainerStarted","Data":"8d57ab41e668588bb604c732cc99486be5f8de813dbc9bfe2e3cf6c0604a0366"}
Sep 08 15:55:58.218927 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.218821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9" event={"ID":"a3568e2a-0e61-4004-a2bb-1686463cf0cd","Type":"ContainerStarted","Data":"5ac67ece6877921ea39564e413deb2bcd2ebcbe3a8a18cb4219ed2ed468d39f7"}
Sep 08 15:55:58.218927 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.218858    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"
Sep 08 15:55:58.230340 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.230167    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" podStartSLOduration=1.8875593849999999 podStartE2EDuration="4.230149955s" totalImagesPullingTime="2.34259057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:55:58.229745122 +0000 UTC m=+704.853437447" watchObservedRunningTime="2026-09-08 15:55:58.230149955 +0000 UTC m=+704.853842279"
Sep 08 15:55:58.245499 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.245435    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:58.246461 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.246426    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9" podStartSLOduration=2.2464124 podStartE2EDuration="2.2464124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:55:58.244931699 +0000 UTC m=+704.868624053" watchObservedRunningTime="2026-09-08 15:55:58.2464124 +0000 UTC m=+704.870104721"
Sep 08 15:55:58.246584 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.246563    2581 patch_prober.go:28] interesting pod/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 08 15:55:58.246644 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.246608    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 08 15:55:58.262189 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.262131    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j" podStartSLOduration=2.2621127420000002 podStartE2EDuration="2.262112742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:55:58.262093068 +0000 UTC m=+704.885785391" watchObservedRunningTime="2026-09-08 15:55:58.262112742 +0000 UTC m=+704.885805067"
Sep 08 15:55:58.328246 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.328184    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"
Sep 08 15:55:58.414983 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.414951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b4d0763-376e-46c3-8600-792f83a05a50-config-file\") pod \"6b4d0763-376e-46c3-8600-792f83a05a50\" (UID: \"6b4d0763-376e-46c3-8600-792f83a05a50\") "
Sep 08 15:55:58.415123 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.415060    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g74cf\" (UniqueName: \"kubernetes.io/projected/6b4d0763-376e-46c3-8600-792f83a05a50-kube-api-access-g74cf\") pod \"6b4d0763-376e-46c3-8600-792f83a05a50\" (UID: \"6b4d0763-376e-46c3-8600-792f83a05a50\") "
Sep 08 15:55:58.415326 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.415300    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4d0763-376e-46c3-8600-792f83a05a50-config-file" (OuterVolumeSpecName: "config-file") pod "6b4d0763-376e-46c3-8600-792f83a05a50" (UID: "6b4d0763-376e-46c3-8600-792f83a05a50"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:55:58.417296 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.417273    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4d0763-376e-46c3-8600-792f83a05a50-kube-api-access-g74cf" (OuterVolumeSpecName: "kube-api-access-g74cf") pod "6b4d0763-376e-46c3-8600-792f83a05a50" (UID: "6b4d0763-376e-46c3-8600-792f83a05a50"). InnerVolumeSpecName "kube-api-access-g74cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:55:58.516208 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.516165    2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b4d0763-376e-46c3-8600-792f83a05a50-config-file\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:55:58.516208 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:58.516206    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g74cf\" (UniqueName: \"kubernetes.io/projected/6b4d0763-376e-46c3-8600-792f83a05a50-kube-api-access-g74cf\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:55:59.224992 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:59.224962    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"
Sep 08 15:55:59.225479 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:59.225019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj" event={"ID":"6b4d0763-376e-46c3-8600-792f83a05a50","Type":"ContainerDied","Data":"720dee4d0bfe9fb5e4e3b22b32324da00332fe78b775870ca0fca5370af07a69"}
Sep 08 15:55:59.225479 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:59.225071    2581 scope.go:117] "RemoveContainer" containerID="4f289f9d178e0f2cb343d3283385e810c623a177c5a672288b9d343586380766"
Sep 08 15:55:59.249824 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:59.249797    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:55:59.250251 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:59.250236    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:55:59.252749 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:59.252718    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-lrcmj"]
Sep 08 15:55:59.903378 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:55:59.903335    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b4d0763-376e-46c3-8600-792f83a05a50" path="/var/lib/kubelet/pods/6b4d0763-376e-46c3-8600-792f83a05a50/volumes"
Sep 08 15:56:00.231122 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.231050    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:56:00.231963 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.231945    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:56:00.611258 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.611176    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"]
Sep 08 15:56:00.611710 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.611692    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b4d0763-376e-46c3-8600-792f83a05a50" containerName="limitador"
Sep 08 15:56:00.611756 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.611713    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b4d0763-376e-46c3-8600-792f83a05a50" containerName="limitador"
Sep 08 15:56:00.611793 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.611787    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b4d0763-376e-46c3-8600-792f83a05a50" containerName="limitador"
Sep 08 15:56:00.616164 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.616147    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"
Sep 08 15:56:00.618591 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.618536    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-423c90-dockercfg-7hfs9\""
Sep 08 15:56:00.618591 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.618554    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-423c90\""
Sep 08 15:56:00.624893 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.624871    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"]
Sep 08 15:56:00.641645 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.641625    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"]
Sep 08 15:56:00.644934 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.644920    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"
Sep 08 15:56:00.654246 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.654226    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"]
Sep 08 15:56:00.736549 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.736519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc5266c-d85c-4b1a-9351-61f82c428fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2\" (UID: \"1dc5266c-d85c-4b1a-9351-61f82c428fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"
Sep 08 15:56:00.736664 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.736567    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krl8v\" (UniqueName: \"kubernetes.io/projected/1dc5266c-d85c-4b1a-9351-61f82c428fc5-kube-api-access-krl8v\") pod \"payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2\" (UID: \"1dc5266c-d85c-4b1a-9351-61f82c428fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"
Sep 08 15:56:00.736664 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.736638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87aced9a-cfcd-48b7-9d40-f3eaa33fecfd-plugins-config-volume\") pod \"payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j\" (UID: \"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd\") " pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"
Sep 08 15:56:00.736744 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.736668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9bzt\" (UniqueName: \"kubernetes.io/projected/87aced9a-cfcd-48b7-9d40-f3eaa33fecfd-kube-api-access-p9bzt\") pod \"payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j\" (UID: \"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd\") " pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"
Sep 08 15:56:00.837676 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.837652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87aced9a-cfcd-48b7-9d40-f3eaa33fecfd-plugins-config-volume\") pod \"payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j\" (UID: \"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd\") " pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"
Sep 08 15:56:00.837764 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.837681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9bzt\" (UniqueName: \"kubernetes.io/projected/87aced9a-cfcd-48b7-9d40-f3eaa33fecfd-kube-api-access-p9bzt\") pod \"payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j\" (UID: \"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd\") " pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"
Sep 08 15:56:00.837764 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.837715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc5266c-d85c-4b1a-9351-61f82c428fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2\" (UID: \"1dc5266c-d85c-4b1a-9351-61f82c428fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"
Sep 08 15:56:00.837843 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.837773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krl8v\" (UniqueName: \"kubernetes.io/projected/1dc5266c-d85c-4b1a-9351-61f82c428fc5-kube-api-access-krl8v\") pod \"payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2\" (UID: \"1dc5266c-d85c-4b1a-9351-61f82c428fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"
Sep 08 15:56:00.838365 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.838348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc5266c-d85c-4b1a-9351-61f82c428fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2\" (UID: \"1dc5266c-d85c-4b1a-9351-61f82c428fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"
Sep 08 15:56:00.838478 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.838461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87aced9a-cfcd-48b7-9d40-f3eaa33fecfd-plugins-config-volume\") pod \"payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j\" (UID: \"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd\") " pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"
Sep 08 15:56:00.852979 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.852956    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9bzt\" (UniqueName: \"kubernetes.io/projected/87aced9a-cfcd-48b7-9d40-f3eaa33fecfd-kube-api-access-p9bzt\") pod \"payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j\" (UID: \"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd\") " pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"
Sep 08 15:56:00.855808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.855784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krl8v\" (UniqueName: \"kubernetes.io/projected/1dc5266c-d85c-4b1a-9351-61f82c428fc5-kube-api-access-krl8v\") pod \"payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2\" (UID: \"1dc5266c-d85c-4b1a-9351-61f82c428fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"
Sep 08 15:56:00.926454 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.926428    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"
Sep 08 15:56:00.952870 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.952841    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"]
Sep 08 15:56:00.954926 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.954900    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"
Sep 08 15:56:00.957766 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.957747    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:00.961334 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.961313    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:56:00.961587 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.961569    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmx7t\""
Sep 08 15:56:00.961982 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.961955    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:56:00.962171 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.962152    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cb6c353e-serving-cert\""
Sep 08 15:56:00.962478 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.962460    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:56:00.974829 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:00.974766    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"]
Sep 08 15:56:01.040547 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.040386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxcx6\" (UniqueName: \"kubernetes.io/projected/af624738-f323-490d-a925-bca1cc7f5e69-kube-api-access-cxcx6\") pod \"maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.040622 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.040550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af624738-f323-490d-a925-bca1cc7f5e69-openshift-service-ca\") pod \"maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.040622 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.040585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-metrics-certs\") pod \"maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.040694 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.040628    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-maas-api-tls\") pod \"maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.141515 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.141482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxcx6\" (UniqueName: \"kubernetes.io/projected/af624738-f323-490d-a925-bca1cc7f5e69-kube-api-access-cxcx6\") pod \"maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.141682 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.141595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af624738-f323-490d-a925-bca1cc7f5e69-openshift-service-ca\") pod \"maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.141682 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.141625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-metrics-certs\") pod \"maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.141682 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.141659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-maas-api-tls\") pod \"maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.142000 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:56:01.141812    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-serving-cert: secret "maas-api-e2e-ait-cb6c353e-serving-cert" not found
Sep 08 15:56:01.142000 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:56:01.141900    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-maas-api-tls podName:af624738-f323-490d-a925-bca1cc7f5e69 nodeName:}" failed. No retries permitted until 2026-09-08 15:56:01.641876163 +0000 UTC m=+708.265568481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-maas-api-tls") pod "maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq" (UID: "af624738-f323-490d-a925-bca1cc7f5e69") : secret "maas-api-e2e-ait-cb6c353e-serving-cert" not found
Sep 08 15:56:01.142220 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.142200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af624738-f323-490d-a925-bca1cc7f5e69-openshift-service-ca\") pod \"maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.144014 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.143995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-metrics-certs\") pod \"maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.149540 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.149519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxcx6\" (UniqueName: \"kubernetes.io/projected/af624738-f323-490d-a925-bca1cc7f5e69-kube-api-access-cxcx6\") pod \"maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.227558 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.227534    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"]
Sep 08 15:56:01.229265 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:56:01.229242    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc5266c_d85c_4b1a_9351_61f82c428fc5.slice/crio-76c755b7d7c3548d0c59d99983e2055023927647390a169debcc6efb637e690d WatchSource:0}: Error finding container 76c755b7d7c3548d0c59d99983e2055023927647390a169debcc6efb637e690d: Status 404 returned error can't find the container with id 76c755b7d7c3548d0c59d99983e2055023927647390a169debcc6efb637e690d
Sep 08 15:56:01.236813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.236788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2" event={"ID":"1dc5266c-d85c-4b1a-9351-61f82c428fc5","Type":"ContainerStarted","Data":"76c755b7d7c3548d0c59d99983e2055023927647390a169debcc6efb637e690d"}
Sep 08 15:56:01.255800 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.255779    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"]
Sep 08 15:56:01.260367 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:56:01.260344    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87aced9a_cfcd_48b7_9d40_f3eaa33fecfd.slice/crio-e3ec666ffb4a37174f7842e141e86a7ba837977ba38a454337221c967f125930 WatchSource:0}: Error finding container e3ec666ffb4a37174f7842e141e86a7ba837977ba38a454337221c967f125930: Status 404 returned error can't find the container with id e3ec666ffb4a37174f7842e141e86a7ba837977ba38a454337221c967f125930
Sep 08 15:56:01.595359 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.595266    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"]
Sep 08 15:56:01.599321 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.599298    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"
Sep 08 15:56:01.601769 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.601747    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cb6c353e\""
Sep 08 15:56:01.608176 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.608147    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"]
Sep 08 15:56:01.623631 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.623605    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"]
Sep 08 15:56:01.626896 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.626879    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"
Sep 08 15:56:01.628996 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.628976    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cb6c353e-dockercfg-gh9sr\""
Sep 08 15:56:01.636554 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.636533    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"]
Sep 08 15:56:01.645739 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.645712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-maas-api-tls\") pod \"maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.645827 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.645784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7e9c2d-1746-4356-ac25-024e9492a13f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw\" (UID: \"ad7e9c2d-1746-4356-ac25-024e9492a13f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"
Sep 08 15:56:01.645827 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.645817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lds24\" (UniqueName: \"kubernetes.io/projected/ad7e9c2d-1746-4356-ac25-024e9492a13f-kube-api-access-lds24\") pod \"payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw\" (UID: \"ad7e9c2d-1746-4356-ac25-024e9492a13f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"
Sep 08 15:56:01.648303 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.648263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-maas-api-tls\") pod \"maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.746545 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.746490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7e9c2d-1746-4356-ac25-024e9492a13f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw\" (UID: \"ad7e9c2d-1746-4356-ac25-024e9492a13f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"
Sep 08 15:56:01.746719 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.746555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lds24\" (UniqueName: \"kubernetes.io/projected/ad7e9c2d-1746-4356-ac25-024e9492a13f-kube-api-access-lds24\") pod \"payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw\" (UID: \"ad7e9c2d-1746-4356-ac25-024e9492a13f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"
Sep 08 15:56:01.746719 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.746598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12613886-416a-4719-8bb3-6df373e00775-plugins-config-volume\") pod \"payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm\" (UID: \"12613886-416a-4719-8bb3-6df373e00775\") " pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"
Sep 08 15:56:01.746719 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.746666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmfgz\" (UniqueName: \"kubernetes.io/projected/12613886-416a-4719-8bb3-6df373e00775-kube-api-access-vmfgz\") pod \"payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm\" (UID: \"12613886-416a-4719-8bb3-6df373e00775\") " pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"
Sep 08 15:56:01.747188 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.747165    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7e9c2d-1746-4356-ac25-024e9492a13f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw\" (UID: \"ad7e9c2d-1746-4356-ac25-024e9492a13f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"
Sep 08 15:56:01.754016 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.753995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lds24\" (UniqueName: \"kubernetes.io/projected/ad7e9c2d-1746-4356-ac25-024e9492a13f-kube-api-access-lds24\") pod \"payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw\" (UID: \"ad7e9c2d-1746-4356-ac25-024e9492a13f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"
Sep 08 15:56:01.847491 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.847416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12613886-416a-4719-8bb3-6df373e00775-plugins-config-volume\") pod \"payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm\" (UID: \"12613886-416a-4719-8bb3-6df373e00775\") " pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"
Sep 08 15:56:01.847491 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.847470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmfgz\" (UniqueName: \"kubernetes.io/projected/12613886-416a-4719-8bb3-6df373e00775-kube-api-access-vmfgz\") pod \"payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm\" (UID: \"12613886-416a-4719-8bb3-6df373e00775\") " pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"
Sep 08 15:56:01.848120 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.848101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12613886-416a-4719-8bb3-6df373e00775-plugins-config-volume\") pod \"payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm\" (UID: \"12613886-416a-4719-8bb3-6df373e00775\") " pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"
Sep 08 15:56:01.855763 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.855741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmfgz\" (UniqueName: \"kubernetes.io/projected/12613886-416a-4719-8bb3-6df373e00775-kube-api-access-vmfgz\") pod \"payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm\" (UID: \"12613886-416a-4719-8bb3-6df373e00775\") " pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"
Sep 08 15:56:01.889596 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.889574    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:01.912989 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.912962    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"
Sep 08 15:56:01.937425 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:01.937348    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"
Sep 08 15:56:02.168646 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:02.168618    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"]
Sep 08 15:56:02.170126 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:56:02.170095    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf624738_f323_490d_a925_bca1cc7f5e69.slice/crio-c71e96ff2ef6f9be948d157aa53e8ddce61b54d8bdeca68364307e70502eedef WatchSource:0}: Error finding container c71e96ff2ef6f9be948d157aa53e8ddce61b54d8bdeca68364307e70502eedef: Status 404 returned error can't find the container with id c71e96ff2ef6f9be948d157aa53e8ddce61b54d8bdeca68364307e70502eedef
Sep 08 15:56:02.201733 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:02.201713    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"]
Sep 08 15:56:02.203665 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:56:02.203642    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7e9c2d_1746_4356_ac25_024e9492a13f.slice/crio-b9ce74db5319bfebadf8dc0f6c91b42187b0409cdde8ce9fe72139ba9c1e97bc WatchSource:0}: Error finding container b9ce74db5319bfebadf8dc0f6c91b42187b0409cdde8ce9fe72139ba9c1e97bc: Status 404 returned error can't find the container with id b9ce74db5319bfebadf8dc0f6c91b42187b0409cdde8ce9fe72139ba9c1e97bc
Sep 08 15:56:02.242639 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:02.242611    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"]
Sep 08 15:56:02.244882 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:02.244860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2" event={"ID":"1dc5266c-d85c-4b1a-9351-61f82c428fc5","Type":"ContainerStarted","Data":"d5b865f26f723ab6191b3fe739fdd425a524415b8e12fb7f61839ac66423d499"}
Sep 08 15:56:02.245014 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:02.244914    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"
Sep 08 15:56:02.246527 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:02.246504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j" event={"ID":"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd","Type":"ContainerStarted","Data":"5cd34d5136e1653f52bd1eaaed593275436688a533359c540205d563f92b48e1"}
Sep 08 15:56:02.246652 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:02.246553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j" event={"ID":"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd","Type":"ContainerStarted","Data":"e3ec666ffb4a37174f7842e141e86a7ba837977ba38a454337221c967f125930"}
Sep 08 15:56:02.246652 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:02.246584    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"
Sep 08 15:56:02.247843 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:02.247820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw" event={"ID":"ad7e9c2d-1746-4356-ac25-024e9492a13f","Type":"ContainerStarted","Data":"b9ce74db5319bfebadf8dc0f6c91b42187b0409cdde8ce9fe72139ba9c1e97bc"}
Sep 08 15:56:02.249024 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:02.249001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq" event={"ID":"af624738-f323-490d-a925-bca1cc7f5e69","Type":"ContainerStarted","Data":"c71e96ff2ef6f9be948d157aa53e8ddce61b54d8bdeca68364307e70502eedef"}
Sep 08 15:56:02.252482 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:56:02.252462    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12613886_416a_4719_8bb3_6df373e00775.slice/crio-234b3b8a22bbbf19ea546dbda6144c6cf0ec00142f23520ed1d7ebd5153d9495 WatchSource:0}: Error finding container 234b3b8a22bbbf19ea546dbda6144c6cf0ec00142f23520ed1d7ebd5153d9495: Status 404 returned error can't find the container with id 234b3b8a22bbbf19ea546dbda6144c6cf0ec00142f23520ed1d7ebd5153d9495
Sep 08 15:56:02.263717 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:02.263680    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2" podStartSLOduration=2.26366789 podStartE2EDuration="2.26366789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:56:02.263299727 +0000 UTC m=+708.886992052" watchObservedRunningTime="2026-09-08 15:56:02.26366789 +0000 UTC m=+708.887360229"
Sep 08 15:56:02.280443 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:02.280380    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j" podStartSLOduration=2.280362031 podStartE2EDuration="2.280362031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:56:02.278314103 +0000 UTC m=+708.902006432" watchObservedRunningTime="2026-09-08 15:56:02.280362031 +0000 UTC m=+708.904054355"
Sep 08 15:56:03.257367 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:03.257319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw" event={"ID":"ad7e9c2d-1746-4356-ac25-024e9492a13f","Type":"ContainerStarted","Data":"6d25b7f7bb3d5b8d97c325690256293a4aecf0fc48ab6706dcdb1f0c5a95d077"}
Sep 08 15:56:03.257864 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:03.257417    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"
Sep 08 15:56:03.259444 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:03.259362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm" event={"ID":"12613886-416a-4719-8bb3-6df373e00775","Type":"ContainerStarted","Data":"92bfb94b0120805973e1a22347f3bf1af3c63160601e3d41150f7c0854f4499f"}
Sep 08 15:56:03.259444 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:03.259438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm" event={"ID":"12613886-416a-4719-8bb3-6df373e00775","Type":"ContainerStarted","Data":"234b3b8a22bbbf19ea546dbda6144c6cf0ec00142f23520ed1d7ebd5153d9495"}
Sep 08 15:56:03.296445 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:03.296371    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw" podStartSLOduration=2.296349706 podStartE2EDuration="2.296349706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:56:03.276650446 +0000 UTC m=+709.900342771" watchObservedRunningTime="2026-09-08 15:56:03.296349706 +0000 UTC m=+709.920042028"
Sep 08 15:56:04.266249 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:04.266207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq" event={"ID":"af624738-f323-490d-a925-bca1cc7f5e69","Type":"ContainerStarted","Data":"bf12c5a2687d2a38a4aa25147c263877390f0cb68b2c752e640798fb7f3a1146"}
Sep 08 15:56:04.266646 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:04.266576    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:04.266646 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:04.266622    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"
Sep 08 15:56:04.295524 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:04.295473    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm" podStartSLOduration=3.295454865 podStartE2EDuration="3.295454865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:56:03.2975374 +0000 UTC m=+709.921229729" watchObservedRunningTime="2026-09-08 15:56:04.295454865 +0000 UTC m=+710.919147206"
Sep 08 15:56:04.298137 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:04.298097    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq" podStartSLOduration=2.656759273 podStartE2EDuration="4.298081538s" totalImagesPullingTime="1.641322265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:56:04.294719971 +0000 UTC m=+710.918412310" watchObservedRunningTime="2026-09-08 15:56:04.298081538 +0000 UTC m=+710.921773864"
Sep 08 15:56:09.226977 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:09.226948    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"
Sep 08 15:56:09.227763 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:09.227743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"
Sep 08 15:56:10.285515 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:10.284680    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:13.261132 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:13.261100    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"
Sep 08 15:56:13.262080 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:13.262059    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"
Sep 08 15:56:14.268536 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:14.268455    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"
Sep 08 15:56:15.274428 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:15.274374    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"
Sep 08 15:56:15.684143 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:15.684059    2581 patch_prober.go:28] interesting pod/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": read tcp 10.132.0.2:36996->10.132.0.46:15021: read: connection reset by peer" start-of-body=
Sep 08 15:56:15.684288 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:15.684164    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": read tcp 10.132.0.2:36996->10.132.0.46:15021: read: connection reset by peer"
Sep 08 15:56:16.344006 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:16.343970    2581 generic.go:356] "Generic (PLEG): container finished" podID="9e161569-3090-47f5-935c-798e9888494c" containerID="12c7521909ec0bf938af7f670a830214a9083be6170b0b7b3fd97ebdeac14639" exitCode=0
Sep 08 15:56:16.344425 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:16.344046    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" event={"ID":"9e161569-3090-47f5-935c-798e9888494c","Type":"ContainerDied","Data":"12c7521909ec0bf938af7f670a830214a9083be6170b0b7b3fd97ebdeac14639"}
Sep 08 15:56:16.344482 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:16.344460    2581 scope.go:117] "RemoveContainer" containerID="12c7521909ec0bf938af7f670a830214a9083be6170b0b7b3fd97ebdeac14639"
Sep 08 15:56:16.345502 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:16.345471    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:56:16.345602 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:16.345558    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:56:16.345602 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:16.345592    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:56:17.351703 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:17.351665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" event={"ID":"9e161569-3090-47f5-935c-798e9888494c","Type":"ContainerStarted","Data":"29560e71e4dd737ebed95bc62645c3f576766621026f6c316ec492f0a4516d3e"}
Sep 08 15:56:18.245568 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:18.245535    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:56:18.245821 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:18.245724    2581 patch_prober.go:28] interesting pod/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 08 15:56:18.245821 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:18.245782    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 08 15:56:19.246154 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:19.246119    2581 patch_prober.go:28] interesting pod/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 08 15:56:19.246553 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:19.246170    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 08 15:56:20.245960 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.245922    2581 patch_prober.go:28] interesting pod/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 08 15:56:20.246148 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.245974    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 08 15:56:20.329603 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.329570    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"]
Sep 08 15:56:20.336675 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.336652    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.339036 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.339015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c96e2b-openshift-default-dockercfg-4wz6m\""
Sep 08 15:56:20.342903 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.342874    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"]
Sep 08 15:56:20.425578 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.425543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-credential-socket\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.425734 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.425594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-token\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.425734 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.425638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-data\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.425734 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.425661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449bdd7a-7b16-469a-a5f4-5e45f834403e-istiod-ca-cert\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.425734 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.425704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-workload-socket\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.425734 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.425723    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-envoy\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.425921 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.425764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-podinfo\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.425921 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.425798    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvs2\" (UniqueName: \"kubernetes.io/projected/449bdd7a-7b16-469a-a5f4-5e45f834403e-kube-api-access-dcvs2\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.425921 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.425827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-workload-certs\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.526379 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.526293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-workload-certs\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.526379 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.526358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-credential-socket\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.526607 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.526417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-token\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.526607 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.526448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-data\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.526607 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.526476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449bdd7a-7b16-469a-a5f4-5e45f834403e-istiod-ca-cert\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.526766 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.526617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-workload-socket\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.526766 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.526667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-envoy\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.526766 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.526735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-podinfo\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.526766 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.526740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-workload-certs\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.526766 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.526761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvs2\" (UniqueName: \"kubernetes.io/projected/449bdd7a-7b16-469a-a5f4-5e45f834403e-kube-api-access-dcvs2\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.527020 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.526777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-credential-socket\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.527020 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.526806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-data\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.527148 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.527125    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-workload-socket\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.527218 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.527199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449bdd7a-7b16-469a-a5f4-5e45f834403e-istiod-ca-cert\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.529141 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.529116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-envoy\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.529435 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.529375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-podinfo\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.534318 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.534290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-token\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.534616 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.534596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvs2\" (UniqueName: \"kubernetes.io/projected/449bdd7a-7b16-469a-a5f4-5e45f834403e-kube-api-access-dcvs2\") pod \"e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") " pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.649664 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.649632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:20.960813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.960785    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"]
Sep 08 15:56:20.962613 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:56:20.962584    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449bdd7a_7b16_469a_a5f4_5e45f834403e.slice/crio-2b0d37cf30e08be109664771b0faf1af8dd8b24b903ed8c799e1236928f8aee0 WatchSource:0}: Error finding container 2b0d37cf30e08be109664771b0faf1af8dd8b24b903ed8c799e1236928f8aee0: Status 404 returned error can't find the container with id 2b0d37cf30e08be109664771b0faf1af8dd8b24b903ed8c799e1236928f8aee0
Sep 08 15:56:20.964980 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.964946    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:56:20.965083 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.965012    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:56:20.965083 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:20.965039    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:56:21.246365 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:21.246267    2581 patch_prober.go:28] interesting pod/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 08 15:56:21.246365 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:21.246331    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 08 15:56:21.377557 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:21.377508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" event={"ID":"449bdd7a-7b16-469a-a5f4-5e45f834403e","Type":"ContainerStarted","Data":"1ecb52f1746f9c7fa347801a086a0cfef810194efdff7861fb9bdee660582275"}
Sep 08 15:56:21.378026 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:21.377815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" event={"ID":"449bdd7a-7b16-469a-a5f4-5e45f834403e","Type":"ContainerStarted","Data":"2b0d37cf30e08be109664771b0faf1af8dd8b24b903ed8c799e1236928f8aee0"}
Sep 08 15:56:21.440575 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:21.440526    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" podStartSLOduration=1.440511022 podStartE2EDuration="1.440511022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:56:21.438981327 +0000 UTC m=+728.062673672" watchObservedRunningTime="2026-09-08 15:56:21.440511022 +0000 UTC m=+728.064203396"
Sep 08 15:56:21.650408 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:21.650364    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:22.245962 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:22.245926    2581 patch_prober.go:28] interesting pod/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 08 15:56:22.246156 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:22.245988    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 08 15:56:22.655679 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:22.655646    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:23.249822 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:23.249795    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:56:23.250098 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:23.250075    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:56:23.250831 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:23.250810    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:56:23.390447 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:23.390420    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:23.391133 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:23.391114    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:39.099916 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:39.098738    2581 patch_prober.go:28] interesting pod/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:59304->10.132.0.54:15021: read: connection reset by peer" start-of-body=
Sep 08 15:56:39.099916 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:39.098829    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:59304->10.132.0.54:15021: read: connection reset by peer"
Sep 08 15:56:39.476298 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:39.476262    2581 generic.go:356] "Generic (PLEG): container finished" podID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerID="1ecb52f1746f9c7fa347801a086a0cfef810194efdff7861fb9bdee660582275" exitCode=0
Sep 08 15:56:39.476513 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:39.476333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" event={"ID":"449bdd7a-7b16-469a-a5f4-5e45f834403e","Type":"ContainerDied","Data":"1ecb52f1746f9c7fa347801a086a0cfef810194efdff7861fb9bdee660582275"}
Sep 08 15:56:39.476837 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:39.476816    2581 scope.go:117] "RemoveContainer" containerID="1ecb52f1746f9c7fa347801a086a0cfef810194efdff7861fb9bdee660582275"
Sep 08 15:56:39.478037 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:39.478003    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:56:39.478103 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:39.478075    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:56:39.478144 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:39.478112    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:56:40.491150 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:40.491103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" event={"ID":"449bdd7a-7b16-469a-a5f4-5e45f834403e","Type":"ContainerStarted","Data":"afa32386171d57d9ee5112c07cb15317191d7f15be52983486a993e8e3643971"}
Sep 08 15:56:40.650830 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:40.650788    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:40.651047 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:40.651027    2581 patch_prober.go:28] interesting pod/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 08 15:56:40.651101 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:40.651074    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 08 15:56:41.650961 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:41.650926    2581 patch_prober.go:28] interesting pod/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 08 15:56:41.651338 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:41.650984    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 08 15:56:41.763883 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:41.763848    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"]
Sep 08 15:56:41.764129 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:41.764098    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq" podUID="af624738-f323-490d-a925-bca1cc7f5e69" containerName="maas-api" containerID="cri-o://bf12c5a2687d2a38a4aa25147c263877390f0cb68b2c752e640798fb7f3a1146" gracePeriod=30
Sep 08 15:56:41.829079 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:41.829045    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"]
Sep 08 15:56:41.829528 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:41.829496    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm" podUID="12613886-416a-4719-8bb3-6df373e00775" containerName="payload-processing" containerID="cri-o://92bfb94b0120805973e1a22347f3bf1af3c63160601e3d41150f7c0854f4499f" gracePeriod=30
Sep 08 15:56:41.837374 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:41.837345    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"]
Sep 08 15:56:41.838276 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:41.837584    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw" podUID="ad7e9c2d-1746-4356-ac25-024e9492a13f" containerName="payload-pre-processing" containerID="cri-o://6d25b7f7bb3d5b8d97c325690256293a4aecf0fc48ab6706dcdb1f0c5a95d077" gracePeriod=30
Sep 08 15:56:42.216553 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.216520    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"
Sep 08 15:56:42.219645 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.219619    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"]
Sep 08 15:56:42.219878 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.219849    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy" containerID="cri-o://29560e71e4dd737ebed95bc62645c3f576766621026f6c316ec492f0a4516d3e" gracePeriod=30
Sep 08 15:56:42.223954 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.223933    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"
Sep 08 15:56:42.227945 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.227924    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:42.237979 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.237961    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lds24\" (UniqueName: \"kubernetes.io/projected/ad7e9c2d-1746-4356-ac25-024e9492a13f-kube-api-access-lds24\") pod \"ad7e9c2d-1746-4356-ac25-024e9492a13f\" (UID: \"ad7e9c2d-1746-4356-ac25-024e9492a13f\") "
Sep 08 15:56:42.238063 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.238028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7e9c2d-1746-4356-ac25-024e9492a13f-plugins-config-volume\") pod \"ad7e9c2d-1746-4356-ac25-024e9492a13f\" (UID: \"ad7e9c2d-1746-4356-ac25-024e9492a13f\") "
Sep 08 15:56:42.238370 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.238347    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7e9c2d-1746-4356-ac25-024e9492a13f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad7e9c2d-1746-4356-ac25-024e9492a13f" (UID: "ad7e9c2d-1746-4356-ac25-024e9492a13f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:56:42.239985 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.239964    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7e9c2d-1746-4356-ac25-024e9492a13f-kube-api-access-lds24" (OuterVolumeSpecName: "kube-api-access-lds24") pod "ad7e9c2d-1746-4356-ac25-024e9492a13f" (UID: "ad7e9c2d-1746-4356-ac25-024e9492a13f"). InnerVolumeSpecName "kube-api-access-lds24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:56:42.338700 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.338659    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-metrics-certs\") pod \"af624738-f323-490d-a925-bca1cc7f5e69\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") "
Sep 08 15:56:42.338700 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.338697    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxcx6\" (UniqueName: \"kubernetes.io/projected/af624738-f323-490d-a925-bca1cc7f5e69-kube-api-access-cxcx6\") pod \"af624738-f323-490d-a925-bca1cc7f5e69\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") "
Sep 08 15:56:42.338951 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.338753    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-maas-api-tls\") pod \"af624738-f323-490d-a925-bca1cc7f5e69\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") "
Sep 08 15:56:42.338951 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.338769    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af624738-f323-490d-a925-bca1cc7f5e69-openshift-service-ca\") pod \"af624738-f323-490d-a925-bca1cc7f5e69\" (UID: \"af624738-f323-490d-a925-bca1cc7f5e69\") "
Sep 08 15:56:42.338951 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.338799    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12613886-416a-4719-8bb3-6df373e00775-plugins-config-volume\") pod \"12613886-416a-4719-8bb3-6df373e00775\" (UID: \"12613886-416a-4719-8bb3-6df373e00775\") "
Sep 08 15:56:42.338951 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.338843    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmfgz\" (UniqueName: \"kubernetes.io/projected/12613886-416a-4719-8bb3-6df373e00775-kube-api-access-vmfgz\") pod \"12613886-416a-4719-8bb3-6df373e00775\" (UID: \"12613886-416a-4719-8bb3-6df373e00775\") "
Sep 08 15:56:42.339177 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.339048    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lds24\" (UniqueName: \"kubernetes.io/projected/ad7e9c2d-1746-4356-ac25-024e9492a13f-kube-api-access-lds24\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:42.339177 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.339060    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7e9c2d-1746-4356-ac25-024e9492a13f-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:42.339634 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.339374    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af624738-f323-490d-a925-bca1cc7f5e69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af624738-f323-490d-a925-bca1cc7f5e69" (UID: "af624738-f323-490d-a925-bca1cc7f5e69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:56:42.339634 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.339534    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12613886-416a-4719-8bb3-6df373e00775-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12613886-416a-4719-8bb3-6df373e00775" (UID: "12613886-416a-4719-8bb3-6df373e00775"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:56:42.341948 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.341916    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "af624738-f323-490d-a925-bca1cc7f5e69" (UID: "af624738-f323-490d-a925-bca1cc7f5e69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:56:42.342037 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.341995    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af624738-f323-490d-a925-bca1cc7f5e69" (UID: "af624738-f323-490d-a925-bca1cc7f5e69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:56:42.342079 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.342037    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12613886-416a-4719-8bb3-6df373e00775-kube-api-access-vmfgz" (OuterVolumeSpecName: "kube-api-access-vmfgz") pod "12613886-416a-4719-8bb3-6df373e00775" (UID: "12613886-416a-4719-8bb3-6df373e00775"). InnerVolumeSpecName "kube-api-access-vmfgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:56:42.342079 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.342056    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af624738-f323-490d-a925-bca1cc7f5e69-kube-api-access-cxcx6" (OuterVolumeSpecName: "kube-api-access-cxcx6") pod "af624738-f323-490d-a925-bca1cc7f5e69" (UID: "af624738-f323-490d-a925-bca1cc7f5e69"). InnerVolumeSpecName "kube-api-access-cxcx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:56:42.439756 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.439714    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-metrics-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:42.439756 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.439748    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxcx6\" (UniqueName: \"kubernetes.io/projected/af624738-f323-490d-a925-bca1cc7f5e69-kube-api-access-cxcx6\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:42.439756 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.439760    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af624738-f323-490d-a925-bca1cc7f5e69-maas-api-tls\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:42.440023 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.439770    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af624738-f323-490d-a925-bca1cc7f5e69-openshift-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:42.440023 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.439778    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12613886-416a-4719-8bb3-6df373e00775-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:42.440023 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.439786    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmfgz\" (UniqueName: \"kubernetes.io/projected/12613886-416a-4719-8bb3-6df373e00775-kube-api-access-vmfgz\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:42.504891 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.504799    2581 generic.go:356] "Generic (PLEG): container finished" podID="12613886-416a-4719-8bb3-6df373e00775" containerID="92bfb94b0120805973e1a22347f3bf1af3c63160601e3d41150f7c0854f4499f" exitCode=0
Sep 08 15:56:42.504891 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.504857    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"
Sep 08 15:56:42.505070 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.504885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm" event={"ID":"12613886-416a-4719-8bb3-6df373e00775","Type":"ContainerDied","Data":"92bfb94b0120805973e1a22347f3bf1af3c63160601e3d41150f7c0854f4499f"}
Sep 08 15:56:42.505070 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.504964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm" event={"ID":"12613886-416a-4719-8bb3-6df373e00775","Type":"ContainerDied","Data":"234b3b8a22bbbf19ea546dbda6144c6cf0ec00142f23520ed1d7ebd5153d9495"}
Sep 08 15:56:42.505070 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.504992    2581 scope.go:117] "RemoveContainer" containerID="92bfb94b0120805973e1a22347f3bf1af3c63160601e3d41150f7c0854f4499f"
Sep 08 15:56:42.506606 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.506559    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad7e9c2d-1746-4356-ac25-024e9492a13f" containerID="6d25b7f7bb3d5b8d97c325690256293a4aecf0fc48ab6706dcdb1f0c5a95d077" exitCode=0
Sep 08 15:56:42.506722 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.506621    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"
Sep 08 15:56:42.506722 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.506638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw" event={"ID":"ad7e9c2d-1746-4356-ac25-024e9492a13f","Type":"ContainerDied","Data":"6d25b7f7bb3d5b8d97c325690256293a4aecf0fc48ab6706dcdb1f0c5a95d077"}
Sep 08 15:56:42.506722 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.506690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw" event={"ID":"ad7e9c2d-1746-4356-ac25-024e9492a13f","Type":"ContainerDied","Data":"b9ce74db5319bfebadf8dc0f6c91b42187b0409cdde8ce9fe72139ba9c1e97bc"}
Sep 08 15:56:42.508320 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.508299    2581 generic.go:356] "Generic (PLEG): container finished" podID="af624738-f323-490d-a925-bca1cc7f5e69" containerID="bf12c5a2687d2a38a4aa25147c263877390f0cb68b2c752e640798fb7f3a1146" exitCode=0
Sep 08 15:56:42.508461 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.508340    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq" event={"ID":"af624738-f323-490d-a925-bca1cc7f5e69","Type":"ContainerDied","Data":"bf12c5a2687d2a38a4aa25147c263877390f0cb68b2c752e640798fb7f3a1146"}
Sep 08 15:56:42.508461 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.508367    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"
Sep 08 15:56:42.508590 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.508383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq" event={"ID":"af624738-f323-490d-a925-bca1cc7f5e69","Type":"ContainerDied","Data":"c71e96ff2ef6f9be948d157aa53e8ddce61b54d8bdeca68364307e70502eedef"}
Sep 08 15:56:42.515712 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.515671    2581 scope.go:117] "RemoveContainer" containerID="92bfb94b0120805973e1a22347f3bf1af3c63160601e3d41150f7c0854f4499f"
Sep 08 15:56:42.515986 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:56:42.515965    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92bfb94b0120805973e1a22347f3bf1af3c63160601e3d41150f7c0854f4499f\": container with ID starting with 92bfb94b0120805973e1a22347f3bf1af3c63160601e3d41150f7c0854f4499f not found: ID does not exist" containerID="92bfb94b0120805973e1a22347f3bf1af3c63160601e3d41150f7c0854f4499f"
Sep 08 15:56:42.516069 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.515996    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bfb94b0120805973e1a22347f3bf1af3c63160601e3d41150f7c0854f4499f"} err="failed to get container status \"92bfb94b0120805973e1a22347f3bf1af3c63160601e3d41150f7c0854f4499f\": rpc error: code = NotFound desc = could not find container \"92bfb94b0120805973e1a22347f3bf1af3c63160601e3d41150f7c0854f4499f\": container with ID starting with 92bfb94b0120805973e1a22347f3bf1af3c63160601e3d41150f7c0854f4499f not found: ID does not exist"
Sep 08 15:56:42.516069 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.516018    2581 scope.go:117] "RemoveContainer" containerID="6d25b7f7bb3d5b8d97c325690256293a4aecf0fc48ab6706dcdb1f0c5a95d077"
Sep 08 15:56:42.528148 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.528127    2581 scope.go:117] "RemoveContainer" containerID="6d25b7f7bb3d5b8d97c325690256293a4aecf0fc48ab6706dcdb1f0c5a95d077"
Sep 08 15:56:42.528944 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:56:42.528922    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d25b7f7bb3d5b8d97c325690256293a4aecf0fc48ab6706dcdb1f0c5a95d077\": container with ID starting with 6d25b7f7bb3d5b8d97c325690256293a4aecf0fc48ab6706dcdb1f0c5a95d077 not found: ID does not exist" containerID="6d25b7f7bb3d5b8d97c325690256293a4aecf0fc48ab6706dcdb1f0c5a95d077"
Sep 08 15:56:42.529013 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.528952    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d25b7f7bb3d5b8d97c325690256293a4aecf0fc48ab6706dcdb1f0c5a95d077"} err="failed to get container status \"6d25b7f7bb3d5b8d97c325690256293a4aecf0fc48ab6706dcdb1f0c5a95d077\": rpc error: code = NotFound desc = could not find container \"6d25b7f7bb3d5b8d97c325690256293a4aecf0fc48ab6706dcdb1f0c5a95d077\": container with ID starting with 6d25b7f7bb3d5b8d97c325690256293a4aecf0fc48ab6706dcdb1f0c5a95d077 not found: ID does not exist"
Sep 08 15:56:42.529013 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.528970    2581 scope.go:117] "RemoveContainer" containerID="bf12c5a2687d2a38a4aa25147c263877390f0cb68b2c752e640798fb7f3a1146"
Sep 08 15:56:42.530020 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.529955    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"]
Sep 08 15:56:42.537274 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.537259    2581 scope.go:117] "RemoveContainer" containerID="bf12c5a2687d2a38a4aa25147c263877390f0cb68b2c752e640798fb7f3a1146"
Sep 08 15:56:42.537582 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:56:42.537549    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf12c5a2687d2a38a4aa25147c263877390f0cb68b2c752e640798fb7f3a1146\": container with ID starting with bf12c5a2687d2a38a4aa25147c263877390f0cb68b2c752e640798fb7f3a1146 not found: ID does not exist" containerID="bf12c5a2687d2a38a4aa25147c263877390f0cb68b2c752e640798fb7f3a1146"
Sep 08 15:56:42.537642 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.537589    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf12c5a2687d2a38a4aa25147c263877390f0cb68b2c752e640798fb7f3a1146"} err="failed to get container status \"bf12c5a2687d2a38a4aa25147c263877390f0cb68b2c752e640798fb7f3a1146\": rpc error: code = NotFound desc = could not find container \"bf12c5a2687d2a38a4aa25147c263877390f0cb68b2c752e640798fb7f3a1146\": container with ID starting with bf12c5a2687d2a38a4aa25147c263877390f0cb68b2c752e640798fb7f3a1146 not found: ID does not exist"
Sep 08 15:56:42.549682 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.549652    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cb6c353e-5bd4d54bc5-tqvgm"]
Sep 08 15:56:42.562539 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.562518    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"]
Sep 08 15:56:42.568328 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.568307    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cb6c353e-54d8c94945-d75zq"]
Sep 08 15:56:42.591417 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.591303    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"]
Sep 08 15:56:42.595015 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.594986    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb6c353e-b9f44c794-jjdjw"]
Sep 08 15:56:42.650181 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.650146    2581 patch_prober.go:28] interesting pod/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 08 15:56:42.650362 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:42.650218    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 08 15:56:43.187421 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.182926    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"]
Sep 08 15:56:43.198414 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.197087    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7e9c2d-1746-4356-ac25-024e9492a13f" containerName="payload-pre-processing"
Sep 08 15:56:43.198414 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.197116    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7e9c2d-1746-4356-ac25-024e9492a13f" containerName="payload-pre-processing"
Sep 08 15:56:43.198414 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.197131    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12613886-416a-4719-8bb3-6df373e00775" containerName="payload-processing"
Sep 08 15:56:43.198414 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.197141    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="12613886-416a-4719-8bb3-6df373e00775" containerName="payload-processing"
Sep 08 15:56:43.198414 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.197152    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af624738-f323-490d-a925-bca1cc7f5e69" containerName="maas-api"
Sep 08 15:56:43.198414 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.197160    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="af624738-f323-490d-a925-bca1cc7f5e69" containerName="maas-api"
Sep 08 15:56:43.198414 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.197274    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad7e9c2d-1746-4356-ac25-024e9492a13f" containerName="payload-pre-processing"
Sep 08 15:56:43.198414 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.197288    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="12613886-416a-4719-8bb3-6df373e00775" containerName="payload-processing"
Sep 08 15:56:43.198414 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.197300    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="af624738-f323-490d-a925-bca1cc7f5e69" containerName="maas-api"
Sep 08 15:56:43.202259 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.202237    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.205315 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.205291    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"]
Sep 08 15:56:43.205435 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.205355    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-173f5911-gw-openshift-default-dockercfg-vczh5\""
Sep 08 15:56:43.248625 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.248600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937bccdb-c05e-469c-881b-c2bdebab7394-istiod-ca-cert\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.248708 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.248667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937bccdb-c05e-469c-881b-c2bdebab7394-istio-token\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.248708 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.248705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-istio-data\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.248788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.248724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-credential-socket\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.248788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.248756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-workload-socket\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.248788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.248776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzrdk\" (UniqueName: \"kubernetes.io/projected/937bccdb-c05e-469c-881b-c2bdebab7394-kube-api-access-kzrdk\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.248924 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.248792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-istio-envoy\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.248924 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.248892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-workload-certs\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.248924 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.248917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937bccdb-c05e-469c-881b-c2bdebab7394-istio-podinfo\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.349793 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.349763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-istio-envoy\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.349941 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.349802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-workload-certs\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.349941 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.349824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937bccdb-c05e-469c-881b-c2bdebab7394-istio-podinfo\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.349941 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.349852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937bccdb-c05e-469c-881b-c2bdebab7394-istiod-ca-cert\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.349941 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.349884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937bccdb-c05e-469c-881b-c2bdebab7394-istio-token\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.349941 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.349908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-istio-data\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.349941 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.349928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-credential-socket\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.350138 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.349951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-workload-socket\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.350138 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.349969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzrdk\" (UniqueName: \"kubernetes.io/projected/937bccdb-c05e-469c-881b-c2bdebab7394-kube-api-access-kzrdk\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.350595 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.350563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-istio-data\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.350828 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.350809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-workload-certs\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.350887 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.350812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-credential-socket\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.351036 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.351020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-workload-socket\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.351260 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.351239    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937bccdb-c05e-469c-881b-c2bdebab7394-istiod-ca-cert\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.352772 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.352754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-istio-envoy\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.352824 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.352783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937bccdb-c05e-469c-881b-c2bdebab7394-istio-podinfo\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.363189 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.363165    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937bccdb-c05e-469c-881b-c2bdebab7394-istio-token\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.363678 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.363658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzrdk\" (UniqueName: \"kubernetes.io/projected/937bccdb-c05e-469c-881b-c2bdebab7394-kube-api-access-kzrdk\") pod \"e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") " pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.516093 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.515994    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:43.650642 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.650608    2581 patch_prober.go:28] interesting pod/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 08 15:56:43.650784 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.650671    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 08 15:56:43.830023 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.829988    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"]
Sep 08 15:56:43.830423 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:56:43.830381    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937bccdb_c05e_469c_881b_c2bdebab7394.slice/crio-a550fb96c4dd2d04689f42bfc50e441ecd4cef7d946f56e71bba39052bbbcc1d WatchSource:0}: Error finding container a550fb96c4dd2d04689f42bfc50e441ecd4cef7d946f56e71bba39052bbbcc1d: Status 404 returned error can't find the container with id a550fb96c4dd2d04689f42bfc50e441ecd4cef7d946f56e71bba39052bbbcc1d
Sep 08 15:56:43.832637 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.832520    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:56:43.832637 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.832605    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:56:43.832781 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.832657    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:56:43.904122 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.904087    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12613886-416a-4719-8bb3-6df373e00775" path="/var/lib/kubelet/pods/12613886-416a-4719-8bb3-6df373e00775/volumes"
Sep 08 15:56:43.904596 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.904573    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad7e9c2d-1746-4356-ac25-024e9492a13f" path="/var/lib/kubelet/pods/ad7e9c2d-1746-4356-ac25-024e9492a13f/volumes"
Sep 08 15:56:43.905016 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:43.904998    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af624738-f323-490d-a925-bca1cc7f5e69" path="/var/lib/kubelet/pods/af624738-f323-490d-a925-bca1cc7f5e69/volumes"
Sep 08 15:56:44.523802 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:44.523768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l" event={"ID":"937bccdb-c05e-469c-881b-c2bdebab7394","Type":"ContainerStarted","Data":"eb1dcab5d0024668bb08d46500c45c73577c94d3704d973d95c5edcc69070dc5"}
Sep 08 15:56:44.524173 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:44.523820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l" event={"ID":"937bccdb-c05e-469c-881b-c2bdebab7394","Type":"ContainerStarted","Data":"a550fb96c4dd2d04689f42bfc50e441ecd4cef7d946f56e71bba39052bbbcc1d"}
Sep 08 15:56:44.545735 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:44.545689    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l" podStartSLOduration=1.5456733379999998 podStartE2EDuration="1.545673338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:56:44.543488783 +0000 UTC m=+751.167181126" watchObservedRunningTime="2026-09-08 15:56:44.545673338 +0000 UTC m=+751.169365662"
Sep 08 15:56:44.650410 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:44.650365    2581 patch_prober.go:28] interesting pod/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 08 15:56:44.650584 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:44.650450    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 08 15:56:45.516789 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:45.516755    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:45.521939 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:45.521914    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:45.528659 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:45.528639    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:45.529319 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:45.529302    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:56:45.650711 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:45.650679    2581 patch_prober.go:28] interesting pod/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 08 15:56:45.650871 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:45.650729    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 08 15:56:46.290785 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.290746    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"]
Sep 08 15:56:46.295432 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.295106    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.300616 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.300528    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:56:46.301838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.300821    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:56:46.301838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.301137    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmx7t\""
Sep 08 15:56:46.301838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.301381    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:56:46.301838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.301590    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-173f5911-serving-cert\""
Sep 08 15:56:46.307062 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.306924    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"]
Sep 08 15:56:46.389816 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.389781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a999a96-e071-4847-9dd5-cabdaff5489b-maas-api-tls\") pod \"maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.390004 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.389844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a999a96-e071-4847-9dd5-cabdaff5489b-metrics-certs\") pod \"maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.390004 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.389863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a999a96-e071-4847-9dd5-cabdaff5489b-openshift-service-ca\") pod \"maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.390089 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.390017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg4kv\" (UniqueName: \"kubernetes.io/projected/7a999a96-e071-4847-9dd5-cabdaff5489b-kube-api-access-wg4kv\") pod \"maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.491027 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.490999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a999a96-e071-4847-9dd5-cabdaff5489b-metrics-certs\") pod \"maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.491027 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.491030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a999a96-e071-4847-9dd5-cabdaff5489b-openshift-service-ca\") pod \"maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.491238 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.491093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg4kv\" (UniqueName: \"kubernetes.io/projected/7a999a96-e071-4847-9dd5-cabdaff5489b-kube-api-access-wg4kv\") pod \"maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.491238 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.491131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a999a96-e071-4847-9dd5-cabdaff5489b-maas-api-tls\") pod \"maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.491841 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.491811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a999a96-e071-4847-9dd5-cabdaff5489b-openshift-service-ca\") pod \"maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.493513 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.493482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a999a96-e071-4847-9dd5-cabdaff5489b-metrics-certs\") pod \"maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.493659 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.493588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a999a96-e071-4847-9dd5-cabdaff5489b-maas-api-tls\") pod \"maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.498470 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.498445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg4kv\" (UniqueName: \"kubernetes.io/projected/7a999a96-e071-4847-9dd5-cabdaff5489b-kube-api-access-wg4kv\") pod \"maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.616834 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.616753    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:46.653856 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.653825    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:46.654134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.654110    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:46.654915 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.654896    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 15:56:46.894178 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.894150    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"]
Sep 08 15:56:46.896650 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:56:46.896614    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a999a96_e071_4847_9dd5_cabdaff5489b.slice/crio-d07a486e333a79f04f0889ed9580ca07612ca5eb70ebfdb0d96ab7337adafc49 WatchSource:0}: Error finding container d07a486e333a79f04f0889ed9580ca07612ca5eb70ebfdb0d96ab7337adafc49: Status 404 returned error can't find the container with id d07a486e333a79f04f0889ed9580ca07612ca5eb70ebfdb0d96ab7337adafc49
Sep 08 15:56:46.999439 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:46.999405    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"]
Sep 08 15:56:47.004195 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.004177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"
Sep 08 15:56:47.006387 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.006364    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-173f5911\""
Sep 08 15:56:47.006696 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.006679    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-173f5911-dockercfg-hwqqk\""
Sep 08 15:56:47.013907 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.013884    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"]
Sep 08 15:56:47.096601 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.096567    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"]
Sep 08 15:56:47.096970 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.096945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5qtd\" (UniqueName: \"kubernetes.io/projected/362192ce-4e65-4f6d-8643-2f7eefcdec32-kube-api-access-c5qtd\") pod \"payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8\" (UID: \"362192ce-4e65-4f6d-8643-2f7eefcdec32\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"
Sep 08 15:56:47.097055 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.097037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362192ce-4e65-4f6d-8643-2f7eefcdec32-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8\" (UID: \"362192ce-4e65-4f6d-8643-2f7eefcdec32\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"
Sep 08 15:56:47.100256 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.100229    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"
Sep 08 15:56:47.113016 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.112994    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"]
Sep 08 15:56:47.198106 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.198017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362192ce-4e65-4f6d-8643-2f7eefcdec32-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8\" (UID: \"362192ce-4e65-4f6d-8643-2f7eefcdec32\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"
Sep 08 15:56:47.198106 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.198062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae7609d-0cd7-40d4-a54e-de12237ba7af-plugins-config-volume\") pod \"payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z\" (UID: \"2ae7609d-0cd7-40d4-a54e-de12237ba7af\") " pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"
Sep 08 15:56:47.198364 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.198126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnnff\" (UniqueName: \"kubernetes.io/projected/2ae7609d-0cd7-40d4-a54e-de12237ba7af-kube-api-access-dnnff\") pod \"payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z\" (UID: \"2ae7609d-0cd7-40d4-a54e-de12237ba7af\") " pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"
Sep 08 15:56:47.198364 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.198163    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5qtd\" (UniqueName: \"kubernetes.io/projected/362192ce-4e65-4f6d-8643-2f7eefcdec32-kube-api-access-c5qtd\") pod \"payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8\" (UID: \"362192ce-4e65-4f6d-8643-2f7eefcdec32\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"
Sep 08 15:56:47.198881 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.198856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362192ce-4e65-4f6d-8643-2f7eefcdec32-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8\" (UID: \"362192ce-4e65-4f6d-8643-2f7eefcdec32\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"
Sep 08 15:56:47.208919 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.208897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5qtd\" (UniqueName: \"kubernetes.io/projected/362192ce-4e65-4f6d-8643-2f7eefcdec32-kube-api-access-c5qtd\") pod \"payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8\" (UID: \"362192ce-4e65-4f6d-8643-2f7eefcdec32\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"
Sep 08 15:56:47.299446 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.299412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae7609d-0cd7-40d4-a54e-de12237ba7af-plugins-config-volume\") pod \"payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z\" (UID: \"2ae7609d-0cd7-40d4-a54e-de12237ba7af\") " pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"
Sep 08 15:56:47.299609 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.299524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnnff\" (UniqueName: \"kubernetes.io/projected/2ae7609d-0cd7-40d4-a54e-de12237ba7af-kube-api-access-dnnff\") pod \"payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z\" (UID: \"2ae7609d-0cd7-40d4-a54e-de12237ba7af\") " pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"
Sep 08 15:56:47.300088 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.300051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae7609d-0cd7-40d4-a54e-de12237ba7af-plugins-config-volume\") pod \"payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z\" (UID: \"2ae7609d-0cd7-40d4-a54e-de12237ba7af\") " pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"
Sep 08 15:56:47.307713 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.307691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnnff\" (UniqueName: \"kubernetes.io/projected/2ae7609d-0cd7-40d4-a54e-de12237ba7af-kube-api-access-dnnff\") pod \"payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z\" (UID: \"2ae7609d-0cd7-40d4-a54e-de12237ba7af\") " pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"
Sep 08 15:56:47.316661 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.316633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"
Sep 08 15:56:47.411133 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.410789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"
Sep 08 15:56:47.527941 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.527910    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:56:47.541015 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.540985    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx" event={"ID":"7a999a96-e071-4847-9dd5-cabdaff5489b","Type":"ContainerStarted","Data":"de5e4392c15ef788be11e59464eba0466a4d5eaae27a97833259332fd83b4c5c"}
Sep 08 15:56:47.541154 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.541041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx" event={"ID":"7a999a96-e071-4847-9dd5-cabdaff5489b","Type":"ContainerStarted","Data":"d07a486e333a79f04f0889ed9580ca07612ca5eb70ebfdb0d96ab7337adafc49"}
Sep 08 15:56:47.541154 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.541069    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:47.542641 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.542618    2581 generic.go:356] "Generic (PLEG): container finished" podID="9e161569-3090-47f5-935c-798e9888494c" containerID="29560e71e4dd737ebed95bc62645c3f576766621026f6c316ec492f0a4516d3e" exitCode=0
Sep 08 15:56:47.542744 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.542651    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"
Sep 08 15:56:47.542744 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.542697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" event={"ID":"9e161569-3090-47f5-935c-798e9888494c","Type":"ContainerDied","Data":"29560e71e4dd737ebed95bc62645c3f576766621026f6c316ec492f0a4516d3e"}
Sep 08 15:56:47.542817 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.542745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g" event={"ID":"9e161569-3090-47f5-935c-798e9888494c","Type":"ContainerDied","Data":"47d177e1b8de40f4912b2d68023ab2de3d2aef47d55ace11c6edd4da05ab02d7"}
Sep 08 15:56:47.542817 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.542768    2581 scope.go:117] "RemoveContainer" containerID="29560e71e4dd737ebed95bc62645c3f576766621026f6c316ec492f0a4516d3e"
Sep 08 15:56:47.553679 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.553653    2581 scope.go:117] "RemoveContainer" containerID="12c7521909ec0bf938af7f670a830214a9083be6170b0b7b3fd97ebdeac14639"
Sep 08 15:56:47.562450 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.562431    2581 scope.go:117] "RemoveContainer" containerID="29560e71e4dd737ebed95bc62645c3f576766621026f6c316ec492f0a4516d3e"
Sep 08 15:56:47.562717 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:56:47.562697    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29560e71e4dd737ebed95bc62645c3f576766621026f6c316ec492f0a4516d3e\": container with ID starting with 29560e71e4dd737ebed95bc62645c3f576766621026f6c316ec492f0a4516d3e not found: ID does not exist" containerID="29560e71e4dd737ebed95bc62645c3f576766621026f6c316ec492f0a4516d3e"
Sep 08 15:56:47.562801 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.562725    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29560e71e4dd737ebed95bc62645c3f576766621026f6c316ec492f0a4516d3e"} err="failed to get container status \"29560e71e4dd737ebed95bc62645c3f576766621026f6c316ec492f0a4516d3e\": rpc error: code = NotFound desc = could not find container \"29560e71e4dd737ebed95bc62645c3f576766621026f6c316ec492f0a4516d3e\": container with ID starting with 29560e71e4dd737ebed95bc62645c3f576766621026f6c316ec492f0a4516d3e not found: ID does not exist"
Sep 08 15:56:47.562801 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.562742    2581 scope.go:117] "RemoveContainer" containerID="12c7521909ec0bf938af7f670a830214a9083be6170b0b7b3fd97ebdeac14639"
Sep 08 15:56:47.562968 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:56:47.562953    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c7521909ec0bf938af7f670a830214a9083be6170b0b7b3fd97ebdeac14639\": container with ID starting with 12c7521909ec0bf938af7f670a830214a9083be6170b0b7b3fd97ebdeac14639 not found: ID does not exist" containerID="12c7521909ec0bf938af7f670a830214a9083be6170b0b7b3fd97ebdeac14639"
Sep 08 15:56:47.563027 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.562971    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c7521909ec0bf938af7f670a830214a9083be6170b0b7b3fd97ebdeac14639"} err="failed to get container status \"12c7521909ec0bf938af7f670a830214a9083be6170b0b7b3fd97ebdeac14639\": rpc error: code = NotFound desc = could not find container \"12c7521909ec0bf938af7f670a830214a9083be6170b0b7b3fd97ebdeac14639\": container with ID starting with 12c7521909ec0bf938af7f670a830214a9083be6170b0b7b3fd97ebdeac14639 not found: ID does not exist"
Sep 08 15:56:47.573813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.573768    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx" podStartSLOduration=1.147493066 podStartE2EDuration="1.573754759s" totalImagesPullingTime="426.261693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:56:47.571684601 +0000 UTC m=+754.195376926" watchObservedRunningTime="2026-09-08 15:56:47.573754759 +0000 UTC m=+754.197447082"
Sep 08 15:56:47.603217 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.603185    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-workload-certs\") pod \"9e161569-3090-47f5-935c-798e9888494c\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") "
Sep 08 15:56:47.603360 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.603232    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-istio-envoy\") pod \"9e161569-3090-47f5-935c-798e9888494c\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") "
Sep 08 15:56:47.603360 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.603276    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-workload-socket\") pod \"9e161569-3090-47f5-935c-798e9888494c\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") "
Sep 08 15:56:47.603360 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.603335    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85c24\" (UniqueName: \"kubernetes.io/projected/9e161569-3090-47f5-935c-798e9888494c-kube-api-access-85c24\") pod \"9e161569-3090-47f5-935c-798e9888494c\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") "
Sep 08 15:56:47.603666 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.603363    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-credential-socket\") pod \"9e161569-3090-47f5-935c-798e9888494c\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") "
Sep 08 15:56:47.603666 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.603439    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e161569-3090-47f5-935c-798e9888494c-istio-podinfo\") pod \"9e161569-3090-47f5-935c-798e9888494c\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") "
Sep 08 15:56:47.603666 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.603491    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e161569-3090-47f5-935c-798e9888494c" (UID: "9e161569-3090-47f5-935c-798e9888494c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:56:47.603666 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.603498    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-istio-data\") pod \"9e161569-3090-47f5-935c-798e9888494c\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") "
Sep 08 15:56:47.603666 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.603542    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e161569-3090-47f5-935c-798e9888494c-istio-token\") pod \"9e161569-3090-47f5-935c-798e9888494c\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") "
Sep 08 15:56:47.603666 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.603573    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e161569-3090-47f5-935c-798e9888494c-istiod-ca-cert\") pod \"9e161569-3090-47f5-935c-798e9888494c\" (UID: \"9e161569-3090-47f5-935c-798e9888494c\") "
Sep 08 15:56:47.604054 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.603940    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e161569-3090-47f5-935c-798e9888494c" (UID: "9e161569-3090-47f5-935c-798e9888494c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:56:47.605734 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.603971    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e161569-3090-47f5-935c-798e9888494c" (UID: "9e161569-3090-47f5-935c-798e9888494c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:56:47.605734 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.604284    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-workload-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.605734 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.604350    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-workload-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.605734 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.604369    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-credential-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.605734 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.604553    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e161569-3090-47f5-935c-798e9888494c" (UID: "9e161569-3090-47f5-935c-798e9888494c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:56:47.605734 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.605658    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e161569-3090-47f5-935c-798e9888494c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e161569-3090-47f5-935c-798e9888494c" (UID: "9e161569-3090-47f5-935c-798e9888494c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:56:47.606431 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.606406    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e161569-3090-47f5-935c-798e9888494c" (UID: "9e161569-3090-47f5-935c-798e9888494c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:56:47.606748 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.606707    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e161569-3090-47f5-935c-798e9888494c-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e161569-3090-47f5-935c-798e9888494c" (UID: "9e161569-3090-47f5-935c-798e9888494c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:56:47.607501 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.607476    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e161569-3090-47f5-935c-798e9888494c-kube-api-access-85c24" (OuterVolumeSpecName: "kube-api-access-85c24") pod "9e161569-3090-47f5-935c-798e9888494c" (UID: "9e161569-3090-47f5-935c-798e9888494c"). InnerVolumeSpecName "kube-api-access-85c24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:56:47.607997 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.607971    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e161569-3090-47f5-935c-798e9888494c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e161569-3090-47f5-935c-798e9888494c" (UID: "9e161569-3090-47f5-935c-798e9888494c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 15:56:47.608620 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.608601    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"]
Sep 08 15:56:47.609979 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:56:47.609958    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362192ce_4e65_4f6d_8643_2f7eefcdec32.slice/crio-c715434f562d0c976c5b6f6096a1935756b3cfd5d1dd32ed8b6e1ae7a0c35ced WatchSource:0}: Error finding container c715434f562d0c976c5b6f6096a1935756b3cfd5d1dd32ed8b6e1ae7a0c35ced: Status 404 returned error can't find the container with id c715434f562d0c976c5b6f6096a1935756b3cfd5d1dd32ed8b6e1ae7a0c35ced
Sep 08 15:56:47.693148 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.693123    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"]
Sep 08 15:56:47.695675 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:56:47.695643    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae7609d_0cd7_40d4_a54e_de12237ba7af.slice/crio-199a3a74761817406db944439468be9ef0af5461fad4c6cbb077f46239bce04d WatchSource:0}: Error finding container 199a3a74761817406db944439468be9ef0af5461fad4c6cbb077f46239bce04d: Status 404 returned error can't find the container with id 199a3a74761817406db944439468be9ef0af5461fad4c6cbb077f46239bce04d
Sep 08 15:56:47.706416 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.705591    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85c24\" (UniqueName: \"kubernetes.io/projected/9e161569-3090-47f5-935c-798e9888494c-kube-api-access-85c24\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.706416 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.705623    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e161569-3090-47f5-935c-798e9888494c-istio-podinfo\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.706416 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.705645    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-istio-data\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.706416 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.705659    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e161569-3090-47f5-935c-798e9888494c-istio-token\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.706416 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.705674    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e161569-3090-47f5-935c-798e9888494c-istiod-ca-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.706416 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.705686    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e161569-3090-47f5-935c-798e9888494c-istio-envoy\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:56:47.868223 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.868194    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"]
Sep 08 15:56:47.877388 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.877356    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cb6c353e-openshift-default-f444bcb6d-q252g"]
Sep 08 15:56:47.902001 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:47.901969    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e161569-3090-47f5-935c-798e9888494c" path="/var/lib/kubelet/pods/9e161569-3090-47f5-935c-798e9888494c/volumes"
Sep 08 15:56:48.549903 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:48.549867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8" event={"ID":"362192ce-4e65-4f6d-8643-2f7eefcdec32","Type":"ContainerStarted","Data":"cb362b3dc8712954882f15ed49aee7f2b515847067a4a8cd6d144b99d1480ad2"}
Sep 08 15:56:48.550059 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:48.549944    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"
Sep 08 15:56:48.550059 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:48.549973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8" event={"ID":"362192ce-4e65-4f6d-8643-2f7eefcdec32","Type":"ContainerStarted","Data":"c715434f562d0c976c5b6f6096a1935756b3cfd5d1dd32ed8b6e1ae7a0c35ced"}
Sep 08 15:56:48.552320 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:48.552290    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z" event={"ID":"2ae7609d-0cd7-40d4-a54e-de12237ba7af","Type":"ContainerStarted","Data":"7da384fd0d2b02272f88871b541922e4e793983ee6601c87ec481694fb7d7c0c"}
Sep 08 15:56:48.552464 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:48.552343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z" event={"ID":"2ae7609d-0cd7-40d4-a54e-de12237ba7af","Type":"ContainerStarted","Data":"199a3a74761817406db944439468be9ef0af5461fad4c6cbb077f46239bce04d"}
Sep 08 15:56:48.552536 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:48.552512    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"
Sep 08 15:56:48.566574 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:48.566534    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8" podStartSLOduration=2.566519586 podStartE2EDuration="2.566519586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:56:48.565787602 +0000 UTC m=+755.189479925" watchObservedRunningTime="2026-09-08 15:56:48.566519586 +0000 UTC m=+755.190211910"
Sep 08 15:56:48.581790 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:48.581749    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z" podStartSLOduration=1.581736277 podStartE2EDuration="1.581736277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:56:48.581134836 +0000 UTC m=+755.204827161" watchObservedRunningTime="2026-09-08 15:56:48.581736277 +0000 UTC m=+755.205428601"
Sep 08 15:56:53.558017 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:53.557989    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:56:59.562274 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:59.562238    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"
Sep 08 15:56:59.562700 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:56:59.562414    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"
Sep 08 15:57:06.654781 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:06.654747    2581 generic.go:356] "Generic (PLEG): container finished" podID="937bccdb-c05e-469c-881b-c2bdebab7394" containerID="eb1dcab5d0024668bb08d46500c45c73577c94d3704d973d95c5edcc69070dc5" exitCode=0
Sep 08 15:57:06.655202 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:06.654820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l" event={"ID":"937bccdb-c05e-469c-881b-c2bdebab7394","Type":"ContainerDied","Data":"eb1dcab5d0024668bb08d46500c45c73577c94d3704d973d95c5edcc69070dc5"}
Sep 08 15:57:06.655202 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:06.655167    2581 scope.go:117] "RemoveContainer" containerID="eb1dcab5d0024668bb08d46500c45c73577c94d3704d973d95c5edcc69070dc5"
Sep 08 15:57:06.656208 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:06.656183    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:57:06.656301 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:06.656256    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:57:06.656301 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:06.656291    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:57:07.661951 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:07.661915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l" event={"ID":"937bccdb-c05e-469c-881b-c2bdebab7394","Type":"ContainerStarted","Data":"dd7ac1a07fbc3a939af1cd0be7828a1e2efa37bd6a30ca1945a4a07f06c6be87"}
Sep 08 15:57:08.516860 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:08.516822    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:57:08.517123 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:08.517024    2581 patch_prober.go:28] interesting pod/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 08 15:57:08.517123 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:08.517083    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l" podUID="937bccdb-c05e-469c-881b-c2bdebab7394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 08 15:57:09.516619 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:09.516584    2581 patch_prober.go:28] interesting pod/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 08 15:57:09.517002 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:09.516637    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l" podUID="937bccdb-c05e-469c-881b-c2bdebab7394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 08 15:57:10.516714 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:10.516682    2581 patch_prober.go:28] interesting pod/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 08 15:57:10.517110 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:10.516733    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l" podUID="937bccdb-c05e-469c-881b-c2bdebab7394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 08 15:57:11.517207 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:11.517167    2581 patch_prober.go:28] interesting pod/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 08 15:57:11.517636 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:11.517230    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l" podUID="937bccdb-c05e-469c-881b-c2bdebab7394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 08 15:57:12.517199 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:12.517154    2581 patch_prober.go:28] interesting pod/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 08 15:57:12.517372 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:12.517222    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l" podUID="937bccdb-c05e-469c-881b-c2bdebab7394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 08 15:57:13.521586 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:13.521555    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:57:13.521977 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:13.521823    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:57:13.522621 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:13.522595    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:57:17.674337 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:17.674293    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"]
Sep 08 15:57:17.674840 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:17.674521    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx" podUID="7a999a96-e071-4847-9dd5-cabdaff5489b" containerName="maas-api" containerID="cri-o://de5e4392c15ef788be11e59464eba0466a4d5eaae27a97833259332fd83b4c5c" gracePeriod=30
Sep 08 15:57:17.737373 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:17.737343    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"]
Sep 08 15:57:17.737620 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:17.737573    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z" podUID="2ae7609d-0cd7-40d4-a54e-de12237ba7af" containerName="payload-processing" containerID="cri-o://7da384fd0d2b02272f88871b541922e4e793983ee6601c87ec481694fb7d7c0c" gracePeriod=30
Sep 08 15:57:17.745505 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:17.745479    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"]
Sep 08 15:57:17.745811 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:17.745752    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8" podUID="362192ce-4e65-4f6d-8643-2f7eefcdec32" containerName="payload-pre-processing" containerID="cri-o://cb362b3dc8712954882f15ed49aee7f2b515847067a4a8cd6d144b99d1480ad2" gracePeriod=30
Sep 08 15:57:18.019990 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.019852    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"
Sep 08 15:57:18.044161 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.044126    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"]
Sep 08 15:57:18.044572 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.044540    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l" podUID="937bccdb-c05e-469c-881b-c2bdebab7394" containerName="istio-proxy" containerID="cri-o://dd7ac1a07fbc3a939af1cd0be7828a1e2efa37bd6a30ca1945a4a07f06c6be87" gracePeriod=30
Sep 08 15:57:18.059790 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.059760    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:57:18.075313 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.075293    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"
Sep 08 15:57:18.107850 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.107826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg4kv\" (UniqueName: \"kubernetes.io/projected/7a999a96-e071-4847-9dd5-cabdaff5489b-kube-api-access-wg4kv\") pod \"7a999a96-e071-4847-9dd5-cabdaff5489b\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") "
Sep 08 15:57:18.107967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.107857    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnnff\" (UniqueName: \"kubernetes.io/projected/2ae7609d-0cd7-40d4-a54e-de12237ba7af-kube-api-access-dnnff\") pod \"2ae7609d-0cd7-40d4-a54e-de12237ba7af\" (UID: \"2ae7609d-0cd7-40d4-a54e-de12237ba7af\") "
Sep 08 15:57:18.107967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.107898    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a999a96-e071-4847-9dd5-cabdaff5489b-metrics-certs\") pod \"7a999a96-e071-4847-9dd5-cabdaff5489b\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") "
Sep 08 15:57:18.107967 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.107961    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae7609d-0cd7-40d4-a54e-de12237ba7af-plugins-config-volume\") pod \"2ae7609d-0cd7-40d4-a54e-de12237ba7af\" (UID: \"2ae7609d-0cd7-40d4-a54e-de12237ba7af\") "
Sep 08 15:57:18.108323 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.108068    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a999a96-e071-4847-9dd5-cabdaff5489b-openshift-service-ca\") pod \"7a999a96-e071-4847-9dd5-cabdaff5489b\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") "
Sep 08 15:57:18.108323 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.108096    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a999a96-e071-4847-9dd5-cabdaff5489b-maas-api-tls\") pod \"7a999a96-e071-4847-9dd5-cabdaff5489b\" (UID: \"7a999a96-e071-4847-9dd5-cabdaff5489b\") "
Sep 08 15:57:18.108712 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.108649    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae7609d-0cd7-40d4-a54e-de12237ba7af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ae7609d-0cd7-40d4-a54e-de12237ba7af" (UID: "2ae7609d-0cd7-40d4-a54e-de12237ba7af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:57:18.108805 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.108747    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a999a96-e071-4847-9dd5-cabdaff5489b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a999a96-e071-4847-9dd5-cabdaff5489b" (UID: "7a999a96-e071-4847-9dd5-cabdaff5489b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:57:18.110679 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.110640    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a999a96-e071-4847-9dd5-cabdaff5489b-kube-api-access-wg4kv" (OuterVolumeSpecName: "kube-api-access-wg4kv") pod "7a999a96-e071-4847-9dd5-cabdaff5489b" (UID: "7a999a96-e071-4847-9dd5-cabdaff5489b"). InnerVolumeSpecName "kube-api-access-wg4kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:18.110761 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.110678    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae7609d-0cd7-40d4-a54e-de12237ba7af-kube-api-access-dnnff" (OuterVolumeSpecName: "kube-api-access-dnnff") pod "2ae7609d-0cd7-40d4-a54e-de12237ba7af" (UID: "2ae7609d-0cd7-40d4-a54e-de12237ba7af"). InnerVolumeSpecName "kube-api-access-dnnff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:18.110999 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.110976    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a999a96-e071-4847-9dd5-cabdaff5489b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a999a96-e071-4847-9dd5-cabdaff5489b" (UID: "7a999a96-e071-4847-9dd5-cabdaff5489b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:57:18.111837 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.111813    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a999a96-e071-4847-9dd5-cabdaff5489b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a999a96-e071-4847-9dd5-cabdaff5489b" (UID: "7a999a96-e071-4847-9dd5-cabdaff5489b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:57:18.209374 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.209291    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362192ce-4e65-4f6d-8643-2f7eefcdec32-plugins-config-volume\") pod \"362192ce-4e65-4f6d-8643-2f7eefcdec32\" (UID: \"362192ce-4e65-4f6d-8643-2f7eefcdec32\") "
Sep 08 15:57:18.209374 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.209355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5qtd\" (UniqueName: \"kubernetes.io/projected/362192ce-4e65-4f6d-8643-2f7eefcdec32-kube-api-access-c5qtd\") pod \"362192ce-4e65-4f6d-8643-2f7eefcdec32\" (UID: \"362192ce-4e65-4f6d-8643-2f7eefcdec32\") "
Sep 08 15:57:18.209665 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.209649    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a999a96-e071-4847-9dd5-cabdaff5489b-metrics-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:18.209731 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.209666    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362192ce-4e65-4f6d-8643-2f7eefcdec32-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "362192ce-4e65-4f6d-8643-2f7eefcdec32" (UID: "362192ce-4e65-4f6d-8643-2f7eefcdec32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:57:18.209731 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.209671    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ae7609d-0cd7-40d4-a54e-de12237ba7af-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:18.209731 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.209708    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a999a96-e071-4847-9dd5-cabdaff5489b-openshift-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:18.209731 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.209721    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a999a96-e071-4847-9dd5-cabdaff5489b-maas-api-tls\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:18.209946 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.209733    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg4kv\" (UniqueName: \"kubernetes.io/projected/7a999a96-e071-4847-9dd5-cabdaff5489b-kube-api-access-wg4kv\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:18.209946 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.209746    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnnff\" (UniqueName: \"kubernetes.io/projected/2ae7609d-0cd7-40d4-a54e-de12237ba7af-kube-api-access-dnnff\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:18.211419 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.211369    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362192ce-4e65-4f6d-8643-2f7eefcdec32-kube-api-access-c5qtd" (OuterVolumeSpecName: "kube-api-access-c5qtd") pod "362192ce-4e65-4f6d-8643-2f7eefcdec32" (UID: "362192ce-4e65-4f6d-8643-2f7eefcdec32"). InnerVolumeSpecName "kube-api-access-c5qtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:18.311074 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.311047    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362192ce-4e65-4f6d-8643-2f7eefcdec32-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:18.311074 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.311070    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5qtd\" (UniqueName: \"kubernetes.io/projected/362192ce-4e65-4f6d-8643-2f7eefcdec32-kube-api-access-c5qtd\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:18.740882 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.740853    2581 generic.go:356] "Generic (PLEG): container finished" podID="362192ce-4e65-4f6d-8643-2f7eefcdec32" containerID="cb362b3dc8712954882f15ed49aee7f2b515847067a4a8cd6d144b99d1480ad2" exitCode=0
Sep 08 15:57:18.741217 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.740897    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"
Sep 08 15:57:18.741217 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.740939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8" event={"ID":"362192ce-4e65-4f6d-8643-2f7eefcdec32","Type":"ContainerDied","Data":"cb362b3dc8712954882f15ed49aee7f2b515847067a4a8cd6d144b99d1480ad2"}
Sep 08 15:57:18.741217 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.740988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8" event={"ID":"362192ce-4e65-4f6d-8643-2f7eefcdec32","Type":"ContainerDied","Data":"c715434f562d0c976c5b6f6096a1935756b3cfd5d1dd32ed8b6e1ae7a0c35ced"}
Sep 08 15:57:18.741217 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.741013    2581 scope.go:117] "RemoveContainer" containerID="cb362b3dc8712954882f15ed49aee7f2b515847067a4a8cd6d144b99d1480ad2"
Sep 08 15:57:18.742381 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.742359    2581 generic.go:356] "Generic (PLEG): container finished" podID="7a999a96-e071-4847-9dd5-cabdaff5489b" containerID="de5e4392c15ef788be11e59464eba0466a4d5eaae27a97833259332fd83b4c5c" exitCode=0
Sep 08 15:57:18.742487 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.742414    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"
Sep 08 15:57:18.742487 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.742425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx" event={"ID":"7a999a96-e071-4847-9dd5-cabdaff5489b","Type":"ContainerDied","Data":"de5e4392c15ef788be11e59464eba0466a4d5eaae27a97833259332fd83b4c5c"}
Sep 08 15:57:18.742487 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.742483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx" event={"ID":"7a999a96-e071-4847-9dd5-cabdaff5489b","Type":"ContainerDied","Data":"d07a486e333a79f04f0889ed9580ca07612ca5eb70ebfdb0d96ab7337adafc49"}
Sep 08 15:57:18.744077 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.744051    2581 generic.go:356] "Generic (PLEG): container finished" podID="2ae7609d-0cd7-40d4-a54e-de12237ba7af" containerID="7da384fd0d2b02272f88871b541922e4e793983ee6601c87ec481694fb7d7c0c" exitCode=0
Sep 08 15:57:18.744182 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.744100    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"
Sep 08 15:57:18.744182 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.744139    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z" event={"ID":"2ae7609d-0cd7-40d4-a54e-de12237ba7af","Type":"ContainerDied","Data":"7da384fd0d2b02272f88871b541922e4e793983ee6601c87ec481694fb7d7c0c"}
Sep 08 15:57:18.744273 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.744190    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z" event={"ID":"2ae7609d-0cd7-40d4-a54e-de12237ba7af","Type":"ContainerDied","Data":"199a3a74761817406db944439468be9ef0af5461fad4c6cbb077f46239bce04d"}
Sep 08 15:57:18.751939 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.751913    2581 scope.go:117] "RemoveContainer" containerID="cb362b3dc8712954882f15ed49aee7f2b515847067a4a8cd6d144b99d1480ad2"
Sep 08 15:57:18.752263 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:57:18.752240    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb362b3dc8712954882f15ed49aee7f2b515847067a4a8cd6d144b99d1480ad2\": container with ID starting with cb362b3dc8712954882f15ed49aee7f2b515847067a4a8cd6d144b99d1480ad2 not found: ID does not exist" containerID="cb362b3dc8712954882f15ed49aee7f2b515847067a4a8cd6d144b99d1480ad2"
Sep 08 15:57:18.752339 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.752272    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb362b3dc8712954882f15ed49aee7f2b515847067a4a8cd6d144b99d1480ad2"} err="failed to get container status \"cb362b3dc8712954882f15ed49aee7f2b515847067a4a8cd6d144b99d1480ad2\": rpc error: code = NotFound desc = could not find container \"cb362b3dc8712954882f15ed49aee7f2b515847067a4a8cd6d144b99d1480ad2\": container with ID starting with cb362b3dc8712954882f15ed49aee7f2b515847067a4a8cd6d144b99d1480ad2 not found: ID does not exist"
Sep 08 15:57:18.752339 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.752294    2581 scope.go:117] "RemoveContainer" containerID="de5e4392c15ef788be11e59464eba0466a4d5eaae27a97833259332fd83b4c5c"
Sep 08 15:57:18.765663 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.765628    2581 scope.go:117] "RemoveContainer" containerID="de5e4392c15ef788be11e59464eba0466a4d5eaae27a97833259332fd83b4c5c"
Sep 08 15:57:18.766079 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:57:18.766039    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5e4392c15ef788be11e59464eba0466a4d5eaae27a97833259332fd83b4c5c\": container with ID starting with de5e4392c15ef788be11e59464eba0466a4d5eaae27a97833259332fd83b4c5c not found: ID does not exist" containerID="de5e4392c15ef788be11e59464eba0466a4d5eaae27a97833259332fd83b4c5c"
Sep 08 15:57:18.766198 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.766072    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5e4392c15ef788be11e59464eba0466a4d5eaae27a97833259332fd83b4c5c"} err="failed to get container status \"de5e4392c15ef788be11e59464eba0466a4d5eaae27a97833259332fd83b4c5c\": rpc error: code = NotFound desc = could not find container \"de5e4392c15ef788be11e59464eba0466a4d5eaae27a97833259332fd83b4c5c\": container with ID starting with de5e4392c15ef788be11e59464eba0466a4d5eaae27a97833259332fd83b4c5c not found: ID does not exist"
Sep 08 15:57:18.766198 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.766096    2581 scope.go:117] "RemoveContainer" containerID="7da384fd0d2b02272f88871b541922e4e793983ee6601c87ec481694fb7d7c0c"
Sep 08 15:57:18.773059 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773037    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"]
Sep 08 15:57:18.773684 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773663    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy"
Sep 08 15:57:18.773754 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773689    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy"
Sep 08 15:57:18.773754 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773736    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy"
Sep 08 15:57:18.773754 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773745    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy"
Sep 08 15:57:18.773909 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773754    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae7609d-0cd7-40d4-a54e-de12237ba7af" containerName="payload-processing"
Sep 08 15:57:18.773909 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773763    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae7609d-0cd7-40d4-a54e-de12237ba7af" containerName="payload-processing"
Sep 08 15:57:18.773909 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773785    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a999a96-e071-4847-9dd5-cabdaff5489b" containerName="maas-api"
Sep 08 15:57:18.773909 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773793    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a999a96-e071-4847-9dd5-cabdaff5489b" containerName="maas-api"
Sep 08 15:57:18.773909 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773802    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362192ce-4e65-4f6d-8643-2f7eefcdec32" containerName="payload-pre-processing"
Sep 08 15:57:18.773909 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773810    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="362192ce-4e65-4f6d-8643-2f7eefcdec32" containerName="payload-pre-processing"
Sep 08 15:57:18.773909 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773908    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a999a96-e071-4847-9dd5-cabdaff5489b" containerName="maas-api"
Sep 08 15:57:18.774306 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773920    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae7609d-0cd7-40d4-a54e-de12237ba7af" containerName="payload-processing"
Sep 08 15:57:18.774306 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773934    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy"
Sep 08 15:57:18.774306 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773944    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="362192ce-4e65-4f6d-8643-2f7eefcdec32" containerName="payload-pre-processing"
Sep 08 15:57:18.774306 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.773954    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e161569-3090-47f5-935c-798e9888494c" containerName="istio-proxy"
Sep 08 15:57:18.778655 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.778564    2581 scope.go:117] "RemoveContainer" containerID="7da384fd0d2b02272f88871b541922e4e793983ee6601c87ec481694fb7d7c0c"
Sep 08 15:57:18.779627 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:57:18.779305    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da384fd0d2b02272f88871b541922e4e793983ee6601c87ec481694fb7d7c0c\": container with ID starting with 7da384fd0d2b02272f88871b541922e4e793983ee6601c87ec481694fb7d7c0c not found: ID does not exist" containerID="7da384fd0d2b02272f88871b541922e4e793983ee6601c87ec481694fb7d7c0c"
Sep 08 15:57:18.779627 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.779347    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da384fd0d2b02272f88871b541922e4e793983ee6601c87ec481694fb7d7c0c"} err="failed to get container status \"7da384fd0d2b02272f88871b541922e4e793983ee6601c87ec481694fb7d7c0c\": rpc error: code = NotFound desc = could not find container \"7da384fd0d2b02272f88871b541922e4e793983ee6601c87ec481694fb7d7c0c\": container with ID starting with 7da384fd0d2b02272f88871b541922e4e793983ee6601c87ec481694fb7d7c0c not found: ID does not exist"
Sep 08 15:57:18.780111 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.779912    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:18.780111 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.780059    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"]
Sep 08 15:57:18.782047 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.782031    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ac3191f1-openshift-default-dockercfg-w2l68\""
Sep 08 15:57:18.783096 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.783078    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-173f5911-7dd4dc9566-6gxf8"]
Sep 08 15:57:18.787043 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.787023    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"]
Sep 08 15:57:18.814234 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.814211    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"]
Sep 08 15:57:18.820097 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.820079    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-173f5911-75fb8f5c7d-mqq4z"]
Sep 08 15:57:18.830134 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.830114    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"]
Sep 08 15:57:18.833827 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.833809    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-173f5911-7d5cd78d55-q8pqx"]
Sep 08 15:57:18.917365 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.917337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-workload-certs\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:18.917507 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.917370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-workload-socket\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:18.917507 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.917413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istiod-ca-cert\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:18.917599 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.917506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-data\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:18.917599 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.917580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-podinfo\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:18.917665 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.917621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-credential-socket\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:18.917665 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.917654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-envoy\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:18.917726 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.917677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc44s\" (UniqueName: \"kubernetes.io/projected/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-kube-api-access-xc44s\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:18.917761 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:18.917733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-token\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.018739 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.018646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-workload-certs\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.018739 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.018694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-workload-socket\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.018739 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.018723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istiod-ca-cert\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.019037 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.018756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-data\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.019037 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.018807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-podinfo\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.019037 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.018833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-credential-socket\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.019037 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.018867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-envoy\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.019037 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.018901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc44s\" (UniqueName: \"kubernetes.io/projected/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-kube-api-access-xc44s\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.019037 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.018947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-token\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.019337 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.019116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-workload-certs\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.019337 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.019173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-workload-socket\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.019337 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.019252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-credential-socket\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.019337 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.019290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-data\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.019575 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.019512    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istiod-ca-cert\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.021177 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.021145    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-envoy\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.021286 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.021270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-podinfo\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.027095 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.027064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc44s\" (UniqueName: \"kubernetes.io/projected/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-kube-api-access-xc44s\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.027095 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.027083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-token\") pod \"e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") " pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.093128 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.093091    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:19.374105 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.374082    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"]
Sep 08 15:57:19.376067 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:57:19.376039    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdacaa10_146a_4bd0_85e1_8ddaa17fa1e5.slice/crio-739f0971b50864614a5e7c3635fe145e84b28f499a6449d9d9ebdf1b69183a0a WatchSource:0}: Error finding container 739f0971b50864614a5e7c3635fe145e84b28f499a6449d9d9ebdf1b69183a0a: Status 404 returned error can't find the container with id 739f0971b50864614a5e7c3635fe145e84b28f499a6449d9d9ebdf1b69183a0a
Sep 08 15:57:19.378417 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.378373    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:57:19.378489 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.378459    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:57:19.378524 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.378489    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:57:19.752154 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.752123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp" event={"ID":"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5","Type":"ContainerStarted","Data":"0f632b8774aac3ddfd5d3403d58d97de34d3c0ad156cc518c37b677f4c0718ba"}
Sep 08 15:57:19.752593 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.752172    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp" event={"ID":"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5","Type":"ContainerStarted","Data":"739f0971b50864614a5e7c3635fe145e84b28f499a6449d9d9ebdf1b69183a0a"}
Sep 08 15:57:19.774865 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.774794    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp" podStartSLOduration=1.774774654 podStartE2EDuration="1.774774654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:57:19.769752138 +0000 UTC m=+786.393444453" watchObservedRunningTime="2026-09-08 15:57:19.774774654 +0000 UTC m=+786.398466979"
Sep 08 15:57:19.900956 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.900923    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ae7609d-0cd7-40d4-a54e-de12237ba7af" path="/var/lib/kubelet/pods/2ae7609d-0cd7-40d4-a54e-de12237ba7af/volumes"
Sep 08 15:57:19.901288 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.901276    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="362192ce-4e65-4f6d-8643-2f7eefcdec32" path="/var/lib/kubelet/pods/362192ce-4e65-4f6d-8643-2f7eefcdec32/volumes"
Sep 08 15:57:19.901661 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:19.901648    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a999a96-e071-4847-9dd5-cabdaff5489b" path="/var/lib/kubelet/pods/7a999a96-e071-4847-9dd5-cabdaff5489b/volumes"
Sep 08 15:57:20.094088 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:20.094010    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:20.098963 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:20.098936    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:20.759105 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:20.759078    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:20.760085 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:20.760066    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:22.215120 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.215090    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"]
Sep 08 15:57:22.219408 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.219376    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.221956 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.221932    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ac3191f1-serving-cert\""
Sep 08 15:57:22.222669 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.222574    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:57:22.222669 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.222654    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:57:22.222849 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.222767    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmx7t\""
Sep 08 15:57:22.222849 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.222832    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:57:22.227740 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.227687    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"]
Sep 08 15:57:22.348459 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.348427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9334547a-3bc3-481e-8189-3a2418f32087-metrics-certs\") pod \"maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.348653 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.348470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9334547a-3bc3-481e-8189-3a2418f32087-maas-api-tls\") pod \"maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.348653 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.348513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9334547a-3bc3-481e-8189-3a2418f32087-openshift-service-ca\") pod \"maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.348653 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.348539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sphjz\" (UniqueName: \"kubernetes.io/projected/9334547a-3bc3-481e-8189-3a2418f32087-kube-api-access-sphjz\") pod \"maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.449788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.449757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9334547a-3bc3-481e-8189-3a2418f32087-metrics-certs\") pod \"maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.449948 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.449794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9334547a-3bc3-481e-8189-3a2418f32087-maas-api-tls\") pod \"maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.449948 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.449821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9334547a-3bc3-481e-8189-3a2418f32087-openshift-service-ca\") pod \"maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.449948 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.449868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sphjz\" (UniqueName: \"kubernetes.io/projected/9334547a-3bc3-481e-8189-3a2418f32087-kube-api-access-sphjz\") pod \"maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.450593 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.450564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9334547a-3bc3-481e-8189-3a2418f32087-openshift-service-ca\") pod \"maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.452199 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.452177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9334547a-3bc3-481e-8189-3a2418f32087-maas-api-tls\") pod \"maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.452314 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.452206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9334547a-3bc3-481e-8189-3a2418f32087-metrics-certs\") pod \"maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.457055 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.457032    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sphjz\" (UniqueName: \"kubernetes.io/projected/9334547a-3bc3-481e-8189-3a2418f32087-kube-api-access-sphjz\") pod \"maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.535819 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.535763    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:22.812124 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.812090    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"]
Sep 08 15:57:22.814264 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:57:22.814239    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9334547a_3bc3_481e_8189_3a2418f32087.slice/crio-9668444af9f2e3f2ca86b1a5bbd8b0b3d59467f29af0e8df3775c4b8ca7fdb01 WatchSource:0}: Error finding container 9668444af9f2e3f2ca86b1a5bbd8b0b3d59467f29af0e8df3775c4b8ca7fdb01: Status 404 returned error can't find the container with id 9668444af9f2e3f2ca86b1a5bbd8b0b3d59467f29af0e8df3775c4b8ca7fdb01
Sep 08 15:57:22.852280 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.852257    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"]
Sep 08 15:57:22.857073 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.857056    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"
Sep 08 15:57:22.859326 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.859305    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ac3191f1\""
Sep 08 15:57:22.859496 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.859378    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ac3191f1-dockercfg-5x9v7\""
Sep 08 15:57:22.866309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.866286    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"]
Sep 08 15:57:22.877472 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.877450    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"]
Sep 08 15:57:22.881022 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.881006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"
Sep 08 15:57:22.889812 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.889791    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"]
Sep 08 15:57:22.955813 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.955748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e2459f-192f-4d15-aa99-b29b4f03ac43-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9\" (UID: \"07e2459f-192f-4d15-aa99-b29b4f03ac43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"
Sep 08 15:57:22.955974 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.955883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr5wp\" (UniqueName: \"kubernetes.io/projected/07e2459f-192f-4d15-aa99-b29b4f03ac43-kube-api-access-gr5wp\") pod \"payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9\" (UID: \"07e2459f-192f-4d15-aa99-b29b4f03ac43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"
Sep 08 15:57:22.955974 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.955963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92314e6f-8a30-4099-813e-1a3f64c1b7c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2\" (UID: \"92314e6f-8a30-4099-813e-1a3f64c1b7c5\") " pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"
Sep 08 15:57:22.956090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:22.956033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzbjh\" (UniqueName: \"kubernetes.io/projected/92314e6f-8a30-4099-813e-1a3f64c1b7c5-kube-api-access-hzbjh\") pod \"payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2\" (UID: \"92314e6f-8a30-4099-813e-1a3f64c1b7c5\") " pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"
Sep 08 15:57:23.057385 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.057173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzbjh\" (UniqueName: \"kubernetes.io/projected/92314e6f-8a30-4099-813e-1a3f64c1b7c5-kube-api-access-hzbjh\") pod \"payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2\" (UID: \"92314e6f-8a30-4099-813e-1a3f64c1b7c5\") " pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"
Sep 08 15:57:23.057385 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.057241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e2459f-192f-4d15-aa99-b29b4f03ac43-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9\" (UID: \"07e2459f-192f-4d15-aa99-b29b4f03ac43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"
Sep 08 15:57:23.057385 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.057284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr5wp\" (UniqueName: \"kubernetes.io/projected/07e2459f-192f-4d15-aa99-b29b4f03ac43-kube-api-access-gr5wp\") pod \"payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9\" (UID: \"07e2459f-192f-4d15-aa99-b29b4f03ac43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"
Sep 08 15:57:23.057385 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.057329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92314e6f-8a30-4099-813e-1a3f64c1b7c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2\" (UID: \"92314e6f-8a30-4099-813e-1a3f64c1b7c5\") " pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"
Sep 08 15:57:23.058069 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.058048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e2459f-192f-4d15-aa99-b29b4f03ac43-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9\" (UID: \"07e2459f-192f-4d15-aa99-b29b4f03ac43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"
Sep 08 15:57:23.058127 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.058068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92314e6f-8a30-4099-813e-1a3f64c1b7c5-plugins-config-volume\") pod \"payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2\" (UID: \"92314e6f-8a30-4099-813e-1a3f64c1b7c5\") " pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"
Sep 08 15:57:23.066469 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.066448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzbjh\" (UniqueName: \"kubernetes.io/projected/92314e6f-8a30-4099-813e-1a3f64c1b7c5-kube-api-access-hzbjh\") pod \"payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2\" (UID: \"92314e6f-8a30-4099-813e-1a3f64c1b7c5\") " pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"
Sep 08 15:57:23.066548 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.066525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr5wp\" (UniqueName: \"kubernetes.io/projected/07e2459f-192f-4d15-aa99-b29b4f03ac43-kube-api-access-gr5wp\") pod \"payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9\" (UID: \"07e2459f-192f-4d15-aa99-b29b4f03ac43\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"
Sep 08 15:57:23.168491 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.168470    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"
Sep 08 15:57:23.192743 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.192714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"
Sep 08 15:57:23.317439 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.317372    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:57:23.440381 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.440356    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"]
Sep 08 15:57:23.455278 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:57:23.455252    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e2459f_192f_4d15_aa99_b29b4f03ac43.slice/crio-07b1fca3a4b34821b3b7d308d1b01256f2fec58b800ce09a342b6117ae7be08f WatchSource:0}: Error finding container 07b1fca3a4b34821b3b7d308d1b01256f2fec58b800ce09a342b6117ae7be08f: Status 404 returned error can't find the container with id 07b1fca3a4b34821b3b7d308d1b01256f2fec58b800ce09a342b6117ae7be08f
Sep 08 15:57:23.459985 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.459957    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-workload-certs\") pod \"937bccdb-c05e-469c-881b-c2bdebab7394\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") "
Sep 08 15:57:23.460073 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.459999    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzrdk\" (UniqueName: \"kubernetes.io/projected/937bccdb-c05e-469c-881b-c2bdebab7394-kube-api-access-kzrdk\") pod \"937bccdb-c05e-469c-881b-c2bdebab7394\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") "
Sep 08 15:57:23.460073 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460019    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-credential-socket\") pod \"937bccdb-c05e-469c-881b-c2bdebab7394\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") "
Sep 08 15:57:23.460073 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460049    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-workload-socket\") pod \"937bccdb-c05e-469c-881b-c2bdebab7394\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") "
Sep 08 15:57:23.460231 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460077    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937bccdb-c05e-469c-881b-c2bdebab7394-istio-token\") pod \"937bccdb-c05e-469c-881b-c2bdebab7394\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") "
Sep 08 15:57:23.460231 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460120    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-istio-envoy\") pod \"937bccdb-c05e-469c-881b-c2bdebab7394\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") "
Sep 08 15:57:23.460231 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460187    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937bccdb-c05e-469c-881b-c2bdebab7394-istio-podinfo\") pod \"937bccdb-c05e-469c-881b-c2bdebab7394\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") "
Sep 08 15:57:23.460414 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460253    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "937bccdb-c05e-469c-881b-c2bdebab7394" (UID: "937bccdb-c05e-469c-881b-c2bdebab7394"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:23.460414 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460327    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "937bccdb-c05e-469c-881b-c2bdebab7394" (UID: "937bccdb-c05e-469c-881b-c2bdebab7394"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:23.460536 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460470    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-istio-data\") pod \"937bccdb-c05e-469c-881b-c2bdebab7394\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") "
Sep 08 15:57:23.460590 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460540    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937bccdb-c05e-469c-881b-c2bdebab7394-istiod-ca-cert\") pod \"937bccdb-c05e-469c-881b-c2bdebab7394\" (UID: \"937bccdb-c05e-469c-881b-c2bdebab7394\") "
Sep 08 15:57:23.460648 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460613    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "937bccdb-c05e-469c-881b-c2bdebab7394" (UID: "937bccdb-c05e-469c-881b-c2bdebab7394"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:23.460843 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460793    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-istio-data" (OuterVolumeSpecName: "istio-data") pod "937bccdb-c05e-469c-881b-c2bdebab7394" (UID: "937bccdb-c05e-469c-881b-c2bdebab7394"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:23.460936 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460894    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-workload-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:23.460936 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460912    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-credential-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:23.460936 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460925    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-workload-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:23.461087 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.460939    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-istio-data\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:23.461144 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.461089    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937bccdb-c05e-469c-881b-c2bdebab7394-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "937bccdb-c05e-469c-881b-c2bdebab7394" (UID: "937bccdb-c05e-469c-881b-c2bdebab7394"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:57:23.462424 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.462373    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/937bccdb-c05e-469c-881b-c2bdebab7394-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "937bccdb-c05e-469c-881b-c2bdebab7394" (UID: "937bccdb-c05e-469c-881b-c2bdebab7394"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 15:57:23.462632 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.462613    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "937bccdb-c05e-469c-881b-c2bdebab7394" (UID: "937bccdb-c05e-469c-881b-c2bdebab7394"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:23.462699 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.462627    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937bccdb-c05e-469c-881b-c2bdebab7394-kube-api-access-kzrdk" (OuterVolumeSpecName: "kube-api-access-kzrdk") pod "937bccdb-c05e-469c-881b-c2bdebab7394" (UID: "937bccdb-c05e-469c-881b-c2bdebab7394"). InnerVolumeSpecName "kube-api-access-kzrdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:23.463107 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.463084    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937bccdb-c05e-469c-881b-c2bdebab7394-istio-token" (OuterVolumeSpecName: "istio-token") pod "937bccdb-c05e-469c-881b-c2bdebab7394" (UID: "937bccdb-c05e-469c-881b-c2bdebab7394"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:23.475052 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.475027    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"]
Sep 08 15:57:23.476854 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:57:23.476832    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92314e6f_8a30_4099_813e_1a3f64c1b7c5.slice/crio-ff805cc2751aaf25889bbba242ddfcdf754253a2e5a13914ae1ac5d8e7732cc2 WatchSource:0}: Error finding container ff805cc2751aaf25889bbba242ddfcdf754253a2e5a13914ae1ac5d8e7732cc2: Status 404 returned error can't find the container with id ff805cc2751aaf25889bbba242ddfcdf754253a2e5a13914ae1ac5d8e7732cc2
Sep 08 15:57:23.562371 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.562342    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzrdk\" (UniqueName: \"kubernetes.io/projected/937bccdb-c05e-469c-881b-c2bdebab7394-kube-api-access-kzrdk\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:23.562371 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.562372    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937bccdb-c05e-469c-881b-c2bdebab7394-istio-token\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:23.562525 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.562386    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937bccdb-c05e-469c-881b-c2bdebab7394-istio-envoy\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:23.562525 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.562426    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937bccdb-c05e-469c-881b-c2bdebab7394-istio-podinfo\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:23.562525 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.562438    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937bccdb-c05e-469c-881b-c2bdebab7394-istiod-ca-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:23.775603 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.775572    2581 generic.go:356] "Generic (PLEG): container finished" podID="937bccdb-c05e-469c-881b-c2bdebab7394" containerID="dd7ac1a07fbc3a939af1cd0be7828a1e2efa37bd6a30ca1945a4a07f06c6be87" exitCode=0
Sep 08 15:57:23.775800 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.775621    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"
Sep 08 15:57:23.775800 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.775659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l" event={"ID":"937bccdb-c05e-469c-881b-c2bdebab7394","Type":"ContainerDied","Data":"dd7ac1a07fbc3a939af1cd0be7828a1e2efa37bd6a30ca1945a4a07f06c6be87"}
Sep 08 15:57:23.775800 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.775716    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l" event={"ID":"937bccdb-c05e-469c-881b-c2bdebab7394","Type":"ContainerDied","Data":"a550fb96c4dd2d04689f42bfc50e441ecd4cef7d946f56e71bba39052bbbcc1d"}
Sep 08 15:57:23.775800 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.775748    2581 scope.go:117] "RemoveContainer" containerID="dd7ac1a07fbc3a939af1cd0be7828a1e2efa37bd6a30ca1945a4a07f06c6be87"
Sep 08 15:57:23.778064 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.777650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2" event={"ID":"92314e6f-8a30-4099-813e-1a3f64c1b7c5","Type":"ContainerStarted","Data":"c20146420df893f2ebfbb7864475fd8b3edaac03318670c4d65135d6d48fd209"}
Sep 08 15:57:23.778064 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.777705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2" event={"ID":"92314e6f-8a30-4099-813e-1a3f64c1b7c5","Type":"ContainerStarted","Data":"ff805cc2751aaf25889bbba242ddfcdf754253a2e5a13914ae1ac5d8e7732cc2"}
Sep 08 15:57:23.778064 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.777749    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"
Sep 08 15:57:23.779555 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.779528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq" event={"ID":"9334547a-3bc3-481e-8189-3a2418f32087","Type":"ContainerStarted","Data":"5b64d2d68909359d40b380e5142413e15cf47122c0f8ae15f5b70e6da8c77d0b"}
Sep 08 15:57:23.779671 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.779577    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq" event={"ID":"9334547a-3bc3-481e-8189-3a2418f32087","Type":"ContainerStarted","Data":"9668444af9f2e3f2ca86b1a5bbd8b0b3d59467f29af0e8df3775c4b8ca7fdb01"}
Sep 08 15:57:23.779821 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.779804    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:23.781493 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.781470    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9" event={"ID":"07e2459f-192f-4d15-aa99-b29b4f03ac43","Type":"ContainerStarted","Data":"30e50e66e131968fda572190155264e62ef661ed6f11f6610a8328927158f335"}
Sep 08 15:57:23.781614 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.781513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9" event={"ID":"07e2459f-192f-4d15-aa99-b29b4f03ac43","Type":"ContainerStarted","Data":"07b1fca3a4b34821b3b7d308d1b01256f2fec58b800ce09a342b6117ae7be08f"}
Sep 08 15:57:23.781614 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.781548    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"
Sep 08 15:57:23.787005 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.786987    2581 scope.go:117] "RemoveContainer" containerID="eb1dcab5d0024668bb08d46500c45c73577c94d3704d973d95c5edcc69070dc5"
Sep 08 15:57:23.794988 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.794973    2581 scope.go:117] "RemoveContainer" containerID="dd7ac1a07fbc3a939af1cd0be7828a1e2efa37bd6a30ca1945a4a07f06c6be87"
Sep 08 15:57:23.795229 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:57:23.795213    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd7ac1a07fbc3a939af1cd0be7828a1e2efa37bd6a30ca1945a4a07f06c6be87\": container with ID starting with dd7ac1a07fbc3a939af1cd0be7828a1e2efa37bd6a30ca1945a4a07f06c6be87 not found: ID does not exist" containerID="dd7ac1a07fbc3a939af1cd0be7828a1e2efa37bd6a30ca1945a4a07f06c6be87"
Sep 08 15:57:23.795293 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.795236    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd7ac1a07fbc3a939af1cd0be7828a1e2efa37bd6a30ca1945a4a07f06c6be87"} err="failed to get container status \"dd7ac1a07fbc3a939af1cd0be7828a1e2efa37bd6a30ca1945a4a07f06c6be87\": rpc error: code = NotFound desc = could not find container \"dd7ac1a07fbc3a939af1cd0be7828a1e2efa37bd6a30ca1945a4a07f06c6be87\": container with ID starting with dd7ac1a07fbc3a939af1cd0be7828a1e2efa37bd6a30ca1945a4a07f06c6be87 not found: ID does not exist"
Sep 08 15:57:23.795293 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.795251    2581 scope.go:117] "RemoveContainer" containerID="eb1dcab5d0024668bb08d46500c45c73577c94d3704d973d95c5edcc69070dc5"
Sep 08 15:57:23.795561 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:57:23.795543    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1dcab5d0024668bb08d46500c45c73577c94d3704d973d95c5edcc69070dc5\": container with ID starting with eb1dcab5d0024668bb08d46500c45c73577c94d3704d973d95c5edcc69070dc5 not found: ID does not exist" containerID="eb1dcab5d0024668bb08d46500c45c73577c94d3704d973d95c5edcc69070dc5"
Sep 08 15:57:23.795625 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.795565    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1dcab5d0024668bb08d46500c45c73577c94d3704d973d95c5edcc69070dc5"} err="failed to get container status \"eb1dcab5d0024668bb08d46500c45c73577c94d3704d973d95c5edcc69070dc5\": rpc error: code = NotFound desc = could not find container \"eb1dcab5d0024668bb08d46500c45c73577c94d3704d973d95c5edcc69070dc5\": container with ID starting with eb1dcab5d0024668bb08d46500c45c73577c94d3704d973d95c5edcc69070dc5 not found: ID does not exist"
Sep 08 15:57:23.796003 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.795963    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2" podStartSLOduration=1.7959469160000001 podStartE2EDuration="1.795946916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:57:23.795847785 +0000 UTC m=+790.419540110" watchObservedRunningTime="2026-09-08 15:57:23.795946916 +0000 UTC m=+790.419639242"
Sep 08 15:57:23.814719 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.814684    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9" podStartSLOduration=1.8146757409999998 podStartE2EDuration="1.814675741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:57:23.81287404 +0000 UTC m=+790.436566363" watchObservedRunningTime="2026-09-08 15:57:23.814675741 +0000 UTC m=+790.438368064"
Sep 08 15:57:23.837228 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.837192    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq" podStartSLOduration=1.229701124 podStartE2EDuration="1.8371807s" totalImagesPullingTime="607.479576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:57:23.836681071 +0000 UTC m=+790.460373396" watchObservedRunningTime="2026-09-08 15:57:23.8371807 +0000 UTC m=+790.460873023"
Sep 08 15:57:23.856879 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.856846    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"]
Sep 08 15:57:23.865177 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.865150    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-173f5911-gw-openshift-default-7745c9d548-9hr7l"]
Sep 08 15:57:23.902254 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:23.902227    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="937bccdb-c05e-469c-881b-c2bdebab7394" path="/var/lib/kubelet/pods/937bccdb-c05e-469c-881b-c2bdebab7394/volumes"
Sep 08 15:57:29.797378 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:29.797348    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:34.791522 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:34.791487    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"
Sep 08 15:57:34.792011 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:34.791580    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"
Sep 08 15:57:37.468444 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.468343    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"]
Sep 08 15:57:37.468837 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.468573    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j" podUID="87aced9a-cfcd-48b7-9d40-f3eaa33fecfd" containerName="payload-processing" containerID="cri-o://5cd34d5136e1653f52bd1eaaed593275436688a533359c540205d563f92b48e1" gracePeriod=30
Sep 08 15:57:37.496135 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.490149    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"]
Sep 08 15:57:37.496135 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.495613    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2" podUID="1dc5266c-d85c-4b1a-9351-61f82c428fc5" containerName="payload-pre-processing" containerID="cri-o://d5b865f26f723ab6191b3fe739fdd425a524415b8e12fb7f61839ac66423d499" gracePeriod=30
Sep 08 15:57:37.804622 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.804587    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"
Sep 08 15:57:37.814154 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.814129    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87aced9a-cfcd-48b7-9d40-f3eaa33fecfd-plugins-config-volume\") pod \"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd\" (UID: \"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd\") "
Sep 08 15:57:37.814274 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.814173    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9bzt\" (UniqueName: \"kubernetes.io/projected/87aced9a-cfcd-48b7-9d40-f3eaa33fecfd-kube-api-access-p9bzt\") pod \"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd\" (UID: \"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd\") "
Sep 08 15:57:37.814575 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.814550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87aced9a-cfcd-48b7-9d40-f3eaa33fecfd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87aced9a-cfcd-48b7-9d40-f3eaa33fecfd" (UID: "87aced9a-cfcd-48b7-9d40-f3eaa33fecfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:57:37.816306 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.816281    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87aced9a-cfcd-48b7-9d40-f3eaa33fecfd-kube-api-access-p9bzt" (OuterVolumeSpecName: "kube-api-access-p9bzt") pod "87aced9a-cfcd-48b7-9d40-f3eaa33fecfd" (UID: "87aced9a-cfcd-48b7-9d40-f3eaa33fecfd"). InnerVolumeSpecName "kube-api-access-p9bzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:37.836879 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.836859    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"
Sep 08 15:57:37.860634 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.860609    2581 generic.go:356] "Generic (PLEG): container finished" podID="87aced9a-cfcd-48b7-9d40-f3eaa33fecfd" containerID="5cd34d5136e1653f52bd1eaaed593275436688a533359c540205d563f92b48e1" exitCode=0
Sep 08 15:57:37.860761 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.860642    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"
Sep 08 15:57:37.860761 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.860691    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j" event={"ID":"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd","Type":"ContainerDied","Data":"5cd34d5136e1653f52bd1eaaed593275436688a533359c540205d563f92b48e1"}
Sep 08 15:57:37.860761 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.860733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j" event={"ID":"87aced9a-cfcd-48b7-9d40-f3eaa33fecfd","Type":"ContainerDied","Data":"e3ec666ffb4a37174f7842e141e86a7ba837977ba38a454337221c967f125930"}
Sep 08 15:57:37.860761 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.860756    2581 scope.go:117] "RemoveContainer" containerID="5cd34d5136e1653f52bd1eaaed593275436688a533359c540205d563f92b48e1"
Sep 08 15:57:37.862383 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.862349    2581 generic.go:356] "Generic (PLEG): container finished" podID="1dc5266c-d85c-4b1a-9351-61f82c428fc5" containerID="d5b865f26f723ab6191b3fe739fdd425a524415b8e12fb7f61839ac66423d499" exitCode=0
Sep 08 15:57:37.862500 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.862422    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2" event={"ID":"1dc5266c-d85c-4b1a-9351-61f82c428fc5","Type":"ContainerDied","Data":"d5b865f26f723ab6191b3fe739fdd425a524415b8e12fb7f61839ac66423d499"}
Sep 08 15:57:37.862500 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.862468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2" event={"ID":"1dc5266c-d85c-4b1a-9351-61f82c428fc5","Type":"ContainerDied","Data":"76c755b7d7c3548d0c59d99983e2055023927647390a169debcc6efb637e690d"}
Sep 08 15:57:37.862500 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.862424    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"
Sep 08 15:57:37.890788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.890529    2581 scope.go:117] "RemoveContainer" containerID="5cd34d5136e1653f52bd1eaaed593275436688a533359c540205d563f92b48e1"
Sep 08 15:57:37.891439 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:57:37.891266    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd34d5136e1653f52bd1eaaed593275436688a533359c540205d563f92b48e1\": container with ID starting with 5cd34d5136e1653f52bd1eaaed593275436688a533359c540205d563f92b48e1 not found: ID does not exist" containerID="5cd34d5136e1653f52bd1eaaed593275436688a533359c540205d563f92b48e1"
Sep 08 15:57:37.891439 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.891305    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd34d5136e1653f52bd1eaaed593275436688a533359c540205d563f92b48e1"} err="failed to get container status \"5cd34d5136e1653f52bd1eaaed593275436688a533359c540205d563f92b48e1\": rpc error: code = NotFound desc = could not find container \"5cd34d5136e1653f52bd1eaaed593275436688a533359c540205d563f92b48e1\": container with ID starting with 5cd34d5136e1653f52bd1eaaed593275436688a533359c540205d563f92b48e1 not found: ID does not exist"
Sep 08 15:57:37.891439 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.891329    2581 scope.go:117] "RemoveContainer" containerID="d5b865f26f723ab6191b3fe739fdd425a524415b8e12fb7f61839ac66423d499"
Sep 08 15:57:37.904013 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.903996    2581 scope.go:117] "RemoveContainer" containerID="d5b865f26f723ab6191b3fe739fdd425a524415b8e12fb7f61839ac66423d499"
Sep 08 15:57:37.904913 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:57:37.904830    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b865f26f723ab6191b3fe739fdd425a524415b8e12fb7f61839ac66423d499\": container with ID starting with d5b865f26f723ab6191b3fe739fdd425a524415b8e12fb7f61839ac66423d499 not found: ID does not exist" containerID="d5b865f26f723ab6191b3fe739fdd425a524415b8e12fb7f61839ac66423d499"
Sep 08 15:57:37.904913 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.904862    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b865f26f723ab6191b3fe739fdd425a524415b8e12fb7f61839ac66423d499"} err="failed to get container status \"d5b865f26f723ab6191b3fe739fdd425a524415b8e12fb7f61839ac66423d499\": rpc error: code = NotFound desc = could not find container \"d5b865f26f723ab6191b3fe739fdd425a524415b8e12fb7f61839ac66423d499\": container with ID starting with d5b865f26f723ab6191b3fe739fdd425a524415b8e12fb7f61839ac66423d499 not found: ID does not exist"
Sep 08 15:57:37.907261 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.907242    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"]
Sep 08 15:57:37.909072 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.909054    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-423c90-57f9d755b7-qxt5j"]
Sep 08 15:57:37.914938 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.914923    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc5266c-d85c-4b1a-9351-61f82c428fc5-plugins-config-volume\") pod \"1dc5266c-d85c-4b1a-9351-61f82c428fc5\" (UID: \"1dc5266c-d85c-4b1a-9351-61f82c428fc5\") "
Sep 08 15:57:37.915026 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.914955    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krl8v\" (UniqueName: \"kubernetes.io/projected/1dc5266c-d85c-4b1a-9351-61f82c428fc5-kube-api-access-krl8v\") pod \"1dc5266c-d85c-4b1a-9351-61f82c428fc5\" (UID: \"1dc5266c-d85c-4b1a-9351-61f82c428fc5\") "
Sep 08 15:57:37.915239 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.915221    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87aced9a-cfcd-48b7-9d40-f3eaa33fecfd-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:37.915278 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.915245    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9bzt\" (UniqueName: \"kubernetes.io/projected/87aced9a-cfcd-48b7-9d40-f3eaa33fecfd-kube-api-access-p9bzt\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:37.915278 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.915243    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc5266c-d85c-4b1a-9351-61f82c428fc5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dc5266c-d85c-4b1a-9351-61f82c428fc5" (UID: "1dc5266c-d85c-4b1a-9351-61f82c428fc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:57:37.916890 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:37.916872    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc5266c-d85c-4b1a-9351-61f82c428fc5-kube-api-access-krl8v" (OuterVolumeSpecName: "kube-api-access-krl8v") pod "1dc5266c-d85c-4b1a-9351-61f82c428fc5" (UID: "1dc5266c-d85c-4b1a-9351-61f82c428fc5"). InnerVolumeSpecName "kube-api-access-krl8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:38.015961 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:38.015884    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc5266c-d85c-4b1a-9351-61f82c428fc5-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:38.015961 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:38.015920    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krl8v\" (UniqueName: \"kubernetes.io/projected/1dc5266c-d85c-4b1a-9351-61f82c428fc5-kube-api-access-krl8v\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:38.186779 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:38.186738    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"]
Sep 08 15:57:38.192801 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:38.192771    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-423c90-7d6cc9487d-n29t2"]
Sep 08 15:57:39.901292 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:39.901261    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dc5266c-d85c-4b1a-9351-61f82c428fc5" path="/var/lib/kubelet/pods/1dc5266c-d85c-4b1a-9351-61f82c428fc5/volumes"
Sep 08 15:57:39.901751 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:39.901732    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87aced9a-cfcd-48b7-9d40-f3eaa33fecfd" path="/var/lib/kubelet/pods/87aced9a-cfcd-48b7-9d40-f3eaa33fecfd/volumes"
Sep 08 15:57:42.860310 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.860279    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"]
Sep 08 15:57:42.860711 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.860650    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc5266c-d85c-4b1a-9351-61f82c428fc5" containerName="payload-pre-processing"
Sep 08 15:57:42.860711 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.860661    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc5266c-d85c-4b1a-9351-61f82c428fc5" containerName="payload-pre-processing"
Sep 08 15:57:42.860711 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.860679    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937bccdb-c05e-469c-881b-c2bdebab7394" containerName="istio-proxy"
Sep 08 15:57:42.860711 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.860684    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="937bccdb-c05e-469c-881b-c2bdebab7394" containerName="istio-proxy"
Sep 08 15:57:42.860711 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.860695    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87aced9a-cfcd-48b7-9d40-f3eaa33fecfd" containerName="payload-processing"
Sep 08 15:57:42.860711 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.860700    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="87aced9a-cfcd-48b7-9d40-f3eaa33fecfd" containerName="payload-processing"
Sep 08 15:57:42.860711 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.860706    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937bccdb-c05e-469c-881b-c2bdebab7394" containerName="istio-proxy"
Sep 08 15:57:42.860711 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.860712    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="937bccdb-c05e-469c-881b-c2bdebab7394" containerName="istio-proxy"
Sep 08 15:57:42.860964 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.860791    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="937bccdb-c05e-469c-881b-c2bdebab7394" containerName="istio-proxy"
Sep 08 15:57:42.860964 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.860800    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc5266c-d85c-4b1a-9351-61f82c428fc5" containerName="payload-pre-processing"
Sep 08 15:57:42.860964 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.860808    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="87aced9a-cfcd-48b7-9d40-f3eaa33fecfd" containerName="payload-processing"
Sep 08 15:57:42.863816 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.863799    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:42.866015 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.865996    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-e6f19b8d-kserve-self-signed-certs\""
Sep 08 15:57:42.870597 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.870571    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"]
Sep 08 15:57:42.894187 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.894165    2581 generic.go:356] "Generic (PLEG): container finished" podID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerID="0f632b8774aac3ddfd5d3403d58d97de34d3c0ad156cc518c37b677f4c0718ba" exitCode=0
Sep 08 15:57:42.894302 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.894240    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp" event={"ID":"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5","Type":"ContainerDied","Data":"0f632b8774aac3ddfd5d3403d58d97de34d3c0ad156cc518c37b677f4c0718ba"}
Sep 08 15:57:42.894628 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.894612    2581 scope.go:117] "RemoveContainer" containerID="0f632b8774aac3ddfd5d3403d58d97de34d3c0ad156cc518c37b677f4c0718ba"
Sep 08 15:57:42.895604 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.895579    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:57:42.895678 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.895643    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:57:42.895723 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.895679    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:57:42.953510 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.953485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-home\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:42.953640 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.953527    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-tmp-dir\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:42.953640 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.953629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c54675-ed87-4e97-a221-4e7fa3bb527f-tls-certs\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:42.953772 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.953755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-dshm\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:42.953832 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.953813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-model-cache\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:42.953889 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:42.953839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9gdn\" (UniqueName: \"kubernetes.io/projected/74c54675-ed87-4e97-a221-4e7fa3bb527f-kube-api-access-n9gdn\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.054497 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.054456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-home\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.054668 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.054536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-tmp-dir\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.054668 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.054596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c54675-ed87-4e97-a221-4e7fa3bb527f-tls-certs\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.054790 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.054679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-dshm\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.054790 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.054724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-model-cache\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.054790 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.054753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9gdn\" (UniqueName: \"kubernetes.io/projected/74c54675-ed87-4e97-a221-4e7fa3bb527f-kube-api-access-n9gdn\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.054964 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.054870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-home\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.054964 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.054895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-tmp-dir\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.055129 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.055107    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-model-cache\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.057242 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.057219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c54675-ed87-4e97-a221-4e7fa3bb527f-tls-certs\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.057322 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.057224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-dshm\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.061983 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.061959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9gdn\" (UniqueName: \"kubernetes.io/projected/74c54675-ed87-4e97-a221-4e7fa3bb527f-kube-api-access-n9gdn\") pod \"e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") " pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.173876 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.173847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:43.451385 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.451350    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"]
Sep 08 15:57:43.451908 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:57:43.451882    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74c54675_ed87_4e97_a221_4e7fa3bb527f.slice/crio-3fe375f60cf6b558f49640bf53fdb0b6f37a3c03e50fcf3ec8aa4e86e909b45a WatchSource:0}: Error finding container 3fe375f60cf6b558f49640bf53fdb0b6f37a3c03e50fcf3ec8aa4e86e909b45a: Status 404 returned error can't find the container with id 3fe375f60cf6b558f49640bf53fdb0b6f37a3c03e50fcf3ec8aa4e86e909b45a
Sep 08 15:57:43.902971 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.902932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf" event={"ID":"74c54675-ed87-4e97-a221-4e7fa3bb527f","Type":"ContainerStarted","Data":"3d957875e5df0a18ba41bd931f1c3f7070fc1eca27e09db38e45e053d86cff3b"}
Sep 08 15:57:43.903704 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.902987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf" event={"ID":"74c54675-ed87-4e97-a221-4e7fa3bb527f","Type":"ContainerStarted","Data":"3fe375f60cf6b558f49640bf53fdb0b6f37a3c03e50fcf3ec8aa4e86e909b45a"}
Sep 08 15:57:43.904209 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.904185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp" event={"ID":"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5","Type":"ContainerStarted","Data":"65aa25b6e5ffc3982adc918312e940454e6d47885a2883fe4d7fa5c1d49aaa0a"}
Sep 08 15:57:43.917427 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:43.917365    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf" podStartSLOduration=1.917352203 podStartE2EDuration="1.917352203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:57:43.916555962 +0000 UTC m=+810.540248315" watchObservedRunningTime="2026-09-08 15:57:43.917352203 +0000 UTC m=+810.541044527"
Sep 08 15:57:44.093570 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:44.093530    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:44.093980 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:44.093949    2581 patch_prober.go:28] interesting pod/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 08 15:57:44.094072 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:44.094016    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp" podUID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 08 15:57:45.093542 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:45.093501    2581 patch_prober.go:28] interesting pod/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 08 15:57:45.094044 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:45.093576    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp" podUID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 08 15:57:46.093858 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:46.093813    2581 patch_prober.go:28] interesting pod/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 08 15:57:46.094283 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:46.093903    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp" podUID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 08 15:57:47.094512 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:47.094469    2581 patch_prober.go:28] interesting pod/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 08 15:57:47.094883 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:47.094530    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp" podUID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 08 15:57:48.094438 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:48.094369    2581 patch_prober.go:28] interesting pod/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 08 15:57:48.094644 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:48.094457    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp" podUID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 08 15:57:49.228003 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:49.227969    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:49.228516 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:49.228427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:49.229308 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:49.229288    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:52.799679 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:52.799642    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"]
Sep 08 15:57:52.800268 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:52.799889    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq" podUID="9334547a-3bc3-481e-8189-3a2418f32087" containerName="maas-api" containerID="cri-o://5b64d2d68909359d40b380e5142413e15cf47122c0f8ae15f5b70e6da8c77d0b" gracePeriod=30
Sep 08 15:57:52.856850 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:52.856817    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"]
Sep 08 15:57:52.857089 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:52.857040    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2" podUID="92314e6f-8a30-4099-813e-1a3f64c1b7c5" containerName="payload-processing" containerID="cri-o://c20146420df893f2ebfbb7864475fd8b3edaac03318670c4d65135d6d48fd209" gracePeriod=30
Sep 08 15:57:52.867096 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:52.867067    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"]
Sep 08 15:57:52.867370 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:52.867324    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9" podUID="07e2459f-192f-4d15-aa99-b29b4f03ac43" containerName="payload-pre-processing" containerID="cri-o://30e50e66e131968fda572190155264e62ef661ed6f11f6610a8328927158f335" gracePeriod=30
Sep 08 15:57:52.954467 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:52.954439    2581 generic.go:356] "Generic (PLEG): container finished" podID="9334547a-3bc3-481e-8189-3a2418f32087" containerID="5b64d2d68909359d40b380e5142413e15cf47122c0f8ae15f5b70e6da8c77d0b" exitCode=0
Sep 08 15:57:52.954581 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:52.954512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq" event={"ID":"9334547a-3bc3-481e-8189-3a2418f32087","Type":"ContainerDied","Data":"5b64d2d68909359d40b380e5142413e15cf47122c0f8ae15f5b70e6da8c77d0b"}
Sep 08 15:57:53.126940 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.126892    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"
Sep 08 15:57:53.143838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.143816    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:53.153150 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.153128    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzbjh\" (UniqueName: \"kubernetes.io/projected/92314e6f-8a30-4099-813e-1a3f64c1b7c5-kube-api-access-hzbjh\") pod \"92314e6f-8a30-4099-813e-1a3f64c1b7c5\" (UID: \"92314e6f-8a30-4099-813e-1a3f64c1b7c5\") "
Sep 08 15:57:53.153275 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.153168    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92314e6f-8a30-4099-813e-1a3f64c1b7c5-plugins-config-volume\") pod \"92314e6f-8a30-4099-813e-1a3f64c1b7c5\" (UID: \"92314e6f-8a30-4099-813e-1a3f64c1b7c5\") "
Sep 08 15:57:53.153788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.153760    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92314e6f-8a30-4099-813e-1a3f64c1b7c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "92314e6f-8a30-4099-813e-1a3f64c1b7c5" (UID: "92314e6f-8a30-4099-813e-1a3f64c1b7c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:57:53.155535 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.155513    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92314e6f-8a30-4099-813e-1a3f64c1b7c5-kube-api-access-hzbjh" (OuterVolumeSpecName: "kube-api-access-hzbjh") pod "92314e6f-8a30-4099-813e-1a3f64c1b7c5" (UID: "92314e6f-8a30-4099-813e-1a3f64c1b7c5"). InnerVolumeSpecName "kube-api-access-hzbjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:53.164417 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.164383    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"
Sep 08 15:57:53.174711 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.174689    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:53.174819 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.174730    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:53.188009 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.187990    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:53.254467 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.254445    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e2459f-192f-4d15-aa99-b29b4f03ac43-plugins-config-volume\") pod \"07e2459f-192f-4d15-aa99-b29b4f03ac43\" (UID: \"07e2459f-192f-4d15-aa99-b29b4f03ac43\") "
Sep 08 15:57:53.254589 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.254477    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sphjz\" (UniqueName: \"kubernetes.io/projected/9334547a-3bc3-481e-8189-3a2418f32087-kube-api-access-sphjz\") pod \"9334547a-3bc3-481e-8189-3a2418f32087\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") "
Sep 08 15:57:53.254589 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.254495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9334547a-3bc3-481e-8189-3a2418f32087-openshift-service-ca\") pod \"9334547a-3bc3-481e-8189-3a2418f32087\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") "
Sep 08 15:57:53.254589 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.254524    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9334547a-3bc3-481e-8189-3a2418f32087-maas-api-tls\") pod \"9334547a-3bc3-481e-8189-3a2418f32087\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") "
Sep 08 15:57:53.254756 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.254640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr5wp\" (UniqueName: \"kubernetes.io/projected/07e2459f-192f-4d15-aa99-b29b4f03ac43-kube-api-access-gr5wp\") pod \"07e2459f-192f-4d15-aa99-b29b4f03ac43\" (UID: \"07e2459f-192f-4d15-aa99-b29b4f03ac43\") "
Sep 08 15:57:53.254756 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.254674    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9334547a-3bc3-481e-8189-3a2418f32087-metrics-certs\") pod \"9334547a-3bc3-481e-8189-3a2418f32087\" (UID: \"9334547a-3bc3-481e-8189-3a2418f32087\") "
Sep 08 15:57:53.254900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.254873    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9334547a-3bc3-481e-8189-3a2418f32087-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9334547a-3bc3-481e-8189-3a2418f32087" (UID: "9334547a-3bc3-481e-8189-3a2418f32087"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:57:53.254962 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.254900    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e2459f-192f-4d15-aa99-b29b4f03ac43-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07e2459f-192f-4d15-aa99-b29b4f03ac43" (UID: "07e2459f-192f-4d15-aa99-b29b4f03ac43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:57:53.255112 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.255093    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92314e6f-8a30-4099-813e-1a3f64c1b7c5-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:53.255182 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.255116    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e2459f-192f-4d15-aa99-b29b4f03ac43-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:53.255182 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.255132    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9334547a-3bc3-481e-8189-3a2418f32087-openshift-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:53.255182 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.255144    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzbjh\" (UniqueName: \"kubernetes.io/projected/92314e6f-8a30-4099-813e-1a3f64c1b7c5-kube-api-access-hzbjh\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:53.256769 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.256748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9334547a-3bc3-481e-8189-3a2418f32087-kube-api-access-sphjz" (OuterVolumeSpecName: "kube-api-access-sphjz") pod "9334547a-3bc3-481e-8189-3a2418f32087" (UID: "9334547a-3bc3-481e-8189-3a2418f32087"). InnerVolumeSpecName "kube-api-access-sphjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:53.256848 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.256776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e2459f-192f-4d15-aa99-b29b4f03ac43-kube-api-access-gr5wp" (OuterVolumeSpecName: "kube-api-access-gr5wp") pod "07e2459f-192f-4d15-aa99-b29b4f03ac43" (UID: "07e2459f-192f-4d15-aa99-b29b4f03ac43"). InnerVolumeSpecName "kube-api-access-gr5wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:53.257018 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.257003    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9334547a-3bc3-481e-8189-3a2418f32087-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9334547a-3bc3-481e-8189-3a2418f32087" (UID: "9334547a-3bc3-481e-8189-3a2418f32087"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:57:53.257065 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.257020    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9334547a-3bc3-481e-8189-3a2418f32087-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9334547a-3bc3-481e-8189-3a2418f32087" (UID: "9334547a-3bc3-481e-8189-3a2418f32087"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:57:53.355543 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.355479    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sphjz\" (UniqueName: \"kubernetes.io/projected/9334547a-3bc3-481e-8189-3a2418f32087-kube-api-access-sphjz\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:53.355543 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.355512    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9334547a-3bc3-481e-8189-3a2418f32087-maas-api-tls\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:53.355543 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.355521    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr5wp\" (UniqueName: \"kubernetes.io/projected/07e2459f-192f-4d15-aa99-b29b4f03ac43-kube-api-access-gr5wp\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:53.355543 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.355530    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9334547a-3bc3-481e-8189-3a2418f32087-metrics-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:53.961522 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.961496    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"
Sep 08 15:57:53.961908 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.961495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq" event={"ID":"9334547a-3bc3-481e-8189-3a2418f32087","Type":"ContainerDied","Data":"9668444af9f2e3f2ca86b1a5bbd8b0b3d59467f29af0e8df3775c4b8ca7fdb01"}
Sep 08 15:57:53.961908 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.961630    2581 scope.go:117] "RemoveContainer" containerID="5b64d2d68909359d40b380e5142413e15cf47122c0f8ae15f5b70e6da8c77d0b"
Sep 08 15:57:53.963133 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.963070    2581 generic.go:356] "Generic (PLEG): container finished" podID="07e2459f-192f-4d15-aa99-b29b4f03ac43" containerID="30e50e66e131968fda572190155264e62ef661ed6f11f6610a8328927158f335" exitCode=0
Sep 08 15:57:53.963133 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.963099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9" event={"ID":"07e2459f-192f-4d15-aa99-b29b4f03ac43","Type":"ContainerDied","Data":"30e50e66e131968fda572190155264e62ef661ed6f11f6610a8328927158f335"}
Sep 08 15:57:53.963133 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.963117    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"
Sep 08 15:57:53.963306 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.963144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9" event={"ID":"07e2459f-192f-4d15-aa99-b29b4f03ac43","Type":"ContainerDied","Data":"07b1fca3a4b34821b3b7d308d1b01256f2fec58b800ce09a342b6117ae7be08f"}
Sep 08 15:57:53.964726 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.964703    2581 generic.go:356] "Generic (PLEG): container finished" podID="92314e6f-8a30-4099-813e-1a3f64c1b7c5" containerID="c20146420df893f2ebfbb7864475fd8b3edaac03318670c4d65135d6d48fd209" exitCode=0
Sep 08 15:57:53.964833 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.964731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2" event={"ID":"92314e6f-8a30-4099-813e-1a3f64c1b7c5","Type":"ContainerDied","Data":"c20146420df893f2ebfbb7864475fd8b3edaac03318670c4d65135d6d48fd209"}
Sep 08 15:57:53.964833 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.964746    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"
Sep 08 15:57:53.964833 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.964781    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2" event={"ID":"92314e6f-8a30-4099-813e-1a3f64c1b7c5","Type":"ContainerDied","Data":"ff805cc2751aaf25889bbba242ddfcdf754253a2e5a13914ae1ac5d8e7732cc2"}
Sep 08 15:57:53.973215 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.973183    2581 scope.go:117] "RemoveContainer" containerID="30e50e66e131968fda572190155264e62ef661ed6f11f6610a8328927158f335"
Sep 08 15:57:53.978655 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.978636    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:57:53.981140 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.981121    2581 scope.go:117] "RemoveContainer" containerID="30e50e66e131968fda572190155264e62ef661ed6f11f6610a8328927158f335"
Sep 08 15:57:53.981405 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:57:53.981377    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e50e66e131968fda572190155264e62ef661ed6f11f6610a8328927158f335\": container with ID starting with 30e50e66e131968fda572190155264e62ef661ed6f11f6610a8328927158f335 not found: ID does not exist" containerID="30e50e66e131968fda572190155264e62ef661ed6f11f6610a8328927158f335"
Sep 08 15:57:53.981481 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.981414    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e50e66e131968fda572190155264e62ef661ed6f11f6610a8328927158f335"} err="failed to get container status \"30e50e66e131968fda572190155264e62ef661ed6f11f6610a8328927158f335\": rpc error: code = NotFound desc = could not find container \"30e50e66e131968fda572190155264e62ef661ed6f11f6610a8328927158f335\": container with ID starting with 30e50e66e131968fda572190155264e62ef661ed6f11f6610a8328927158f335 not found: ID does not exist"
Sep 08 15:57:53.981481 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.981432    2581 scope.go:117] "RemoveContainer" containerID="c20146420df893f2ebfbb7864475fd8b3edaac03318670c4d65135d6d48fd209"
Sep 08 15:57:53.985565 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.985543    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"]
Sep 08 15:57:53.989421 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.989400    2581 scope.go:117] "RemoveContainer" containerID="c20146420df893f2ebfbb7864475fd8b3edaac03318670c4d65135d6d48fd209"
Sep 08 15:57:53.989659 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.989641    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac3191f1-d98cf584f-bwhs2"]
Sep 08 15:57:53.989723 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:57:53.989678    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c20146420df893f2ebfbb7864475fd8b3edaac03318670c4d65135d6d48fd209\": container with ID starting with c20146420df893f2ebfbb7864475fd8b3edaac03318670c4d65135d6d48fd209 not found: ID does not exist" containerID="c20146420df893f2ebfbb7864475fd8b3edaac03318670c4d65135d6d48fd209"
Sep 08 15:57:53.989723 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.989700    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c20146420df893f2ebfbb7864475fd8b3edaac03318670c4d65135d6d48fd209"} err="failed to get container status \"c20146420df893f2ebfbb7864475fd8b3edaac03318670c4d65135d6d48fd209\": rpc error: code = NotFound desc = could not find container \"c20146420df893f2ebfbb7864475fd8b3edaac03318670c4d65135d6d48fd209\": container with ID starting with c20146420df893f2ebfbb7864475fd8b3edaac03318670c4d65135d6d48fd209 not found: ID does not exist"
Sep 08 15:57:53.999578 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:53.999557    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"]
Sep 08 15:57:54.003634 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.003615    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac3191f1-79bdbcf4b-fbdw9"]
Sep 08 15:57:54.012575 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.012552    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"]
Sep 08 15:57:54.018532 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.018514    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac3191f1-557cd6fff8-9wdcq"]
Sep 08 15:57:54.124340 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.124261    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"]
Sep 08 15:57:54.124530 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.124507    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp" podUID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerName="istio-proxy" containerID="cri-o://65aa25b6e5ffc3982adc918312e940454e6d47885a2883fe4d7fa5c1d49aaa0a" gracePeriod=30
Sep 08 15:57:54.846920 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.846873    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"]
Sep 08 15:57:54.847439 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.847423    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9334547a-3bc3-481e-8189-3a2418f32087" containerName="maas-api"
Sep 08 15:57:54.847500 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.847443    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9334547a-3bc3-481e-8189-3a2418f32087" containerName="maas-api"
Sep 08 15:57:54.847500 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.847465    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92314e6f-8a30-4099-813e-1a3f64c1b7c5" containerName="payload-processing"
Sep 08 15:57:54.847500 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.847475    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="92314e6f-8a30-4099-813e-1a3f64c1b7c5" containerName="payload-processing"
Sep 08 15:57:54.847500 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.847492    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e2459f-192f-4d15-aa99-b29b4f03ac43" containerName="payload-pre-processing"
Sep 08 15:57:54.847645 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.847501    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e2459f-192f-4d15-aa99-b29b4f03ac43" containerName="payload-pre-processing"
Sep 08 15:57:54.847645 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.847598    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e2459f-192f-4d15-aa99-b29b4f03ac43" containerName="payload-pre-processing"
Sep 08 15:57:54.847645 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.847612    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9334547a-3bc3-481e-8189-3a2418f32087" containerName="maas-api"
Sep 08 15:57:54.847645 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.847624    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="937bccdb-c05e-469c-881b-c2bdebab7394" containerName="istio-proxy"
Sep 08 15:57:54.847645 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.847633    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="92314e6f-8a30-4099-813e-1a3f64c1b7c5" containerName="payload-processing"
Sep 08 15:57:54.852329 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.852308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:54.855013 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.854995    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-9333348f-openshift-default-dockercfg-zjrh7\""
Sep 08 15:57:54.861864 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.861841    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"]
Sep 08 15:57:54.970378 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.970345    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-workload-certs\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:54.970828 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.970408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istiod-ca-cert\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:54.970828 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.970533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-credential-socket\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:54.970828 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.970597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-podinfo\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:54.970828 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.970646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-envoy\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:54.970828 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.970718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-token\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:54.970828 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.970772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-workload-socket\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:54.970828 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.970815    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fl4\" (UniqueName: \"kubernetes.io/projected/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-kube-api-access-j9fl4\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:54.971152 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:54.970880    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-data\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.072243 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.072206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-envoy\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.072470 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.072267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-token\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.072470 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.072304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-workload-socket\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.072470 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.072326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fl4\" (UniqueName: \"kubernetes.io/projected/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-kube-api-access-j9fl4\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.072470 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.072371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-data\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.072470 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.072433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-workload-certs\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.072821 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.072614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istiod-ca-cert\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.072821 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.072704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-workload-socket\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.072821 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.072754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-credential-socket\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.072821 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.072791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-podinfo\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.073019 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.072941    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-data\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.073154 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.073132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-workload-certs\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.073216 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.073168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-credential-socket\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.073586 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.073566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istiod-ca-cert\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.074695 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.074670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-envoy\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.075150 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.075132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-podinfo\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.079660 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.079637    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fl4\" (UniqueName: \"kubernetes.io/projected/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-kube-api-access-j9fl4\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.080309 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.080283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-token\") pod \"e2e-derive-9333348f-openshift-default-89d7555b8-q26kj\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") " pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.165296 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.165269    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:55.446789 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:57:55.446750    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f37dbe8_1802_4cf9_a9a9_f0e107ca2b0d.slice/crio-16c7c326c9bfc0445c7be23b215d559bfb06bcd79d552fa2f3b9d6c4221c11c2 WatchSource:0}: Error finding container 16c7c326c9bfc0445c7be23b215d559bfb06bcd79d552fa2f3b9d6c4221c11c2: Status 404 returned error can't find the container with id 16c7c326c9bfc0445c7be23b215d559bfb06bcd79d552fa2f3b9d6c4221c11c2
Sep 08 15:57:55.448613 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.448591    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"]
Sep 08 15:57:55.449105 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.449073    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:57:55.449186 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.449134    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:57:55.449186 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.449161    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:57:55.900620 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.900590    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e2459f-192f-4d15-aa99-b29b4f03ac43" path="/var/lib/kubelet/pods/07e2459f-192f-4d15-aa99-b29b4f03ac43/volumes"
Sep 08 15:57:55.900952 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.900939    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92314e6f-8a30-4099-813e-1a3f64c1b7c5" path="/var/lib/kubelet/pods/92314e6f-8a30-4099-813e-1a3f64c1b7c5/volumes"
Sep 08 15:57:55.901286 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.901274    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9334547a-3bc3-481e-8189-3a2418f32087" path="/var/lib/kubelet/pods/9334547a-3bc3-481e-8189-3a2418f32087/volumes"
Sep 08 15:57:55.979899 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.979873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" event={"ID":"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d","Type":"ContainerStarted","Data":"43447c7485cafebfdf554103f3ff09c555a60d71584a7c2d3648085e7b057102"}
Sep 08 15:57:55.980290 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:55.979915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" event={"ID":"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d","Type":"ContainerStarted","Data":"16c7c326c9bfc0445c7be23b215d559bfb06bcd79d552fa2f3b9d6c4221c11c2"}
Sep 08 15:57:56.002030 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:56.001987    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" podStartSLOduration=2.001970769 podStartE2EDuration="2.001970769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:57:55.999649329 +0000 UTC m=+822.623341673" watchObservedRunningTime="2026-09-08 15:57:56.001970769 +0000 UTC m=+822.625663095"
Sep 08 15:57:56.166366 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:56.166290    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:56.171458 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:56.171435    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:56.985743 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:56.985716    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:56.986702 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:56.986682    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:57:58.485598 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.485560    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"]
Sep 08 15:57:58.490917 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.490897    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:58.493220 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.493196    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:57:58.493344 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.493243    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9333348f-serving-cert\""
Sep 08 15:57:58.493426 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.493385    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:57:58.493861 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.493838    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmx7t\""
Sep 08 15:57:58.493968 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.493914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:57:58.498502 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.498481    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"]
Sep 08 15:57:58.602030 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.601998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2f1411-de19-4a95-9c25-e59c32572040-metrics-certs\") pod \"maas-api-e2e-derive-9333348f-64f5466bf5-qtldz\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:58.602174 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.602120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrl6x\" (UniqueName: \"kubernetes.io/projected/fe2f1411-de19-4a95-9c25-e59c32572040-kube-api-access-lrl6x\") pod \"maas-api-e2e-derive-9333348f-64f5466bf5-qtldz\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:58.602174 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.602148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe2f1411-de19-4a95-9c25-e59c32572040-maas-api-tls\") pod \"maas-api-e2e-derive-9333348f-64f5466bf5-qtldz\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:58.602174 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.602169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2f1411-de19-4a95-9c25-e59c32572040-openshift-service-ca\") pod \"maas-api-e2e-derive-9333348f-64f5466bf5-qtldz\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:58.703507 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.703482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrl6x\" (UniqueName: \"kubernetes.io/projected/fe2f1411-de19-4a95-9c25-e59c32572040-kube-api-access-lrl6x\") pod \"maas-api-e2e-derive-9333348f-64f5466bf5-qtldz\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:58.703650 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.703521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe2f1411-de19-4a95-9c25-e59c32572040-maas-api-tls\") pod \"maas-api-e2e-derive-9333348f-64f5466bf5-qtldz\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:58.703650 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.703542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2f1411-de19-4a95-9c25-e59c32572040-openshift-service-ca\") pod \"maas-api-e2e-derive-9333348f-64f5466bf5-qtldz\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:58.703650 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.703579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2f1411-de19-4a95-9c25-e59c32572040-metrics-certs\") pod \"maas-api-e2e-derive-9333348f-64f5466bf5-qtldz\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:58.704185 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.704163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2f1411-de19-4a95-9c25-e59c32572040-openshift-service-ca\") pod \"maas-api-e2e-derive-9333348f-64f5466bf5-qtldz\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:58.705971 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.705944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe2f1411-de19-4a95-9c25-e59c32572040-maas-api-tls\") pod \"maas-api-e2e-derive-9333348f-64f5466bf5-qtldz\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:58.706056 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.705975    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2f1411-de19-4a95-9c25-e59c32572040-metrics-certs\") pod \"maas-api-e2e-derive-9333348f-64f5466bf5-qtldz\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:58.711059 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.711040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrl6x\" (UniqueName: \"kubernetes.io/projected/fe2f1411-de19-4a95-9c25-e59c32572040-kube-api-access-lrl6x\") pod \"maas-api-e2e-derive-9333348f-64f5466bf5-qtldz\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:58.804647 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:58.804592    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:57:59.081640 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.081459    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"]
Sep 08 15:57:59.082925 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:57:59.082895    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2f1411_de19_4a95_9c25_e59c32572040.slice/crio-cb69db84cf327d61ad6d108fa6e8b1fc36da61e366a41c23cad01d70780d3556 WatchSource:0}: Error finding container cb69db84cf327d61ad6d108fa6e8b1fc36da61e366a41c23cad01d70780d3556: Status 404 returned error can't find the container with id cb69db84cf327d61ad6d108fa6e8b1fc36da61e366a41c23cad01d70780d3556
Sep 08 15:57:59.094831 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.094805    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"]
Sep 08 15:57:59.100978 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.100953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"
Sep 08 15:57:59.102992 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.102972    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9333348f\""
Sep 08 15:57:59.103094 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.103060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9333348f-dockercfg-6mh9q\""
Sep 08 15:57:59.110547 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.110524    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"]
Sep 08 15:57:59.119481 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.119457    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"]
Sep 08 15:57:59.123037 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.123021    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"
Sep 08 15:57:59.134941 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.134921    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"]
Sep 08 15:57:59.209307 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.209265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9714ac49-431c-4421-8c4d-206634c9eefe-plugins-config-volume\") pod \"payload-processing-e2e-derive-9333348f-7576fbffb9-264t6\" (UID: \"9714ac49-431c-4421-8c4d-206634c9eefe\") " pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"
Sep 08 15:57:59.209502 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.209441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knwps\" (UniqueName: \"kubernetes.io/projected/788e0a9f-b648-4ed4-88ac-fdf54d596016-kube-api-access-knwps\") pod \"payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg\" (UID: \"788e0a9f-b648-4ed4-88ac-fdf54d596016\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"
Sep 08 15:57:59.209502 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.209482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4zzd\" (UniqueName: \"kubernetes.io/projected/9714ac49-431c-4421-8c4d-206634c9eefe-kube-api-access-v4zzd\") pod \"payload-processing-e2e-derive-9333348f-7576fbffb9-264t6\" (UID: \"9714ac49-431c-4421-8c4d-206634c9eefe\") " pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"
Sep 08 15:57:59.209634 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.209508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788e0a9f-b648-4ed4-88ac-fdf54d596016-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg\" (UID: \"788e0a9f-b648-4ed4-88ac-fdf54d596016\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"
Sep 08 15:57:59.310258 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.310219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knwps\" (UniqueName: \"kubernetes.io/projected/788e0a9f-b648-4ed4-88ac-fdf54d596016-kube-api-access-knwps\") pod \"payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg\" (UID: \"788e0a9f-b648-4ed4-88ac-fdf54d596016\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"
Sep 08 15:57:59.310976 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.310289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4zzd\" (UniqueName: \"kubernetes.io/projected/9714ac49-431c-4421-8c4d-206634c9eefe-kube-api-access-v4zzd\") pod \"payload-processing-e2e-derive-9333348f-7576fbffb9-264t6\" (UID: \"9714ac49-431c-4421-8c4d-206634c9eefe\") " pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"
Sep 08 15:57:59.310976 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.310320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788e0a9f-b648-4ed4-88ac-fdf54d596016-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg\" (UID: \"788e0a9f-b648-4ed4-88ac-fdf54d596016\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"
Sep 08 15:57:59.310976 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.310371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9714ac49-431c-4421-8c4d-206634c9eefe-plugins-config-volume\") pod \"payload-processing-e2e-derive-9333348f-7576fbffb9-264t6\" (UID: \"9714ac49-431c-4421-8c4d-206634c9eefe\") " pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"
Sep 08 15:57:59.311429 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.311383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9714ac49-431c-4421-8c4d-206634c9eefe-plugins-config-volume\") pod \"payload-processing-e2e-derive-9333348f-7576fbffb9-264t6\" (UID: \"9714ac49-431c-4421-8c4d-206634c9eefe\") " pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"
Sep 08 15:57:59.312293 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.311579    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788e0a9f-b648-4ed4-88ac-fdf54d596016-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg\" (UID: \"788e0a9f-b648-4ed4-88ac-fdf54d596016\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"
Sep 08 15:57:59.318991 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.318967    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knwps\" (UniqueName: \"kubernetes.io/projected/788e0a9f-b648-4ed4-88ac-fdf54d596016-kube-api-access-knwps\") pod \"payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg\" (UID: \"788e0a9f-b648-4ed4-88ac-fdf54d596016\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"
Sep 08 15:57:59.319099 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.319079    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4zzd\" (UniqueName: \"kubernetes.io/projected/9714ac49-431c-4421-8c4d-206634c9eefe-kube-api-access-v4zzd\") pod \"payload-processing-e2e-derive-9333348f-7576fbffb9-264t6\" (UID: \"9714ac49-431c-4421-8c4d-206634c9eefe\") " pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"
Sep 08 15:57:59.366453 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.366432    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:57:59.410876 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.410851    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-podinfo\") pod \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") "
Sep 08 15:57:59.410994 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.410910    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istiod-ca-cert\") pod \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") "
Sep 08 15:57:59.410994 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.410938    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-envoy\") pod \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") "
Sep 08 15:57:59.410994 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.410973    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-data\") pod \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") "
Sep 08 15:57:59.411204 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.411024    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc44s\" (UniqueName: \"kubernetes.io/projected/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-kube-api-access-xc44s\") pod \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") "
Sep 08 15:57:59.411204 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.411071    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-credential-socket\") pod \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") "
Sep 08 15:57:59.411204 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.411127    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-workload-certs\") pod \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") "
Sep 08 15:57:59.411204 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.411160    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-workload-socket\") pod \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") "
Sep 08 15:57:59.411204 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.411187    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-token\") pod \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\" (UID: \"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5\") "
Sep 08 15:57:59.411509 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.411310    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-data" (OuterVolumeSpecName: "istio-data") pod "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" (UID: "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:59.411509 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.411352    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" (UID: "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:57:59.411509 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.411454    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" (UID: "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:59.411663 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.411624    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" (UID: "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:59.411726 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.411670    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" (UID: "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:59.411838 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.411820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"
Sep 08 15:57:59.412014 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.411991    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-credential-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:59.412086 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.412022    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-workload-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:59.412086 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.412039    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-workload-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:59.412086 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.412054    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istiod-ca-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:59.412086 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.412069    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-data\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:59.413280 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.413256    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" (UID: "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:57:59.413369 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.413348    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-kube-api-access-xc44s" (OuterVolumeSpecName: "kube-api-access-xc44s") pod "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" (UID: "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5"). InnerVolumeSpecName "kube-api-access-xc44s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:59.413480 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.413458    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" (UID: "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 15:57:59.413563 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.413546    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-token" (OuterVolumeSpecName: "istio-token") pod "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" (UID: "bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:57:59.436934 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.436911    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"
Sep 08 15:57:59.513531 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.513504    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-podinfo\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:59.513531 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.513531    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-envoy\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:59.513859 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.513542    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc44s\" (UniqueName: \"kubernetes.io/projected/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-kube-api-access-xc44s\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:59.513859 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.513553    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5-istio-token\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:57:59.680895 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.680874    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"]
Sep 08 15:57:59.683369 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:57:59.683345    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788e0a9f_b648_4ed4_88ac_fdf54d596016.slice/crio-a8cb6cde7459f343a80854df98c9a9da531ac717a9eebc7f257aa411ffb98f0c WatchSource:0}: Error finding container a8cb6cde7459f343a80854df98c9a9da531ac717a9eebc7f257aa411ffb98f0c: Status 404 returned error can't find the container with id a8cb6cde7459f343a80854df98c9a9da531ac717a9eebc7f257aa411ffb98f0c
Sep 08 15:57:59.712581 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:57:59.712549    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"]
Sep 08 15:57:59.714301 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:57:59.714278    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9714ac49_431c_4421_8c4d_206634c9eefe.slice/crio-be9cf187855834e6e5df51deb42506999522eced5100732d4c9a3e4367c7348e WatchSource:0}: Error finding container be9cf187855834e6e5df51deb42506999522eced5100732d4c9a3e4367c7348e: Status 404 returned error can't find the container with id be9cf187855834e6e5df51deb42506999522eced5100732d4c9a3e4367c7348e
Sep 08 15:58:00.003236 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.003198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6" event={"ID":"9714ac49-431c-4421-8c4d-206634c9eefe","Type":"ContainerStarted","Data":"560ce13408719291e0eed52c5de5bc0ab1396038c4b7fd0fbe5dd1673c066845"}
Sep 08 15:58:00.003475 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.003265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6" event={"ID":"9714ac49-431c-4421-8c4d-206634c9eefe","Type":"ContainerStarted","Data":"be9cf187855834e6e5df51deb42506999522eced5100732d4c9a3e4367c7348e"}
Sep 08 15:58:00.003475 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.003299    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"
Sep 08 15:58:00.004979 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.004950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz" event={"ID":"fe2f1411-de19-4a95-9c25-e59c32572040","Type":"ContainerStarted","Data":"907cfe49d1f9f449487c07837afac55bce26cf3f59cfc7058c58d0e88c99470e"}
Sep 08 15:58:00.005102 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.004996    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz" event={"ID":"fe2f1411-de19-4a95-9c25-e59c32572040","Type":"ContainerStarted","Data":"cb69db84cf327d61ad6d108fa6e8b1fc36da61e366a41c23cad01d70780d3556"}
Sep 08 15:58:00.005102 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.005043    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:58:00.006624 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.006589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg" event={"ID":"788e0a9f-b648-4ed4-88ac-fdf54d596016","Type":"ContainerStarted","Data":"a33a4743a4cedfe91860f680efe274a40e6ac52b8fd7333dff9aa5e1191d80e6"}
Sep 08 15:58:00.006756 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.006639    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"
Sep 08 15:58:00.006756 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.006663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg" event={"ID":"788e0a9f-b648-4ed4-88ac-fdf54d596016","Type":"ContainerStarted","Data":"a8cb6cde7459f343a80854df98c9a9da531ac717a9eebc7f257aa411ffb98f0c"}
Sep 08 15:58:00.008442 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.008418    2581 generic.go:356] "Generic (PLEG): container finished" podID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerID="65aa25b6e5ffc3982adc918312e940454e6d47885a2883fe4d7fa5c1d49aaa0a" exitCode=0
Sep 08 15:58:00.008544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.008453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp" event={"ID":"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5","Type":"ContainerDied","Data":"65aa25b6e5ffc3982adc918312e940454e6d47885a2883fe4d7fa5c1d49aaa0a"}
Sep 08 15:58:00.008544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.008458    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"
Sep 08 15:58:00.008544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.008493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp" event={"ID":"bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5","Type":"ContainerDied","Data":"739f0971b50864614a5e7c3635fe145e84b28f499a6449d9d9ebdf1b69183a0a"}
Sep 08 15:58:00.008544 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.008517    2581 scope.go:117] "RemoveContainer" containerID="65aa25b6e5ffc3982adc918312e940454e6d47885a2883fe4d7fa5c1d49aaa0a"
Sep 08 15:58:00.017712 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.017689    2581 scope.go:117] "RemoveContainer" containerID="0f632b8774aac3ddfd5d3403d58d97de34d3c0ad156cc518c37b677f4c0718ba"
Sep 08 15:58:00.019832 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.019792    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6" podStartSLOduration=1.019778985 podStartE2EDuration="1.019778985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:58:00.017491562 +0000 UTC m=+826.641183898" watchObservedRunningTime="2026-09-08 15:58:00.019778985 +0000 UTC m=+826.643471309"
Sep 08 15:58:00.028178 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.028159    2581 scope.go:117] "RemoveContainer" containerID="65aa25b6e5ffc3982adc918312e940454e6d47885a2883fe4d7fa5c1d49aaa0a"
Sep 08 15:58:00.028501 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:58:00.028477    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65aa25b6e5ffc3982adc918312e940454e6d47885a2883fe4d7fa5c1d49aaa0a\": container with ID starting with 65aa25b6e5ffc3982adc918312e940454e6d47885a2883fe4d7fa5c1d49aaa0a not found: ID does not exist" containerID="65aa25b6e5ffc3982adc918312e940454e6d47885a2883fe4d7fa5c1d49aaa0a"
Sep 08 15:58:00.028607 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.028504    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65aa25b6e5ffc3982adc918312e940454e6d47885a2883fe4d7fa5c1d49aaa0a"} err="failed to get container status \"65aa25b6e5ffc3982adc918312e940454e6d47885a2883fe4d7fa5c1d49aaa0a\": rpc error: code = NotFound desc = could not find container \"65aa25b6e5ffc3982adc918312e940454e6d47885a2883fe4d7fa5c1d49aaa0a\": container with ID starting with 65aa25b6e5ffc3982adc918312e940454e6d47885a2883fe4d7fa5c1d49aaa0a not found: ID does not exist"
Sep 08 15:58:00.028607 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.028521    2581 scope.go:117] "RemoveContainer" containerID="0f632b8774aac3ddfd5d3403d58d97de34d3c0ad156cc518c37b677f4c0718ba"
Sep 08 15:58:00.028809 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:58:00.028787    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f632b8774aac3ddfd5d3403d58d97de34d3c0ad156cc518c37b677f4c0718ba\": container with ID starting with 0f632b8774aac3ddfd5d3403d58d97de34d3c0ad156cc518c37b677f4c0718ba not found: ID does not exist" containerID="0f632b8774aac3ddfd5d3403d58d97de34d3c0ad156cc518c37b677f4c0718ba"
Sep 08 15:58:00.028872 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.028819    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f632b8774aac3ddfd5d3403d58d97de34d3c0ad156cc518c37b677f4c0718ba"} err="failed to get container status \"0f632b8774aac3ddfd5d3403d58d97de34d3c0ad156cc518c37b677f4c0718ba\": rpc error: code = NotFound desc = could not find container \"0f632b8774aac3ddfd5d3403d58d97de34d3c0ad156cc518c37b677f4c0718ba\": container with ID starting with 0f632b8774aac3ddfd5d3403d58d97de34d3c0ad156cc518c37b677f4c0718ba not found: ID does not exist"
Sep 08 15:58:00.029808 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.029788    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"]
Sep 08 15:58:00.033435 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.033411    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ac3191f1-openshift-default-75b4998df4-vkmwp"]
Sep 08 15:58:00.047218 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.047137    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg" podStartSLOduration=1.047119878 podStartE2EDuration="1.047119878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:58:00.044968532 +0000 UTC m=+826.668660857" watchObservedRunningTime="2026-09-08 15:58:00.047119878 +0000 UTC m=+826.670812198"
Sep 08 15:58:00.060750 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:00.060711    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz" podStartSLOduration=1.5215063610000001 podStartE2EDuration="2.060698731s" totalImagesPullingTime="539.19237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:58:00.059342901 +0000 UTC m=+826.683035225" watchObservedRunningTime="2026-09-08 15:58:00.060698731 +0000 UTC m=+826.684391081"
Sep 08 15:58:01.901283 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:01.901241    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" path="/var/lib/kubelet/pods/bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5/volumes"
Sep 08 15:58:06.020834 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:06.020806    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:58:11.017682 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:11.017590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"
Sep 08 15:58:11.018090 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:11.018000    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"
Sep 08 15:58:13.027560 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:13.027516    2581 patch_prober.go:28] interesting pod/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 15:58:13.027938 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:13.027589    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 15:58:14.083824 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:14.083793    2581 generic.go:356] "Generic (PLEG): container finished" podID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerID="43447c7485cafebfdf554103f3ff09c555a60d71584a7c2d3648085e7b057102" exitCode=0
Sep 08 15:58:14.084248 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:14.083851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" event={"ID":"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d","Type":"ContainerDied","Data":"43447c7485cafebfdf554103f3ff09c555a60d71584a7c2d3648085e7b057102"}
Sep 08 15:58:14.084444 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:14.084424    2581 scope.go:117] "RemoveContainer" containerID="43447c7485cafebfdf554103f3ff09c555a60d71584a7c2d3648085e7b057102"
Sep 08 15:58:14.085472 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:14.085440    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:58:14.085636 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:14.085505    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:58:14.085636 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:14.085537    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:58:15.094691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:15.094645    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" event={"ID":"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d","Type":"ContainerStarted","Data":"19c5cc2b6b5d23c08de88deff2d2f757758e39a10eb5ff93d80a1b68b4018e52"}
Sep 08 15:58:15.165932 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:15.165894    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:58:15.166143 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:15.166121    2581 patch_prober.go:28] interesting pod/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 08 15:58:15.166201 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:15.166167    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 08 15:58:16.165710 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:16.165673    2581 patch_prober.go:28] interesting pod/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 08 15:58:16.166098 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:16.165726    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 08 15:58:17.166087 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:17.166053    2581 patch_prober.go:28] interesting pod/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 08 15:58:17.166484 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:17.166116    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 08 15:58:18.166215 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:18.166181    2581 patch_prober.go:28] interesting pod/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 08 15:58:18.166599 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:18.166234    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 08 15:58:19.166408 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:19.166353    2581 patch_prober.go:28] interesting pod/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 08 15:58:19.166771 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:19.166443    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 08 15:58:20.201753 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:20.201719    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:58:20.202239 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:20.202155    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:58:20.203499 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:20.203478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:58:23.533342 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:23.533308    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"]
Sep 08 15:58:28.304302 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.304269    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"]
Sep 08 15:58:28.304806 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.304700    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerName="istio-proxy"
Sep 08 15:58:28.304806 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.304712    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerName="istio-proxy"
Sep 08 15:58:28.304806 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.304727    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerName="istio-proxy"
Sep 08 15:58:28.304806 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.304732    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerName="istio-proxy"
Sep 08 15:58:28.304806 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.304799    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerName="istio-proxy"
Sep 08 15:58:28.304806 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.304810    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdacaa10-146a-4bd0-85e1-8ddaa17fa1e5" containerName="istio-proxy"
Sep 08 15:58:28.307980 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.307962    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.310378 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.310350    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-3b98e2-openshift-default-dockercfg-7b88g\""
Sep 08 15:58:28.322952 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.322927    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"]
Sep 08 15:58:28.397227 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.397202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-token\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.397363 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.397240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-workload-socket\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.397363 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.397258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-envoy\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.397363 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.397294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-credential-socket\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.397363 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.397337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-data\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.397617 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.397405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-podinfo\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.397617 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.397458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8n5f\" (UniqueName: \"kubernetes.io/projected/6e983b15-0405-4370-a9c5-8e00e49c1f59-kube-api-access-m8n5f\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.397617 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.397509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-workload-certs\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.397617 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.397554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e983b15-0405-4370-a9c5-8e00e49c1f59-istiod-ca-cert\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.498710 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.498678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-credential-socket\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.498849 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.498744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-data\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.498849 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.498773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-podinfo\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.498849 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.498805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8n5f\" (UniqueName: \"kubernetes.io/projected/6e983b15-0405-4370-a9c5-8e00e49c1f59-kube-api-access-m8n5f\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.499007 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.498848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-workload-certs\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.499007 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.498883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e983b15-0405-4370-a9c5-8e00e49c1f59-istiod-ca-cert\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.499107 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.499057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-token\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.499164 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.499119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-credential-socket\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.499164 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.499125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-workload-socket\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.499265 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.499170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-envoy\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.499370 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.499351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-workload-certs\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.499485 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.499459    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-workload-socket\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.499554 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.499465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-data\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.499727 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.499704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e983b15-0405-4370-a9c5-8e00e49c1f59-istiod-ca-cert\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.501515 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.501490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-envoy\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.501661 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.501640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-podinfo\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.507109 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.507089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-token\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.507289 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.507268    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8n5f\" (UniqueName: \"kubernetes.io/projected/6e983b15-0405-4370-a9c5-8e00e49c1f59-kube-api-access-m8n5f\") pod \"e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") " pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.618477 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.618358    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:28.893302 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.893279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"]
Sep 08 15:58:28.895457 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:58:28.895430    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e983b15_0405_4370_a9c5_8e00e49c1f59.slice/crio-c5edf1859cfc3ca5b188083b790dc20cf22b9be16a86a7ccaf844bc7d9d92a0d WatchSource:0}: Error finding container c5edf1859cfc3ca5b188083b790dc20cf22b9be16a86a7ccaf844bc7d9d92a0d: Status 404 returned error can't find the container with id c5edf1859cfc3ca5b188083b790dc20cf22b9be16a86a7ccaf844bc7d9d92a0d
Sep 08 15:58:28.897790 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.897749    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:58:28.897880 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.897845    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:58:28.897919 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:28.897891    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:58:29.171525 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.171435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" event={"ID":"6e983b15-0405-4370-a9c5-8e00e49c1f59","Type":"ContainerStarted","Data":"350a95f9a83a51b0da7102712b5b1d1b068e0ffee9773bec10ce0fefe7af8734"}
Sep 08 15:58:29.171525 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.171493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" event={"ID":"6e983b15-0405-4370-a9c5-8e00e49c1f59","Type":"ContainerStarted","Data":"c5edf1859cfc3ca5b188083b790dc20cf22b9be16a86a7ccaf844bc7d9d92a0d"}
Sep 08 15:58:29.197484 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.197436    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" podStartSLOduration=1.197416246 podStartE2EDuration="1.197416246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:58:29.195224885 +0000 UTC m=+855.818917208" watchObservedRunningTime="2026-09-08 15:58:29.197416246 +0000 UTC m=+855.821108571"
Sep 08 15:58:29.382020 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.381986    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"]
Sep 08 15:58:29.382452 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.382187    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz" podUID="fe2f1411-de19-4a95-9c25-e59c32572040" containerName="maas-api" containerID="cri-o://907cfe49d1f9f449487c07837afac55bce26cf3f59cfc7058c58d0e88c99470e" gracePeriod=30
Sep 08 15:58:29.438506 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.438425    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"]
Sep 08 15:58:29.438727 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.438687    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6" podUID="9714ac49-431c-4421-8c4d-206634c9eefe" containerName="payload-processing" containerID="cri-o://560ce13408719291e0eed52c5de5bc0ab1396038c4b7fd0fbe5dd1673c066845" gracePeriod=30
Sep 08 15:58:29.447117 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.446725    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"]
Sep 08 15:58:29.447117 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.446937    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg" podUID="788e0a9f-b648-4ed4-88ac-fdf54d596016" containerName="payload-pre-processing" containerID="cri-o://a33a4743a4cedfe91860f680efe274a40e6ac52b8fd7333dff9aa5e1191d80e6" gracePeriod=30
Sep 08 15:58:29.620374 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.620343    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:29.625210 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.625186    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:29.736114 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.736055    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"]
Sep 08 15:58:29.737605 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.736339    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerName="istio-proxy" containerID="cri-o://19c5cc2b6b5d23c08de88deff2d2f757758e39a10eb5ff93d80a1b68b4018e52" gracePeriod=30
Sep 08 15:58:29.737605 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.737275    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:58:29.745266 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.745244    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"
Sep 08 15:58:29.771079 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.771058    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"
Sep 08 15:58:29.812785 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.812762    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2f1411-de19-4a95-9c25-e59c32572040-openshift-service-ca\") pod \"fe2f1411-de19-4a95-9c25-e59c32572040\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") "
Sep 08 15:58:29.812909 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.812828    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9714ac49-431c-4421-8c4d-206634c9eefe-plugins-config-volume\") pod \"9714ac49-431c-4421-8c4d-206634c9eefe\" (UID: \"9714ac49-431c-4421-8c4d-206634c9eefe\") "
Sep 08 15:58:29.812909 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.812886    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788e0a9f-b648-4ed4-88ac-fdf54d596016-plugins-config-volume\") pod \"788e0a9f-b648-4ed4-88ac-fdf54d596016\" (UID: \"788e0a9f-b648-4ed4-88ac-fdf54d596016\") "
Sep 08 15:58:29.813041 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.812915    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4zzd\" (UniqueName: \"kubernetes.io/projected/9714ac49-431c-4421-8c4d-206634c9eefe-kube-api-access-v4zzd\") pod \"9714ac49-431c-4421-8c4d-206634c9eefe\" (UID: \"9714ac49-431c-4421-8c4d-206634c9eefe\") "
Sep 08 15:58:29.813041 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.812945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe2f1411-de19-4a95-9c25-e59c32572040-maas-api-tls\") pod \"fe2f1411-de19-4a95-9c25-e59c32572040\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") "
Sep 08 15:58:29.813041 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.812977    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrl6x\" (UniqueName: \"kubernetes.io/projected/fe2f1411-de19-4a95-9c25-e59c32572040-kube-api-access-lrl6x\") pod \"fe2f1411-de19-4a95-9c25-e59c32572040\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") "
Sep 08 15:58:29.813041 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.813028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knwps\" (UniqueName: \"kubernetes.io/projected/788e0a9f-b648-4ed4-88ac-fdf54d596016-kube-api-access-knwps\") pod \"788e0a9f-b648-4ed4-88ac-fdf54d596016\" (UID: \"788e0a9f-b648-4ed4-88ac-fdf54d596016\") "
Sep 08 15:58:29.813234 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.813087    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2f1411-de19-4a95-9c25-e59c32572040-metrics-certs\") pod \"fe2f1411-de19-4a95-9c25-e59c32572040\" (UID: \"fe2f1411-de19-4a95-9c25-e59c32572040\") "
Sep 08 15:58:29.813234 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.813122    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe2f1411-de19-4a95-9c25-e59c32572040-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe2f1411-de19-4a95-9c25-e59c32572040" (UID: "fe2f1411-de19-4a95-9c25-e59c32572040"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:58:29.813234 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.813186    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9714ac49-431c-4421-8c4d-206634c9eefe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9714ac49-431c-4421-8c4d-206634c9eefe" (UID: "9714ac49-431c-4421-8c4d-206634c9eefe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:58:29.813501 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.813480    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2f1411-de19-4a95-9c25-e59c32572040-openshift-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:29.813585 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.813504    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9714ac49-431c-4421-8c4d-206634c9eefe-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:29.813977 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.813918    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788e0a9f-b648-4ed4-88ac-fdf54d596016-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "788e0a9f-b648-4ed4-88ac-fdf54d596016" (UID: "788e0a9f-b648-4ed4-88ac-fdf54d596016"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:58:29.817710 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.817675    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2f1411-de19-4a95-9c25-e59c32572040-kube-api-access-lrl6x" (OuterVolumeSpecName: "kube-api-access-lrl6x") pod "fe2f1411-de19-4a95-9c25-e59c32572040" (UID: "fe2f1411-de19-4a95-9c25-e59c32572040"). InnerVolumeSpecName "kube-api-access-lrl6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:58:29.817923 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.817891    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe2f1411-de19-4a95-9c25-e59c32572040-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe2f1411-de19-4a95-9c25-e59c32572040" (UID: "fe2f1411-de19-4a95-9c25-e59c32572040"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:58:29.818436 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.818376    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788e0a9f-b648-4ed4-88ac-fdf54d596016-kube-api-access-knwps" (OuterVolumeSpecName: "kube-api-access-knwps") pod "788e0a9f-b648-4ed4-88ac-fdf54d596016" (UID: "788e0a9f-b648-4ed4-88ac-fdf54d596016"). InnerVolumeSpecName "kube-api-access-knwps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:58:29.818534 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.818431    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9714ac49-431c-4421-8c4d-206634c9eefe-kube-api-access-v4zzd" (OuterVolumeSpecName: "kube-api-access-v4zzd") pod "9714ac49-431c-4421-8c4d-206634c9eefe" (UID: "9714ac49-431c-4421-8c4d-206634c9eefe"). InnerVolumeSpecName "kube-api-access-v4zzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:58:29.818722 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.818695    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe2f1411-de19-4a95-9c25-e59c32572040-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fe2f1411-de19-4a95-9c25-e59c32572040" (UID: "fe2f1411-de19-4a95-9c25-e59c32572040"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:58:29.914215 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.914107    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knwps\" (UniqueName: \"kubernetes.io/projected/788e0a9f-b648-4ed4-88ac-fdf54d596016-kube-api-access-knwps\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:29.914215 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.914136    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2f1411-de19-4a95-9c25-e59c32572040-metrics-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:29.914215 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.914151    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788e0a9f-b648-4ed4-88ac-fdf54d596016-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:29.914215 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.914166    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4zzd\" (UniqueName: \"kubernetes.io/projected/9714ac49-431c-4421-8c4d-206634c9eefe-kube-api-access-v4zzd\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:29.914215 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.914179    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe2f1411-de19-4a95-9c25-e59c32572040-maas-api-tls\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:29.914215 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:29.914193    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrl6x\" (UniqueName: \"kubernetes.io/projected/fe2f1411-de19-4a95-9c25-e59c32572040-kube-api-access-lrl6x\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:30.178493 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.178461    2581 generic.go:356] "Generic (PLEG): container finished" podID="9714ac49-431c-4421-8c4d-206634c9eefe" containerID="560ce13408719291e0eed52c5de5bc0ab1396038c4b7fd0fbe5dd1673c066845" exitCode=0
Sep 08 15:58:30.178654 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.178507    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"
Sep 08 15:58:30.178654 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.178548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6" event={"ID":"9714ac49-431c-4421-8c4d-206634c9eefe","Type":"ContainerDied","Data":"560ce13408719291e0eed52c5de5bc0ab1396038c4b7fd0fbe5dd1673c066845"}
Sep 08 15:58:30.178654 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.178607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6" event={"ID":"9714ac49-431c-4421-8c4d-206634c9eefe","Type":"ContainerDied","Data":"be9cf187855834e6e5df51deb42506999522eced5100732d4c9a3e4367c7348e"}
Sep 08 15:58:30.178654 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.178635    2581 scope.go:117] "RemoveContainer" containerID="560ce13408719291e0eed52c5de5bc0ab1396038c4b7fd0fbe5dd1673c066845"
Sep 08 15:58:30.180045 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.180021    2581 generic.go:356] "Generic (PLEG): container finished" podID="fe2f1411-de19-4a95-9c25-e59c32572040" containerID="907cfe49d1f9f449487c07837afac55bce26cf3f59cfc7058c58d0e88c99470e" exitCode=0
Sep 08 15:58:30.180140 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.180058    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"
Sep 08 15:58:30.180140 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.180101    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz" event={"ID":"fe2f1411-de19-4a95-9c25-e59c32572040","Type":"ContainerDied","Data":"907cfe49d1f9f449487c07837afac55bce26cf3f59cfc7058c58d0e88c99470e"}
Sep 08 15:58:30.180269 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.180148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz" event={"ID":"fe2f1411-de19-4a95-9c25-e59c32572040","Type":"ContainerDied","Data":"cb69db84cf327d61ad6d108fa6e8b1fc36da61e366a41c23cad01d70780d3556"}
Sep 08 15:58:30.181785 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.181761    2581 generic.go:356] "Generic (PLEG): container finished" podID="788e0a9f-b648-4ed4-88ac-fdf54d596016" containerID="a33a4743a4cedfe91860f680efe274a40e6ac52b8fd7333dff9aa5e1191d80e6" exitCode=0
Sep 08 15:58:30.181901 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.181811    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"
Sep 08 15:58:30.181901 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.181821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg" event={"ID":"788e0a9f-b648-4ed4-88ac-fdf54d596016","Type":"ContainerDied","Data":"a33a4743a4cedfe91860f680efe274a40e6ac52b8fd7333dff9aa5e1191d80e6"}
Sep 08 15:58:30.181901 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.181888    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg" event={"ID":"788e0a9f-b648-4ed4-88ac-fdf54d596016","Type":"ContainerDied","Data":"a8cb6cde7459f343a80854df98c9a9da531ac717a9eebc7f257aa411ffb98f0c"}
Sep 08 15:58:30.182415 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.182328    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:30.183543 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.183523    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:30.191375 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.189542    2581 scope.go:117] "RemoveContainer" containerID="560ce13408719291e0eed52c5de5bc0ab1396038c4b7fd0fbe5dd1673c066845"
Sep 08 15:58:30.191375 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:58:30.189853    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560ce13408719291e0eed52c5de5bc0ab1396038c4b7fd0fbe5dd1673c066845\": container with ID starting with 560ce13408719291e0eed52c5de5bc0ab1396038c4b7fd0fbe5dd1673c066845 not found: ID does not exist" containerID="560ce13408719291e0eed52c5de5bc0ab1396038c4b7fd0fbe5dd1673c066845"
Sep 08 15:58:30.191375 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.189884    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560ce13408719291e0eed52c5de5bc0ab1396038c4b7fd0fbe5dd1673c066845"} err="failed to get container status \"560ce13408719291e0eed52c5de5bc0ab1396038c4b7fd0fbe5dd1673c066845\": rpc error: code = NotFound desc = could not find container \"560ce13408719291e0eed52c5de5bc0ab1396038c4b7fd0fbe5dd1673c066845\": container with ID starting with 560ce13408719291e0eed52c5de5bc0ab1396038c4b7fd0fbe5dd1673c066845 not found: ID does not exist"
Sep 08 15:58:30.191375 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.189906    2581 scope.go:117] "RemoveContainer" containerID="907cfe49d1f9f449487c07837afac55bce26cf3f59cfc7058c58d0e88c99470e"
Sep 08 15:58:30.198885 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.198869    2581 scope.go:117] "RemoveContainer" containerID="907cfe49d1f9f449487c07837afac55bce26cf3f59cfc7058c58d0e88c99470e"
Sep 08 15:58:30.199136 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:58:30.199120    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907cfe49d1f9f449487c07837afac55bce26cf3f59cfc7058c58d0e88c99470e\": container with ID starting with 907cfe49d1f9f449487c07837afac55bce26cf3f59cfc7058c58d0e88c99470e not found: ID does not exist" containerID="907cfe49d1f9f449487c07837afac55bce26cf3f59cfc7058c58d0e88c99470e"
Sep 08 15:58:30.199183 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.199141    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907cfe49d1f9f449487c07837afac55bce26cf3f59cfc7058c58d0e88c99470e"} err="failed to get container status \"907cfe49d1f9f449487c07837afac55bce26cf3f59cfc7058c58d0e88c99470e\": rpc error: code = NotFound desc = could not find container \"907cfe49d1f9f449487c07837afac55bce26cf3f59cfc7058c58d0e88c99470e\": container with ID starting with 907cfe49d1f9f449487c07837afac55bce26cf3f59cfc7058c58d0e88c99470e not found: ID does not exist"
Sep 08 15:58:30.199183 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.199155    2581 scope.go:117] "RemoveContainer" containerID="a33a4743a4cedfe91860f680efe274a40e6ac52b8fd7333dff9aa5e1191d80e6"
Sep 08 15:58:30.202433 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.202412    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"]
Sep 08 15:58:30.205213 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.205191    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9333348f-f689ff46-cvpzg"]
Sep 08 15:58:30.207460 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.207445    2581 scope.go:117] "RemoveContainer" containerID="a33a4743a4cedfe91860f680efe274a40e6ac52b8fd7333dff9aa5e1191d80e6"
Sep 08 15:58:30.207712 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:58:30.207695    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33a4743a4cedfe91860f680efe274a40e6ac52b8fd7333dff9aa5e1191d80e6\": container with ID starting with a33a4743a4cedfe91860f680efe274a40e6ac52b8fd7333dff9aa5e1191d80e6 not found: ID does not exist" containerID="a33a4743a4cedfe91860f680efe274a40e6ac52b8fd7333dff9aa5e1191d80e6"
Sep 08 15:58:30.207765 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.207718    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33a4743a4cedfe91860f680efe274a40e6ac52b8fd7333dff9aa5e1191d80e6"} err="failed to get container status \"a33a4743a4cedfe91860f680efe274a40e6ac52b8fd7333dff9aa5e1191d80e6\": rpc error: code = NotFound desc = could not find container \"a33a4743a4cedfe91860f680efe274a40e6ac52b8fd7333dff9aa5e1191d80e6\": container with ID starting with a33a4743a4cedfe91860f680efe274a40e6ac52b8fd7333dff9aa5e1191d80e6 not found: ID does not exist"
Sep 08 15:58:30.218418 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.217677    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"]
Sep 08 15:58:30.221646 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.221623    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9333348f-7576fbffb9-264t6"]
Sep 08 15:58:30.230281 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.230261    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"]
Sep 08 15:58:30.234311 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:30.234288    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9333348f-64f5466bf5-qtldz"]
Sep 08 15:58:31.901069 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:31.901034    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="788e0a9f-b648-4ed4-88ac-fdf54d596016" path="/var/lib/kubelet/pods/788e0a9f-b648-4ed4-88ac-fdf54d596016/volumes"
Sep 08 15:58:31.901439 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:31.901424    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9714ac49-431c-4421-8c4d-206634c9eefe" path="/var/lib/kubelet/pods/9714ac49-431c-4421-8c4d-206634c9eefe/volumes"
Sep 08 15:58:31.901780 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:31.901768    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe2f1411-de19-4a95-9c25-e59c32572040" path="/var/lib/kubelet/pods/fe2f1411-de19-4a95-9c25-e59c32572040/volumes"
Sep 08 15:58:32.386350 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.386325    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"]
Sep 08 15:58:32.386900 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.386882    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="788e0a9f-b648-4ed4-88ac-fdf54d596016" containerName="payload-pre-processing"
Sep 08 15:58:32.386970 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.386903    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="788e0a9f-b648-4ed4-88ac-fdf54d596016" containerName="payload-pre-processing"
Sep 08 15:58:32.386970 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.386930    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe2f1411-de19-4a95-9c25-e59c32572040" containerName="maas-api"
Sep 08 15:58:32.386970 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.386940    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe2f1411-de19-4a95-9c25-e59c32572040" containerName="maas-api"
Sep 08 15:58:32.386970 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.386957    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9714ac49-431c-4421-8c4d-206634c9eefe" containerName="payload-processing"
Sep 08 15:58:32.386970 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.386966    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9714ac49-431c-4421-8c4d-206634c9eefe" containerName="payload-processing"
Sep 08 15:58:32.387201 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.387064    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="788e0a9f-b648-4ed4-88ac-fdf54d596016" containerName="payload-pre-processing"
Sep 08 15:58:32.387201 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.387076    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9714ac49-431c-4421-8c4d-206634c9eefe" containerName="payload-processing"
Sep 08 15:58:32.387201 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.387091    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe2f1411-de19-4a95-9c25-e59c32572040" containerName="maas-api"
Sep 08 15:58:32.392303 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.392279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:32.396046 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.396023    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:58:32.396218 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.396198    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-3b98e2-serving-cert\""
Sep 08 15:58:32.396327 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.396244    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmx7t\""
Sep 08 15:58:32.396434 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.396335    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:58:32.396434 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.396369    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:58:32.401003 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.400956    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"]
Sep 08 15:58:32.440817 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.440785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5jz4\" (UniqueName: \"kubernetes.io/projected/63db8c88-5dc3-4779-84c9-fc31de3c5833-kube-api-access-k5jz4\") pod \"maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:32.441005 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.440845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-maas-api-tls\") pod \"maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:32.441005 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.440925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63db8c88-5dc3-4779-84c9-fc31de3c5833-openshift-service-ca\") pod \"maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:32.441005 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.440984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-metrics-certs\") pod \"maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:32.542047 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.542016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5jz4\" (UniqueName: \"kubernetes.io/projected/63db8c88-5dc3-4779-84c9-fc31de3c5833-kube-api-access-k5jz4\") pod \"maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:32.542205 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.542089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-maas-api-tls\") pod \"maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:32.542205 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.542141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63db8c88-5dc3-4779-84c9-fc31de3c5833-openshift-service-ca\") pod \"maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:32.542205 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.542197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-metrics-certs\") pod \"maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:32.542338 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:58:32.542239    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-serving-cert: secret "maas-api-e2e-explicit-3b98e2-serving-cert" not found
Sep 08 15:58:32.542338 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:58:32.542298    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-maas-api-tls podName:63db8c88-5dc3-4779-84c9-fc31de3c5833 nodeName:}" failed. No retries permitted until 2026-09-08 15:58:33.042281451 +0000 UTC m=+859.665973758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-maas-api-tls") pod "maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n" (UID: "63db8c88-5dc3-4779-84c9-fc31de3c5833") : secret "maas-api-e2e-explicit-3b98e2-serving-cert" not found
Sep 08 15:58:32.542971 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.542947    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63db8c88-5dc3-4779-84c9-fc31de3c5833-openshift-service-ca\") pod \"maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:32.544772 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.544749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-metrics-certs\") pod \"maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:32.553167 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:32.553144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5jz4\" (UniqueName: \"kubernetes.io/projected/63db8c88-5dc3-4779-84c9-fc31de3c5833-kube-api-access-k5jz4\") pod \"maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:33.017290 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.017254    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"]
Sep 08 15:58:33.021816 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.021800    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"
Sep 08 15:58:33.024695 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.024670    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-3b98e2-dockercfg-bg9kc\""
Sep 08 15:58:33.024695 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.024672    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-3b98e2\""
Sep 08 15:58:33.031144 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.031118    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"]
Sep 08 15:58:33.047608 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.047584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-maas-api-tls\") pod \"maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:33.049303 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.049276    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"]
Sep 08 15:58:33.050219 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.050193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-maas-api-tls\") pod \"maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:33.052974 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.052956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"
Sep 08 15:58:33.065863 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.065808    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"]
Sep 08 15:58:33.148542 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.148510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47753cc9-7ca9-4b4e-88bd-86f698d147fa-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k\" (UID: \"47753cc9-7ca9-4b4e-88bd-86f698d147fa\") " pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"
Sep 08 15:58:33.148745 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.148555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d8j7\" (UniqueName: \"kubernetes.io/projected/47753cc9-7ca9-4b4e-88bd-86f698d147fa-kube-api-access-2d8j7\") pod \"payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k\" (UID: \"47753cc9-7ca9-4b4e-88bd-86f698d147fa\") " pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"
Sep 08 15:58:33.148745 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.148596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzl9m\" (UniqueName: \"kubernetes.io/projected/13241786-5d59-4845-ad4f-519af08979a5-kube-api-access-kzl9m\") pod \"payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn\" (UID: \"13241786-5d59-4845-ad4f-519af08979a5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"
Sep 08 15:58:33.148745 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.148659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13241786-5d59-4845-ad4f-519af08979a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn\" (UID: \"13241786-5d59-4845-ad4f-519af08979a5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"
Sep 08 15:58:33.249992 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.249964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47753cc9-7ca9-4b4e-88bd-86f698d147fa-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k\" (UID: \"47753cc9-7ca9-4b4e-88bd-86f698d147fa\") " pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"
Sep 08 15:58:33.250162 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.250001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d8j7\" (UniqueName: \"kubernetes.io/projected/47753cc9-7ca9-4b4e-88bd-86f698d147fa-kube-api-access-2d8j7\") pod \"payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k\" (UID: \"47753cc9-7ca9-4b4e-88bd-86f698d147fa\") " pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"
Sep 08 15:58:33.250162 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.250029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzl9m\" (UniqueName: \"kubernetes.io/projected/13241786-5d59-4845-ad4f-519af08979a5-kube-api-access-kzl9m\") pod \"payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn\" (UID: \"13241786-5d59-4845-ad4f-519af08979a5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"
Sep 08 15:58:33.250243 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.250193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13241786-5d59-4845-ad4f-519af08979a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn\" (UID: \"13241786-5d59-4845-ad4f-519af08979a5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"
Sep 08 15:58:33.250806 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.250788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47753cc9-7ca9-4b4e-88bd-86f698d147fa-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k\" (UID: \"47753cc9-7ca9-4b4e-88bd-86f698d147fa\") " pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"
Sep 08 15:58:33.250893 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.250875    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13241786-5d59-4845-ad4f-519af08979a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn\" (UID: \"13241786-5d59-4845-ad4f-519af08979a5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"
Sep 08 15:58:33.257999 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.257969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d8j7\" (UniqueName: \"kubernetes.io/projected/47753cc9-7ca9-4b4e-88bd-86f698d147fa-kube-api-access-2d8j7\") pod \"payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k\" (UID: \"47753cc9-7ca9-4b4e-88bd-86f698d147fa\") " pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"
Sep 08 15:58:33.258102 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.258021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzl9m\" (UniqueName: \"kubernetes.io/projected/13241786-5d59-4845-ad4f-519af08979a5-kube-api-access-kzl9m\") pod \"payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn\" (UID: \"13241786-5d59-4845-ad4f-519af08979a5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"
Sep 08 15:58:33.308871 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.308820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:33.333094 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.333072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"
Sep 08 15:58:33.383260 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.382906    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"
Sep 08 15:58:33.581370 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.581344    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"]
Sep 08 15:58:33.583464 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:58:33.583439    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63db8c88_5dc3_4779_84c9_fc31de3c5833.slice/crio-b611aa1d7cefd06272faff68ffa83a1e9aa35f2d3728e0e1887b7438bbc19773 WatchSource:0}: Error finding container b611aa1d7cefd06272faff68ffa83a1e9aa35f2d3728e0e1887b7438bbc19773: Status 404 returned error can't find the container with id b611aa1d7cefd06272faff68ffa83a1e9aa35f2d3728e0e1887b7438bbc19773
Sep 08 15:58:33.610170 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.610147    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"]
Sep 08 15:58:33.612199 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:58:33.612176    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13241786_5d59_4845_ad4f_519af08979a5.slice/crio-74e6a9dc4d3b98df39afb9aff610582f8d2720f70038ddccb6d8cd77736b50ee WatchSource:0}: Error finding container 74e6a9dc4d3b98df39afb9aff610582f8d2720f70038ddccb6d8cd77736b50ee: Status 404 returned error can't find the container with id 74e6a9dc4d3b98df39afb9aff610582f8d2720f70038ddccb6d8cd77736b50ee
Sep 08 15:58:33.663095 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:33.663074    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"]
Sep 08 15:58:33.665852 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:58:33.665828    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47753cc9_7ca9_4b4e_88bd_86f698d147fa.slice/crio-b1b182822c3dc5c9ad30a8b158e5134521005046a96c10aa2c048b9c34fbdd16 WatchSource:0}: Error finding container b1b182822c3dc5c9ad30a8b158e5134521005046a96c10aa2c048b9c34fbdd16: Status 404 returned error can't find the container with id b1b182822c3dc5c9ad30a8b158e5134521005046a96c10aa2c048b9c34fbdd16
Sep 08 15:58:34.206189 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:34.206138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k" event={"ID":"47753cc9-7ca9-4b4e-88bd-86f698d147fa","Type":"ContainerStarted","Data":"5c6d4eba245c497f0ef0c97b5d80fef70653b84d2e1c5b0e0eae15bf330f91b9"}
Sep 08 15:58:34.206626 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:34.206206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k" event={"ID":"47753cc9-7ca9-4b4e-88bd-86f698d147fa","Type":"ContainerStarted","Data":"b1b182822c3dc5c9ad30a8b158e5134521005046a96c10aa2c048b9c34fbdd16"}
Sep 08 15:58:34.206626 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:34.206234    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"
Sep 08 15:58:34.208076 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:34.208050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn" event={"ID":"13241786-5d59-4845-ad4f-519af08979a5","Type":"ContainerStarted","Data":"f477f8ab17674c94022afe20e7eb0f486e46fd6ea505ccff90aa3a4630db7294"}
Sep 08 15:58:34.208204 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:34.208097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn" event={"ID":"13241786-5d59-4845-ad4f-519af08979a5","Type":"ContainerStarted","Data":"74e6a9dc4d3b98df39afb9aff610582f8d2720f70038ddccb6d8cd77736b50ee"}
Sep 08 15:58:34.208204 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:34.208164    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"
Sep 08 15:58:34.209821 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:34.209795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n" event={"ID":"63db8c88-5dc3-4779-84c9-fc31de3c5833","Type":"ContainerStarted","Data":"3aa59845a85f0dbd3f94bfa607264f30950dae6ff993d812651c75464d82eb14"}
Sep 08 15:58:34.209934 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:34.209840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n" event={"ID":"63db8c88-5dc3-4779-84c9-fc31de3c5833","Type":"ContainerStarted","Data":"b611aa1d7cefd06272faff68ffa83a1e9aa35f2d3728e0e1887b7438bbc19773"}
Sep 08 15:58:34.209985 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:34.209972    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:34.225811 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:34.225770    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k" podStartSLOduration=1.225757576 podStartE2EDuration="1.225757576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:58:34.225174273 +0000 UTC m=+860.848866599" watchObservedRunningTime="2026-09-08 15:58:34.225757576 +0000 UTC m=+860.849449921"
Sep 08 15:58:34.242443 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:34.242378    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n" podStartSLOduration=1.766936751 podStartE2EDuration="2.242361425s" totalImagesPullingTime="475.424674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:58:34.241371251 +0000 UTC m=+860.865063594" watchObservedRunningTime="2026-09-08 15:58:34.242361425 +0000 UTC m=+860.866053750"
Sep 08 15:58:34.257370 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:34.257339    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn" podStartSLOduration=2.257326432 podStartE2EDuration="2.257326432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:58:34.255785279 +0000 UTC m=+860.879477628" watchObservedRunningTime="2026-09-08 15:58:34.257326432 +0000 UTC m=+860.881018756"
Sep 08 15:58:35.005596 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.005544    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:58:35.066573 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.066548    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-credential-socket\") pod \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") "
Sep 08 15:58:35.066718 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.066580    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-envoy\") pod \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") "
Sep 08 15:58:35.066718 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.066606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-workload-certs\") pod \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") "
Sep 08 15:58:35.066718 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.066644    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9fl4\" (UniqueName: \"kubernetes.io/projected/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-kube-api-access-j9fl4\") pod \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") "
Sep 08 15:58:35.066718 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.066663    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-podinfo\") pod \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") "
Sep 08 15:58:35.066718 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.066677    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-token\") pod \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") "
Sep 08 15:58:35.066973 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.066797    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-data\") pod \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") "
Sep 08 15:58:35.066973 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.066831    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istiod-ca-cert\") pod \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") "
Sep 08 15:58:35.066973 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.066855    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" (UID: "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:35.066973 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.066934    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-workload-socket\") pod \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\" (UID: \"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d\") "
Sep 08 15:58:35.067181 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.066981    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" (UID: "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:35.067181 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.067053    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" (UID: "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:35.067293 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.067245    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" (UID: "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:35.067293 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.067280    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" (UID: "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:58:35.067543 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.067328    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-workload-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.067543 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.067343    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-data\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.067543 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.067358    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-workload-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.067543 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.067371    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-credential-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.068982 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.068957    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" (UID: "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:58:35.069117 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.068987    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" (UID: "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 15:58:35.069326 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.069311    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" (UID: "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:35.069375 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.069351    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-kube-api-access-j9fl4" (OuterVolumeSpecName: "kube-api-access-j9fl4") pod "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" (UID: "5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d"). InnerVolumeSpecName "kube-api-access-j9fl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:58:35.167798 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.167777    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9fl4\" (UniqueName: \"kubernetes.io/projected/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-kube-api-access-j9fl4\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.167798 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.167797    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-podinfo\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.167938 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.167806    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-token\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.167938 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.167815    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istiod-ca-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.167938 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.167824    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d-istio-envoy\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:35.218115 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.218081    2581 generic.go:356] "Generic (PLEG): container finished" podID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerID="19c5cc2b6b5d23c08de88deff2d2f757758e39a10eb5ff93d80a1b68b4018e52" exitCode=0
Sep 08 15:58:35.218503 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.218115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" event={"ID":"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d","Type":"ContainerDied","Data":"19c5cc2b6b5d23c08de88deff2d2f757758e39a10eb5ff93d80a1b68b4018e52"}
Sep 08 15:58:35.218503 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.218149    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"
Sep 08 15:58:35.218503 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.218193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj" event={"ID":"5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d","Type":"ContainerDied","Data":"16c7c326c9bfc0445c7be23b215d559bfb06bcd79d552fa2f3b9d6c4221c11c2"}
Sep 08 15:58:35.218503 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.218244    2581 scope.go:117] "RemoveContainer" containerID="19c5cc2b6b5d23c08de88deff2d2f757758e39a10eb5ff93d80a1b68b4018e52"
Sep 08 15:58:35.228916 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.228862    2581 scope.go:117] "RemoveContainer" containerID="43447c7485cafebfdf554103f3ff09c555a60d71584a7c2d3648085e7b057102"
Sep 08 15:58:35.239403 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.239369    2581 scope.go:117] "RemoveContainer" containerID="19c5cc2b6b5d23c08de88deff2d2f757758e39a10eb5ff93d80a1b68b4018e52"
Sep 08 15:58:35.239720 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:58:35.239699    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c5cc2b6b5d23c08de88deff2d2f757758e39a10eb5ff93d80a1b68b4018e52\": container with ID starting with 19c5cc2b6b5d23c08de88deff2d2f757758e39a10eb5ff93d80a1b68b4018e52 not found: ID does not exist" containerID="19c5cc2b6b5d23c08de88deff2d2f757758e39a10eb5ff93d80a1b68b4018e52"
Sep 08 15:58:35.239790 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.239729    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c5cc2b6b5d23c08de88deff2d2f757758e39a10eb5ff93d80a1b68b4018e52"} err="failed to get container status \"19c5cc2b6b5d23c08de88deff2d2f757758e39a10eb5ff93d80a1b68b4018e52\": rpc error: code = NotFound desc = could not find container \"19c5cc2b6b5d23c08de88deff2d2f757758e39a10eb5ff93d80a1b68b4018e52\": container with ID starting with 19c5cc2b6b5d23c08de88deff2d2f757758e39a10eb5ff93d80a1b68b4018e52 not found: ID does not exist"
Sep 08 15:58:35.239790 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.239752    2581 scope.go:117] "RemoveContainer" containerID="43447c7485cafebfdf554103f3ff09c555a60d71584a7c2d3648085e7b057102"
Sep 08 15:58:35.240017 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:58:35.239998    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43447c7485cafebfdf554103f3ff09c555a60d71584a7c2d3648085e7b057102\": container with ID starting with 43447c7485cafebfdf554103f3ff09c555a60d71584a7c2d3648085e7b057102 not found: ID does not exist" containerID="43447c7485cafebfdf554103f3ff09c555a60d71584a7c2d3648085e7b057102"
Sep 08 15:58:35.240084 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.240026    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43447c7485cafebfdf554103f3ff09c555a60d71584a7c2d3648085e7b057102"} err="failed to get container status \"43447c7485cafebfdf554103f3ff09c555a60d71584a7c2d3648085e7b057102\": rpc error: code = NotFound desc = could not find container \"43447c7485cafebfdf554103f3ff09c555a60d71584a7c2d3648085e7b057102\": container with ID starting with 43447c7485cafebfdf554103f3ff09c555a60d71584a7c2d3648085e7b057102 not found: ID does not exist"
Sep 08 15:58:35.244935 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.244911    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"]
Sep 08 15:58:35.255482 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.255459    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-9333348f-openshift-default-89d7555b8-q26kj"]
Sep 08 15:58:35.901778 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:35.901739    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" path="/var/lib/kubelet/pods/5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d/volumes"
Sep 08 15:58:38.553565 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.553522    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf" podUID="74c54675-ed87-4e97-a221-4e7fa3bb527f" containerName="main" containerID="cri-o://3d957875e5df0a18ba41bd931f1c3f7070fc1eca27e09db38e45e053d86cff3b" gracePeriod=45
Sep 08 15:58:38.804683 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.804631    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:58:38.898145 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.898116    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-model-cache\") pod \"74c54675-ed87-4e97-a221-4e7fa3bb527f\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") "
Sep 08 15:58:38.898286 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.898166    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-home\") pod \"74c54675-ed87-4e97-a221-4e7fa3bb527f\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") "
Sep 08 15:58:38.898286 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.898194    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-dshm\") pod \"74c54675-ed87-4e97-a221-4e7fa3bb527f\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") "
Sep 08 15:58:38.898286 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.898210    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-tmp-dir\") pod \"74c54675-ed87-4e97-a221-4e7fa3bb527f\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") "
Sep 08 15:58:38.898286 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.898248    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c54675-ed87-4e97-a221-4e7fa3bb527f-tls-certs\") pod \"74c54675-ed87-4e97-a221-4e7fa3bb527f\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") "
Sep 08 15:58:38.898286 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.898277    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9gdn\" (UniqueName: \"kubernetes.io/projected/74c54675-ed87-4e97-a221-4e7fa3bb527f-kube-api-access-n9gdn\") pod \"74c54675-ed87-4e97-a221-4e7fa3bb527f\" (UID: \"74c54675-ed87-4e97-a221-4e7fa3bb527f\") "
Sep 08 15:58:38.898577 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.898440    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-model-cache" (OuterVolumeSpecName: "model-cache") pod "74c54675-ed87-4e97-a221-4e7fa3bb527f" (UID: "74c54675-ed87-4e97-a221-4e7fa3bb527f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:38.898577 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.898489    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-home" (OuterVolumeSpecName: "home") pod "74c54675-ed87-4e97-a221-4e7fa3bb527f" (UID: "74c54675-ed87-4e97-a221-4e7fa3bb527f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:38.898577 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.898565    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "74c54675-ed87-4e97-a221-4e7fa3bb527f" (UID: "74c54675-ed87-4e97-a221-4e7fa3bb527f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:38.898736 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.898637    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-model-cache\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:38.898736 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.898652    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-home\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:38.900467 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.900439    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-dshm" (OuterVolumeSpecName: "dshm") pod "74c54675-ed87-4e97-a221-4e7fa3bb527f" (UID: "74c54675-ed87-4e97-a221-4e7fa3bb527f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:58:38.900563 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.900523    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c54675-ed87-4e97-a221-4e7fa3bb527f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74c54675-ed87-4e97-a221-4e7fa3bb527f" (UID: "74c54675-ed87-4e97-a221-4e7fa3bb527f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:58:38.900563 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.900539    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c54675-ed87-4e97-a221-4e7fa3bb527f-kube-api-access-n9gdn" (OuterVolumeSpecName: "kube-api-access-n9gdn") pod "74c54675-ed87-4e97-a221-4e7fa3bb527f" (UID: "74c54675-ed87-4e97-a221-4e7fa3bb527f"). InnerVolumeSpecName "kube-api-access-n9gdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:58:38.999457 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.999429    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c54675-ed87-4e97-a221-4e7fa3bb527f-tls-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:38.999457 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.999452    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9gdn\" (UniqueName: \"kubernetes.io/projected/74c54675-ed87-4e97-a221-4e7fa3bb527f-kube-api-access-n9gdn\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:38.999457 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.999461    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-dshm\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:38.999630 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:38.999469    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c54675-ed87-4e97-a221-4e7fa3bb527f-tmp-dir\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:58:39.241217 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:39.241180    2581 generic.go:356] "Generic (PLEG): container finished" podID="74c54675-ed87-4e97-a221-4e7fa3bb527f" containerID="3d957875e5df0a18ba41bd931f1c3f7070fc1eca27e09db38e45e053d86cff3b" exitCode=0
Sep 08 15:58:39.241379 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:39.241220    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"
Sep 08 15:58:39.241379 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:39.241262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf" event={"ID":"74c54675-ed87-4e97-a221-4e7fa3bb527f","Type":"ContainerDied","Data":"3d957875e5df0a18ba41bd931f1c3f7070fc1eca27e09db38e45e053d86cff3b"}
Sep 08 15:58:39.241379 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:39.241307    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf" event={"ID":"74c54675-ed87-4e97-a221-4e7fa3bb527f","Type":"ContainerDied","Data":"3fe375f60cf6b558f49640bf53fdb0b6f37a3c03e50fcf3ec8aa4e86e909b45a"}
Sep 08 15:58:39.241379 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:39.241325    2581 scope.go:117] "RemoveContainer" containerID="3d957875e5df0a18ba41bd931f1c3f7070fc1eca27e09db38e45e053d86cff3b"
Sep 08 15:58:39.250499 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:39.250483    2581 scope.go:117] "RemoveContainer" containerID="3d957875e5df0a18ba41bd931f1c3f7070fc1eca27e09db38e45e053d86cff3b"
Sep 08 15:58:39.250793 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:58:39.250776    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d957875e5df0a18ba41bd931f1c3f7070fc1eca27e09db38e45e053d86cff3b\": container with ID starting with 3d957875e5df0a18ba41bd931f1c3f7070fc1eca27e09db38e45e053d86cff3b not found: ID does not exist" containerID="3d957875e5df0a18ba41bd931f1c3f7070fc1eca27e09db38e45e053d86cff3b"
Sep 08 15:58:39.250842 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:39.250802    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d957875e5df0a18ba41bd931f1c3f7070fc1eca27e09db38e45e053d86cff3b"} err="failed to get container status \"3d957875e5df0a18ba41bd931f1c3f7070fc1eca27e09db38e45e053d86cff3b\": rpc error: code = NotFound desc = could not find container \"3d957875e5df0a18ba41bd931f1c3f7070fc1eca27e09db38e45e053d86cff3b\": container with ID starting with 3d957875e5df0a18ba41bd931f1c3f7070fc1eca27e09db38e45e053d86cff3b not found: ID does not exist"
Sep 08 15:58:39.265138 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:39.265112    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"]
Sep 08 15:58:39.271448 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:39.271427    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-e6f19b8d-kserve-7bc5f66d4-vltxf"]
Sep 08 15:58:39.902720 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:39.902686    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74c54675-ed87-4e97-a221-4e7fa3bb527f" path="/var/lib/kubelet/pods/74c54675-ed87-4e97-a221-4e7fa3bb527f/volumes"
Sep 08 15:58:40.223975 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:40.223899    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:58:45.221159 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:45.221124    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"
Sep 08 15:58:45.221627 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:45.221294    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"
Sep 08 15:58:52.326223 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:52.326182    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerID="350a95f9a83a51b0da7102712b5b1d1b068e0ffee9773bec10ce0fefe7af8734" exitCode=0
Sep 08 15:58:52.326659 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:52.326253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" event={"ID":"6e983b15-0405-4370-a9c5-8e00e49c1f59","Type":"ContainerDied","Data":"350a95f9a83a51b0da7102712b5b1d1b068e0ffee9773bec10ce0fefe7af8734"}
Sep 08 15:58:52.326659 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:52.326638    2581 scope.go:117] "RemoveContainer" containerID="350a95f9a83a51b0da7102712b5b1d1b068e0ffee9773bec10ce0fefe7af8734"
Sep 08 15:58:52.327685 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:52.327653    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:58:52.327809 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:52.327738    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:58:52.327809 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:52.327785    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:58:53.333871 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:53.333835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" event={"ID":"6e983b15-0405-4370-a9c5-8e00e49c1f59","Type":"ContainerStarted","Data":"fa48bbc6b2592b1526f6cdbcdaf9ab7268111c01adb82a4a17be8e058481defe"}
Sep 08 15:58:53.619447 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:53.619355    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:53.619623 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:53.619596    2581 patch_prober.go:28] interesting pod/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 08 15:58:53.619714 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:53.619691    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 08 15:58:54.619011 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:54.618972    2581 patch_prober.go:28] interesting pod/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 08 15:58:54.619473 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:54.619030    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 08 15:58:55.618843 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:55.618808    2581 patch_prober.go:28] interesting pod/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 08 15:58:55.619009 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:55.618894    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 08 15:58:56.619690 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:56.619653    2581 patch_prober.go:28] interesting pod/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 08 15:58:56.620067 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:56.619719    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 08 15:58:57.619166 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:57.619130    2581 patch_prober.go:28] interesting pod/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 08 15:58:57.619341 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:57.619185    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 08 15:58:58.623290 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:58.623255    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:58.623691 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:58.623608    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:58:58.624283 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:58:58.624260    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 15:59:13.871428 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:13.871379    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 15:59:13.874351 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:13.874330    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 15:59:49.738265 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:49.738224    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"]
Sep 08 15:59:49.738798 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:49.738586    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n" podUID="63db8c88-5dc3-4779-84c9-fc31de3c5833" containerName="maas-api" containerID="cri-o://3aa59845a85f0dbd3f94bfa607264f30950dae6ff993d812651c75464d82eb14" gracePeriod=30
Sep 08 15:59:49.788826 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:49.788788    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"]
Sep 08 15:59:49.789083 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:49.789053    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k" podUID="47753cc9-7ca9-4b4e-88bd-86f698d147fa" containerName="payload-processing" containerID="cri-o://5c6d4eba245c497f0ef0c97b5d80fef70653b84d2e1c5b0e0eae15bf330f91b9" gracePeriod=30
Sep 08 15:59:49.797101 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:49.797076    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"]
Sep 08 15:59:49.797520 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:49.797488    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn" podUID="13241786-5d59-4845-ad4f-519af08979a5" containerName="payload-pre-processing" containerID="cri-o://f477f8ab17674c94022afe20e7eb0f486e46fd6ea505ccff90aa3a4630db7294" gracePeriod=30
Sep 08 15:59:50.068442 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.068330    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:59:50.094305 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.094177    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"
Sep 08 15:59:50.118290 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.118261    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"
Sep 08 15:59:50.205235 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.205204    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5jz4\" (UniqueName: \"kubernetes.io/projected/63db8c88-5dc3-4779-84c9-fc31de3c5833-kube-api-access-k5jz4\") pod \"63db8c88-5dc3-4779-84c9-fc31de3c5833\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") "
Sep 08 15:59:50.205452 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.205283    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d8j7\" (UniqueName: \"kubernetes.io/projected/47753cc9-7ca9-4b4e-88bd-86f698d147fa-kube-api-access-2d8j7\") pod \"47753cc9-7ca9-4b4e-88bd-86f698d147fa\" (UID: \"47753cc9-7ca9-4b4e-88bd-86f698d147fa\") "
Sep 08 15:59:50.205452 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.205315    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-maas-api-tls\") pod \"63db8c88-5dc3-4779-84c9-fc31de3c5833\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") "
Sep 08 15:59:50.205452 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.205343    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47753cc9-7ca9-4b4e-88bd-86f698d147fa-plugins-config-volume\") pod \"47753cc9-7ca9-4b4e-88bd-86f698d147fa\" (UID: \"47753cc9-7ca9-4b4e-88bd-86f698d147fa\") "
Sep 08 15:59:50.205452 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.205375    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzl9m\" (UniqueName: \"kubernetes.io/projected/13241786-5d59-4845-ad4f-519af08979a5-kube-api-access-kzl9m\") pod \"13241786-5d59-4845-ad4f-519af08979a5\" (UID: \"13241786-5d59-4845-ad4f-519af08979a5\") "
Sep 08 15:59:50.205452 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.205413    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13241786-5d59-4845-ad4f-519af08979a5-plugins-config-volume\") pod \"13241786-5d59-4845-ad4f-519af08979a5\" (UID: \"13241786-5d59-4845-ad4f-519af08979a5\") "
Sep 08 15:59:50.205756 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.205474    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63db8c88-5dc3-4779-84c9-fc31de3c5833-openshift-service-ca\") pod \"63db8c88-5dc3-4779-84c9-fc31de3c5833\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") "
Sep 08 15:59:50.205756 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.205499    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-metrics-certs\") pod \"63db8c88-5dc3-4779-84c9-fc31de3c5833\" (UID: \"63db8c88-5dc3-4779-84c9-fc31de3c5833\") "
Sep 08 15:59:50.205861 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.205769    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47753cc9-7ca9-4b4e-88bd-86f698d147fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "47753cc9-7ca9-4b4e-88bd-86f698d147fa" (UID: "47753cc9-7ca9-4b4e-88bd-86f698d147fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:59:50.206028 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.205999    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47753cc9-7ca9-4b4e-88bd-86f698d147fa-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:59:50.206028 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.206000    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63db8c88-5dc3-4779-84c9-fc31de3c5833-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63db8c88-5dc3-4779-84c9-fc31de3c5833" (UID: "63db8c88-5dc3-4779-84c9-fc31de3c5833"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:59:50.206211 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.206187    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13241786-5d59-4845-ad4f-519af08979a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13241786-5d59-4845-ad4f-519af08979a5" (UID: "13241786-5d59-4845-ad4f-519af08979a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:59:50.207749 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.207714    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13241786-5d59-4845-ad4f-519af08979a5-kube-api-access-kzl9m" (OuterVolumeSpecName: "kube-api-access-kzl9m") pod "13241786-5d59-4845-ad4f-519af08979a5" (UID: "13241786-5d59-4845-ad4f-519af08979a5"). InnerVolumeSpecName "kube-api-access-kzl9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:59:50.207951 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.207927    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63db8c88-5dc3-4779-84c9-fc31de3c5833-kube-api-access-k5jz4" (OuterVolumeSpecName: "kube-api-access-k5jz4") pod "63db8c88-5dc3-4779-84c9-fc31de3c5833" (UID: "63db8c88-5dc3-4779-84c9-fc31de3c5833"). InnerVolumeSpecName "kube-api-access-k5jz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:59:50.208030 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.207933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63db8c88-5dc3-4779-84c9-fc31de3c5833" (UID: "63db8c88-5dc3-4779-84c9-fc31de3c5833"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:59:50.208092 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.208070    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47753cc9-7ca9-4b4e-88bd-86f698d147fa-kube-api-access-2d8j7" (OuterVolumeSpecName: "kube-api-access-2d8j7") pod "47753cc9-7ca9-4b4e-88bd-86f698d147fa" (UID: "47753cc9-7ca9-4b4e-88bd-86f698d147fa"). InnerVolumeSpecName "kube-api-access-2d8j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:59:50.208092 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.208076    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "63db8c88-5dc3-4779-84c9-fc31de3c5833" (UID: "63db8c88-5dc3-4779-84c9-fc31de3c5833"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:59:50.307231 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.307130    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5jz4\" (UniqueName: \"kubernetes.io/projected/63db8c88-5dc3-4779-84c9-fc31de3c5833-kube-api-access-k5jz4\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:59:50.307231 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.307165    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d8j7\" (UniqueName: \"kubernetes.io/projected/47753cc9-7ca9-4b4e-88bd-86f698d147fa-kube-api-access-2d8j7\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:59:50.307231 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.307179    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-maas-api-tls\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:59:50.307231 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.307192    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzl9m\" (UniqueName: \"kubernetes.io/projected/13241786-5d59-4845-ad4f-519af08979a5-kube-api-access-kzl9m\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:59:50.307231 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.307203    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13241786-5d59-4845-ad4f-519af08979a5-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:59:50.307231 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.307215    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63db8c88-5dc3-4779-84c9-fc31de3c5833-openshift-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:59:50.307231 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.307226    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63db8c88-5dc3-4779-84c9-fc31de3c5833-metrics-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 15:59:50.624362 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.624275    2581 generic.go:356] "Generic (PLEG): container finished" podID="63db8c88-5dc3-4779-84c9-fc31de3c5833" containerID="3aa59845a85f0dbd3f94bfa607264f30950dae6ff993d812651c75464d82eb14" exitCode=0
Sep 08 15:59:50.624362 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.624324    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"
Sep 08 15:59:50.624609 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.624364    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n" event={"ID":"63db8c88-5dc3-4779-84c9-fc31de3c5833","Type":"ContainerDied","Data":"3aa59845a85f0dbd3f94bfa607264f30950dae6ff993d812651c75464d82eb14"}
Sep 08 15:59:50.624609 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.624436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n" event={"ID":"63db8c88-5dc3-4779-84c9-fc31de3c5833","Type":"ContainerDied","Data":"b611aa1d7cefd06272faff68ffa83a1e9aa35f2d3728e0e1887b7438bbc19773"}
Sep 08 15:59:50.624609 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.624461    2581 scope.go:117] "RemoveContainer" containerID="3aa59845a85f0dbd3f94bfa607264f30950dae6ff993d812651c75464d82eb14"
Sep 08 15:59:50.625680 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.625659    2581 generic.go:356] "Generic (PLEG): container finished" podID="47753cc9-7ca9-4b4e-88bd-86f698d147fa" containerID="5c6d4eba245c497f0ef0c97b5d80fef70653b84d2e1c5b0e0eae15bf330f91b9" exitCode=0
Sep 08 15:59:50.625789 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.625724    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"
Sep 08 15:59:50.625789 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.625759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k" event={"ID":"47753cc9-7ca9-4b4e-88bd-86f698d147fa","Type":"ContainerDied","Data":"5c6d4eba245c497f0ef0c97b5d80fef70653b84d2e1c5b0e0eae15bf330f91b9"}
Sep 08 15:59:50.625908 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.625801    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k" event={"ID":"47753cc9-7ca9-4b4e-88bd-86f698d147fa","Type":"ContainerDied","Data":"b1b182822c3dc5c9ad30a8b158e5134521005046a96c10aa2c048b9c34fbdd16"}
Sep 08 15:59:50.627315 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.627288    2581 generic.go:356] "Generic (PLEG): container finished" podID="13241786-5d59-4845-ad4f-519af08979a5" containerID="f477f8ab17674c94022afe20e7eb0f486e46fd6ea505ccff90aa3a4630db7294" exitCode=0
Sep 08 15:59:50.627440 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.627327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn" event={"ID":"13241786-5d59-4845-ad4f-519af08979a5","Type":"ContainerDied","Data":"f477f8ab17674c94022afe20e7eb0f486e46fd6ea505ccff90aa3a4630db7294"}
Sep 08 15:59:50.627440 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.627367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn" event={"ID":"13241786-5d59-4845-ad4f-519af08979a5","Type":"ContainerDied","Data":"74e6a9dc4d3b98df39afb9aff610582f8d2720f70038ddccb6d8cd77736b50ee"}
Sep 08 15:59:50.627440 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.627331    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"
Sep 08 15:59:50.635365 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.635346    2581 scope.go:117] "RemoveContainer" containerID="3aa59845a85f0dbd3f94bfa607264f30950dae6ff993d812651c75464d82eb14"
Sep 08 15:59:50.635691 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:59:50.635664    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa59845a85f0dbd3f94bfa607264f30950dae6ff993d812651c75464d82eb14\": container with ID starting with 3aa59845a85f0dbd3f94bfa607264f30950dae6ff993d812651c75464d82eb14 not found: ID does not exist" containerID="3aa59845a85f0dbd3f94bfa607264f30950dae6ff993d812651c75464d82eb14"
Sep 08 15:59:50.635765 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.635704    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa59845a85f0dbd3f94bfa607264f30950dae6ff993d812651c75464d82eb14"} err="failed to get container status \"3aa59845a85f0dbd3f94bfa607264f30950dae6ff993d812651c75464d82eb14\": rpc error: code = NotFound desc = could not find container \"3aa59845a85f0dbd3f94bfa607264f30950dae6ff993d812651c75464d82eb14\": container with ID starting with 3aa59845a85f0dbd3f94bfa607264f30950dae6ff993d812651c75464d82eb14 not found: ID does not exist"
Sep 08 15:59:50.635765 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.635728    2581 scope.go:117] "RemoveContainer" containerID="5c6d4eba245c497f0ef0c97b5d80fef70653b84d2e1c5b0e0eae15bf330f91b9"
Sep 08 15:59:50.643911 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.643895    2581 scope.go:117] "RemoveContainer" containerID="5c6d4eba245c497f0ef0c97b5d80fef70653b84d2e1c5b0e0eae15bf330f91b9"
Sep 08 15:59:50.644188 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:59:50.644166    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6d4eba245c497f0ef0c97b5d80fef70653b84d2e1c5b0e0eae15bf330f91b9\": container with ID starting with 5c6d4eba245c497f0ef0c97b5d80fef70653b84d2e1c5b0e0eae15bf330f91b9 not found: ID does not exist" containerID="5c6d4eba245c497f0ef0c97b5d80fef70653b84d2e1c5b0e0eae15bf330f91b9"
Sep 08 15:59:50.644257 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.644195    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6d4eba245c497f0ef0c97b5d80fef70653b84d2e1c5b0e0eae15bf330f91b9"} err="failed to get container status \"5c6d4eba245c497f0ef0c97b5d80fef70653b84d2e1c5b0e0eae15bf330f91b9\": rpc error: code = NotFound desc = could not find container \"5c6d4eba245c497f0ef0c97b5d80fef70653b84d2e1c5b0e0eae15bf330f91b9\": container with ID starting with 5c6d4eba245c497f0ef0c97b5d80fef70653b84d2e1c5b0e0eae15bf330f91b9 not found: ID does not exist"
Sep 08 15:59:50.644257 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.644210    2581 scope.go:117] "RemoveContainer" containerID="f477f8ab17674c94022afe20e7eb0f486e46fd6ea505ccff90aa3a4630db7294"
Sep 08 15:59:50.651839 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.651810    2581 scope.go:117] "RemoveContainer" containerID="f477f8ab17674c94022afe20e7eb0f486e46fd6ea505ccff90aa3a4630db7294"
Sep 08 15:59:50.652120 ip-10-0-134-107 kubenswrapper[2581]: E0908 15:59:50.652100    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f477f8ab17674c94022afe20e7eb0f486e46fd6ea505ccff90aa3a4630db7294\": container with ID starting with f477f8ab17674c94022afe20e7eb0f486e46fd6ea505ccff90aa3a4630db7294 not found: ID does not exist" containerID="f477f8ab17674c94022afe20e7eb0f486e46fd6ea505ccff90aa3a4630db7294"
Sep 08 15:59:50.652183 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.652127    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f477f8ab17674c94022afe20e7eb0f486e46fd6ea505ccff90aa3a4630db7294"} err="failed to get container status \"f477f8ab17674c94022afe20e7eb0f486e46fd6ea505ccff90aa3a4630db7294\": rpc error: code = NotFound desc = could not find container \"f477f8ab17674c94022afe20e7eb0f486e46fd6ea505ccff90aa3a4630db7294\": container with ID starting with f477f8ab17674c94022afe20e7eb0f486e46fd6ea505ccff90aa3a4630db7294 not found: ID does not exist"
Sep 08 15:59:50.659045 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.659012    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"]
Sep 08 15:59:50.665100 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.665078    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3b98e2-55df488c6-fqd9n"]
Sep 08 15:59:50.674953 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.674934    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"]
Sep 08 15:59:50.680156 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.680136    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3b98e2-5c7d5976b5-4nd6k"]
Sep 08 15:59:50.690277 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.690258    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"]
Sep 08 15:59:50.694182 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:50.694163    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3b98e2-6994fbdcb6-5xtnn"]
Sep 08 15:59:51.901923 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:51.901891    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13241786-5d59-4845-ad4f-519af08979a5" path="/var/lib/kubelet/pods/13241786-5d59-4845-ad4f-519af08979a5/volumes"
Sep 08 15:59:51.902381 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:51.902346    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47753cc9-7ca9-4b4e-88bd-86f698d147fa" path="/var/lib/kubelet/pods/47753cc9-7ca9-4b4e-88bd-86f698d147fa/volumes"
Sep 08 15:59:51.902767 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:51.902750    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63db8c88-5dc3-4779-84c9-fc31de3c5833" path="/var/lib/kubelet/pods/63db8c88-5dc3-4779-84c9-fc31de3c5833/volumes"
Sep 08 15:59:56.556329 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:56.556295    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"]
Sep 08 15:59:56.556721 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:56.556497    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerName="istio-proxy" containerID="cri-o://fa48bbc6b2592b1526f6cdbcdaf9ab7268111c01adb82a4a17be8e058481defe" gracePeriod=30
Sep 08 15:59:57.381756 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.381714    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"]
Sep 08 15:59:57.382136 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382122    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerName="istio-proxy"
Sep 08 15:59:57.382206 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382138    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerName="istio-proxy"
Sep 08 15:59:57.382206 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382152    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerName="istio-proxy"
Sep 08 15:59:57.382206 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382157    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerName="istio-proxy"
Sep 08 15:59:57.382206 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382165    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13241786-5d59-4845-ad4f-519af08979a5" containerName="payload-pre-processing"
Sep 08 15:59:57.382206 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382171    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="13241786-5d59-4845-ad4f-519af08979a5" containerName="payload-pre-processing"
Sep 08 15:59:57.382206 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382182    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47753cc9-7ca9-4b4e-88bd-86f698d147fa" containerName="payload-processing"
Sep 08 15:59:57.382206 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382188    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="47753cc9-7ca9-4b4e-88bd-86f698d147fa" containerName="payload-processing"
Sep 08 15:59:57.382206 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382198    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63db8c88-5dc3-4779-84c9-fc31de3c5833" containerName="maas-api"
Sep 08 15:59:57.382206 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382203    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="63db8c88-5dc3-4779-84c9-fc31de3c5833" containerName="maas-api"
Sep 08 15:59:57.382657 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382217    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c54675-ed87-4e97-a221-4e7fa3bb527f" containerName="main"
Sep 08 15:59:57.382657 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382222    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c54675-ed87-4e97-a221-4e7fa3bb527f" containerName="main"
Sep 08 15:59:57.382657 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382279    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerName="istio-proxy"
Sep 08 15:59:57.382657 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382286    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f37dbe8-1802-4cf9-a9a9-f0e107ca2b0d" containerName="istio-proxy"
Sep 08 15:59:57.382657 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382292    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="47753cc9-7ca9-4b4e-88bd-86f698d147fa" containerName="payload-processing"
Sep 08 15:59:57.382657 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382300    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c54675-ed87-4e97-a221-4e7fa3bb527f" containerName="main"
Sep 08 15:59:57.382657 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382307    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="63db8c88-5dc3-4779-84c9-fc31de3c5833" containerName="maas-api"
Sep 08 15:59:57.382657 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.382314    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="13241786-5d59-4845-ad4f-519af08979a5" containerName="payload-pre-processing"
Sep 08 15:59:57.386766 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.386745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.389785 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.389764    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-f50ef0-openshift-default-dockercfg-mp79z\""
Sep 08 15:59:57.400884 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.400862    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"]
Sep 08 15:59:57.573523 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.573486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-podinfo\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.573873 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.573542    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istiod-ca-cert\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.573873 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.573654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-envoy\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.573873 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.573693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-credential-socket\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.573873 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.573744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-workload-socket\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.573873 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.573781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-token\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.573873 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.573813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjq9t\" (UniqueName: \"kubernetes.io/projected/fd125bef-1cc2-455d-8cce-7663ee3ab48b-kube-api-access-zjq9t\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.573873 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.573854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-data\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.574093 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.573906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-workload-certs\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.674405 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.674373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-token\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.674563 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.674440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjq9t\" (UniqueName: \"kubernetes.io/projected/fd125bef-1cc2-455d-8cce-7663ee3ab48b-kube-api-access-zjq9t\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.674563 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.674483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-data\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.674563 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.674535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-workload-certs\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.674762 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.674577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-podinfo\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.674762 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.674615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istiod-ca-cert\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.674762 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.674687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-envoy\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.674762 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.674713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-credential-socket\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.674762 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.674749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-workload-socket\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.674994 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.674969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-workload-certs\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.675053 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.675019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-data\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.675123 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.675100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-credential-socket\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.675195 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.675175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-workload-socket\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.675323 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.675305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istiod-ca-cert\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.676822 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.676803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-envoy\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.677070 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.677053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-podinfo\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.682421 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.682384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-token\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.682421 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.682408    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjq9t\" (UniqueName: \"kubernetes.io/projected/fd125bef-1cc2-455d-8cce-7663ee3ab48b-kube-api-access-zjq9t\") pod \"e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") " pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.697401 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.697367    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:57.972335 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.972309    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"]
Sep 08 15:59:57.974337 ip-10-0-134-107 kubenswrapper[2581]: W0908 15:59:57.974312    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd125bef_1cc2_455d_8cce_7663ee3ab48b.slice/crio-e24406eca948e373fb3947061dc5ca59fd2017728b281f23f8224a21509b468d WatchSource:0}: Error finding container e24406eca948e373fb3947061dc5ca59fd2017728b281f23f8224a21509b468d: Status 404 returned error can't find the container with id e24406eca948e373fb3947061dc5ca59fd2017728b281f23f8224a21509b468d
Sep 08 15:59:57.976318 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.976285    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:59:57.976419 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.976345    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:59:57.976419 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:57.976374    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 15:59:58.626407 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:58.626356    2581 patch_prober.go:28] interesting pod/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 15:59:58.626782 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:58.626435    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 15:59:58.674788 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:58.674758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm" event={"ID":"fd125bef-1cc2-455d-8cce-7663ee3ab48b","Type":"ContainerStarted","Data":"b7eeb5af2a8c11f540d15b9b8134c73babdd69a8162bca03b55eedfd1104ed84"}
Sep 08 15:59:58.674923 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:58.674810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm" event={"ID":"fd125bef-1cc2-455d-8cce-7663ee3ab48b","Type":"ContainerStarted","Data":"e24406eca948e373fb3947061dc5ca59fd2017728b281f23f8224a21509b468d"}
Sep 08 15:59:58.695572 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:58.695525    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm" podStartSLOduration=1.695506151 podStartE2EDuration="1.695506151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:59:58.694243277 +0000 UTC m=+945.317935624" watchObservedRunningTime="2026-09-08 15:59:58.695506151 +0000 UTC m=+945.319198476"
Sep 08 15:59:58.698286 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:58.698266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 15:59:59.702960 ip-10-0-134-107 kubenswrapper[2581]: I0908 15:59:59.702933    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 16:00:00.142052 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.142022    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4"]
Sep 08 16:00:00.145585 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.145564    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8"]
Sep 08 16:00:00.145712 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.145678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4"
Sep 08 16:00:00.148329 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.148311    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65h2p\""
Sep 08 16:00:00.148920 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.148903    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8"
Sep 08 16:00:00.155871 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.155849    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8"]
Sep 08 16:00:00.158415 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.158379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4"
Sep 08 16:00:00.158572 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.158522    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4"]
Sep 08 16:00:00.167267 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.167246    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8"
Sep 08 16:00:00.450649 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.450622    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4"]
Sep 08 16:00:00.451940 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:00:00.451905    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee9f470_3a23_4904_878f_6833adcfa2a4.slice/crio-d5c3ee48e37ec03e2fdf83c59014e031fdfe21c48f816ffbdf318004f7537473 WatchSource:0}: Error finding container d5c3ee48e37ec03e2fdf83c59014e031fdfe21c48f816ffbdf318004f7537473: Status 404 returned error can't find the container with id d5c3ee48e37ec03e2fdf83c59014e031fdfe21c48f816ffbdf318004f7537473
Sep 08 16:00:00.453006 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.452979    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8"]
Sep 08 16:00:00.453145 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:00:00.453125    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5e8231_308e_473c_bacd_21c448e3a666.slice/crio-75f05b32d60423061ea72c09631e3b3a5044f09e17bf155bb6f3bb2ee1799227 WatchSource:0}: Error finding container 75f05b32d60423061ea72c09631e3b3a5044f09e17bf155bb6f3bb2ee1799227: Status 404 returned error can't find the container with id 75f05b32d60423061ea72c09631e3b3a5044f09e17bf155bb6f3bb2ee1799227
Sep 08 16:00:00.685888 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.685855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8" event={"ID":"5f5e8231-308e-473c-bacd-21c448e3a666","Type":"ContainerStarted","Data":"75f05b32d60423061ea72c09631e3b3a5044f09e17bf155bb6f3bb2ee1799227"}
Sep 08 16:00:00.687135 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.687112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4" event={"ID":"4ee9f470-3a23-4904-878f-6833adcfa2a4","Type":"ContainerStarted","Data":"d5c3ee48e37ec03e2fdf83c59014e031fdfe21c48f816ffbdf318004f7537473"}
Sep 08 16:00:00.687274 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.687256    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 16:00:00.688561 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:00.688543    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 16:00:01.693772 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.693742    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerID="fa48bbc6b2592b1526f6cdbcdaf9ab7268111c01adb82a4a17be8e058481defe" exitCode=0
Sep 08 16:00:01.694220 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.693811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" event={"ID":"6e983b15-0405-4370-a9c5-8e00e49c1f59","Type":"ContainerDied","Data":"fa48bbc6b2592b1526f6cdbcdaf9ab7268111c01adb82a4a17be8e058481defe"}
Sep 08 16:00:01.694220 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.693870    2581 scope.go:117] "RemoveContainer" containerID="350a95f9a83a51b0da7102712b5b1d1b068e0ffee9773bec10ce0fefe7af8734"
Sep 08 16:00:01.833635 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.833556    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 16:00:01.912760 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.911830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-credential-socket\") pod \"6e983b15-0405-4370-a9c5-8e00e49c1f59\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") "
Sep 08 16:00:01.912760 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.911877    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-workload-socket\") pod \"6e983b15-0405-4370-a9c5-8e00e49c1f59\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") "
Sep 08 16:00:01.912760 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.911917    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-token\") pod \"6e983b15-0405-4370-a9c5-8e00e49c1f59\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") "
Sep 08 16:00:01.912760 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.911960    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-data\") pod \"6e983b15-0405-4370-a9c5-8e00e49c1f59\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") "
Sep 08 16:00:01.912760 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.911990    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-workload-certs\") pod \"6e983b15-0405-4370-a9c5-8e00e49c1f59\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") "
Sep 08 16:00:01.912760 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.912017    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e983b15-0405-4370-a9c5-8e00e49c1f59-istiod-ca-cert\") pod \"6e983b15-0405-4370-a9c5-8e00e49c1f59\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") "
Sep 08 16:00:01.912760 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.912072    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-envoy\") pod \"6e983b15-0405-4370-a9c5-8e00e49c1f59\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") "
Sep 08 16:00:01.912760 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.912142    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-podinfo\") pod \"6e983b15-0405-4370-a9c5-8e00e49c1f59\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") "
Sep 08 16:00:01.912760 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.912203    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8n5f\" (UniqueName: \"kubernetes.io/projected/6e983b15-0405-4370-a9c5-8e00e49c1f59-kube-api-access-m8n5f\") pod \"6e983b15-0405-4370-a9c5-8e00e49c1f59\" (UID: \"6e983b15-0405-4370-a9c5-8e00e49c1f59\") "
Sep 08 16:00:01.912760 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.912259    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6e983b15-0405-4370-a9c5-8e00e49c1f59" (UID: "6e983b15-0405-4370-a9c5-8e00e49c1f59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:01.912760 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.912539    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-credential-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:01.912760 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.912582    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e983b15-0405-4370-a9c5-8e00e49c1f59-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6e983b15-0405-4370-a9c5-8e00e49c1f59" (UID: "6e983b15-0405-4370-a9c5-8e00e49c1f59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:00:01.913522 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.912837    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-data" (OuterVolumeSpecName: "istio-data") pod "6e983b15-0405-4370-a9c5-8e00e49c1f59" (UID: "6e983b15-0405-4370-a9c5-8e00e49c1f59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:01.913522 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.912960    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6e983b15-0405-4370-a9c5-8e00e49c1f59" (UID: "6e983b15-0405-4370-a9c5-8e00e49c1f59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:01.913522 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.913038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6e983b15-0405-4370-a9c5-8e00e49c1f59" (UID: "6e983b15-0405-4370-a9c5-8e00e49c1f59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:01.915291 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.915263    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-token" (OuterVolumeSpecName: "istio-token") pod "6e983b15-0405-4370-a9c5-8e00e49c1f59" (UID: "6e983b15-0405-4370-a9c5-8e00e49c1f59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:00:01.915546 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.915520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6e983b15-0405-4370-a9c5-8e00e49c1f59" (UID: "6e983b15-0405-4370-a9c5-8e00e49c1f59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:01.918498 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.918457    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e983b15-0405-4370-a9c5-8e00e49c1f59-kube-api-access-m8n5f" (OuterVolumeSpecName: "kube-api-access-m8n5f") pod "6e983b15-0405-4370-a9c5-8e00e49c1f59" (UID: "6e983b15-0405-4370-a9c5-8e00e49c1f59"). InnerVolumeSpecName "kube-api-access-m8n5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:00:01.918993 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:01.918957    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6e983b15-0405-4370-a9c5-8e00e49c1f59" (UID: "6e983b15-0405-4370-a9c5-8e00e49c1f59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:00:02.013713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.013565    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-podinfo\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:02.013713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.013597    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8n5f\" (UniqueName: \"kubernetes.io/projected/6e983b15-0405-4370-a9c5-8e00e49c1f59-kube-api-access-m8n5f\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:02.013713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.013612    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-workload-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:02.013713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.013625    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-token\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:02.013713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.013640    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-data\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:02.013713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.013652    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-workload-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:02.013713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.013663    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e983b15-0405-4370-a9c5-8e00e49c1f59-istiod-ca-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:02.013713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.013691    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e983b15-0405-4370-a9c5-8e00e49c1f59-istio-envoy\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:02.666273 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.666241    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"]
Sep 08 16:00:02.666786 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.666761    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerName="istio-proxy"
Sep 08 16:00:02.666786 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.666785    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerName="istio-proxy"
Sep 08 16:00:02.666985 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.666903    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerName="istio-proxy"
Sep 08 16:00:02.666985 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.666914    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerName="istio-proxy"
Sep 08 16:00:02.673508 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.673490    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.678306 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.678286    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f50ef0\"/\"openshift-service-ca.crt\""
Sep 08 16:00:02.678416 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.678300    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f50ef0\"/\"kube-root-ca.crt\""
Sep 08 16:00:02.679172 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.679138    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f50ef0\"/\"noten-model-f50ef0-kserve-self-signed-certs\""
Sep 08 16:00:02.679260 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.679178    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f50ef0\"/\"default-dockercfg-2d5dc\""
Sep 08 16:00:02.684708 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.684686    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"]
Sep 08 16:00:02.701878 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.701847    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"
Sep 08 16:00:02.702221 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.701883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v" event={"ID":"6e983b15-0405-4370-a9c5-8e00e49c1f59","Type":"ContainerDied","Data":"c5edf1859cfc3ca5b188083b790dc20cf22b9be16a86a7ccaf844bc7d9d92a0d"}
Sep 08 16:00:02.702221 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.701951    2581 scope.go:117] "RemoveContainer" containerID="fa48bbc6b2592b1526f6cdbcdaf9ab7268111c01adb82a4a17be8e058481defe"
Sep 08 16:00:02.719595 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.719575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-dshm\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.719699 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.719619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-model-cache\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.719699 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.719650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-home\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.719819 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.719771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn7wp\" (UniqueName: \"kubernetes.io/projected/0da03951-86b1-4c02-8db9-2beec8adbe6a-kube-api-access-kn7wp\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.719881 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.719829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-tmp-dir\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.719881 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.719856    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da03951-86b1-4c02-8db9-2beec8adbe6a-tls-certs\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.729356 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.729334    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"]
Sep 08 16:00:02.736176 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.736148    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-3b98e2-openshift-default-5947d6bcb4-j6v4v"]
Sep 08 16:00:02.821068 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.821034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-dshm\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.821235 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.821203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-model-cache\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.821305 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.821257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-home\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.821366 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.821331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7wp\" (UniqueName: \"kubernetes.io/projected/0da03951-86b1-4c02-8db9-2beec8adbe6a-kube-api-access-kn7wp\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.821456 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.821433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-tmp-dir\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.821522 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.821466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da03951-86b1-4c02-8db9-2beec8adbe6a-tls-certs\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.821581 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.821563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-model-cache\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.821650 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.821626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-home\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.821789 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.821765    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-tmp-dir\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.823701 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.823670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-dshm\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.824113 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.824092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da03951-86b1-4c02-8db9-2beec8adbe6a-tls-certs\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.829679 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.829656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7wp\" (UniqueName: \"kubernetes.io/projected/0da03951-86b1-4c02-8db9-2beec8adbe6a-kube-api-access-kn7wp\") pod \"noten-model-f50ef0-kserve-6f95f69f75-tk26j\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") " pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:02.984981 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:02.984906    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:03.384780 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:03.384757    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"]
Sep 08 16:00:03.388477 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:00:03.388452    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0da03951_86b1_4c02_8db9_2beec8adbe6a.slice/crio-b922ba4bf1c6eb1d4c45dea4e3c98dd4d23dea6108b8e5451794f1717929a312 WatchSource:0}: Error finding container b922ba4bf1c6eb1d4c45dea4e3c98dd4d23dea6108b8e5451794f1717929a312: Status 404 returned error can't find the container with id b922ba4bf1c6eb1d4c45dea4e3c98dd4d23dea6108b8e5451794f1717929a312
Sep 08 16:00:03.709105 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:03.709075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4" event={"ID":"4ee9f470-3a23-4904-878f-6833adcfa2a4","Type":"ContainerStarted","Data":"01dd5c0cba5c9ff8571068f536bde815f0aaa567172816f02c1991f0de555799"}
Sep 08 16:00:03.710791 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:03.710766    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j" event={"ID":"0da03951-86b1-4c02-8db9-2beec8adbe6a","Type":"ContainerStarted","Data":"4144d6d503882d81e3de4a0e97e7140eb1587739d853aa40963a63ffa46afce5"}
Sep 08 16:00:03.710930 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:03.710802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j" event={"ID":"0da03951-86b1-4c02-8db9-2beec8adbe6a","Type":"ContainerStarted","Data":"b922ba4bf1c6eb1d4c45dea4e3c98dd4d23dea6108b8e5451794f1717929a312"}
Sep 08 16:00:03.712474 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:03.712445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8" event={"ID":"5f5e8231-308e-473c-bacd-21c448e3a666","Type":"ContainerStarted","Data":"87b5f097dfa69fba669765e59beb5f8628b892f04b7671791c6a9c5e698187d6"}
Sep 08 16:00:03.725074 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:03.725033    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4" podStartSLOduration=0.978577337 podStartE2EDuration="3.725020672s" totalImagesPullingTime="2.746443335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:00:03.723334022 +0000 UTC m=+950.347026345" watchObservedRunningTime="2026-09-08 16:00:03.725020672 +0000 UTC m=+950.348712996"
Sep 08 16:00:03.740667 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:03.740624    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j" podStartSLOduration=1.740605762 podStartE2EDuration="1.740605762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:00:03.739016058 +0000 UTC m=+950.362708382" watchObservedRunningTime="2026-09-08 16:00:03.740605762 +0000 UTC m=+950.364298087"
Sep 08 16:00:03.754013 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:03.753973    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8" podStartSLOduration=1.003674314 podStartE2EDuration="3.753961527s" totalImagesPullingTime="2.750287213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:00:03.752170045 +0000 UTC m=+950.375862372" watchObservedRunningTime="2026-09-08 16:00:03.753961527 +0000 UTC m=+950.377653851"
Sep 08 16:00:03.901684 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:03.901653    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" path="/var/lib/kubelet/pods/6e983b15-0405-4370-a9c5-8e00e49c1f59/volumes"
Sep 08 16:00:04.720298 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:04.720264    2581 generic.go:356] "Generic (PLEG): container finished" podID="4ee9f470-3a23-4904-878f-6833adcfa2a4" containerID="01dd5c0cba5c9ff8571068f536bde815f0aaa567172816f02c1991f0de555799" exitCode=0
Sep 08 16:00:04.720709 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:04.720350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4" event={"ID":"4ee9f470-3a23-4904-878f-6833adcfa2a4","Type":"ContainerDied","Data":"01dd5c0cba5c9ff8571068f536bde815f0aaa567172816f02c1991f0de555799"}
Sep 08 16:00:04.721955 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:04.721933    2581 generic.go:356] "Generic (PLEG): container finished" podID="5f5e8231-308e-473c-bacd-21c448e3a666" containerID="87b5f097dfa69fba669765e59beb5f8628b892f04b7671791c6a9c5e698187d6" exitCode=0
Sep 08 16:00:04.722081 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:04.721990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8" event={"ID":"5f5e8231-308e-473c-bacd-21c448e3a666","Type":"ContainerDied","Data":"87b5f097dfa69fba669765e59beb5f8628b892f04b7671791c6a9c5e698187d6"}
Sep 08 16:00:05.856942 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:05.856915    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8"
Sep 08 16:00:05.892488 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:05.892466    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4"
Sep 08 16:00:06.732964 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:06.732930    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4"
Sep 08 16:00:06.733144 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:06.732962    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4" event={"ID":"4ee9f470-3a23-4904-878f-6833adcfa2a4","Type":"ContainerDied","Data":"d5c3ee48e37ec03e2fdf83c59014e031fdfe21c48f816ffbdf318004f7537473"}
Sep 08 16:00:06.733144 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:06.733015    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5c3ee48e37ec03e2fdf83c59014e031fdfe21c48f816ffbdf318004f7537473"
Sep 08 16:00:06.734356 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:06.734336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8"
Sep 08 16:00:06.734480 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:06.734336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8" event={"ID":"5f5e8231-308e-473c-bacd-21c448e3a666","Type":"ContainerDied","Data":"75f05b32d60423061ea72c09631e3b3a5044f09e17bf155bb6f3bb2ee1799227"}
Sep 08 16:00:06.734480 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:06.734462    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75f05b32d60423061ea72c09631e3b3a5044f09e17bf155bb6f3bb2ee1799227"
Sep 08 16:00:08.457709 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.457653    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"]
Sep 08 16:00:08.458200 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.458179    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5e8231-308e-473c-bacd-21c448e3a666" containerName="cleanup"
Sep 08 16:00:08.458295 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.458204    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5e8231-308e-473c-bacd-21c448e3a666" containerName="cleanup"
Sep 08 16:00:08.458295 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.458221    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee9f470-3a23-4904-878f-6833adcfa2a4" containerName="cleanup"
Sep 08 16:00:08.458295 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.458229    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee9f470-3a23-4904-878f-6833adcfa2a4" containerName="cleanup"
Sep 08 16:00:08.458295 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.458249    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerName="istio-proxy"
Sep 08 16:00:08.458295 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.458257    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e983b15-0405-4370-a9c5-8e00e49c1f59" containerName="istio-proxy"
Sep 08 16:00:08.458559 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.458428    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ee9f470-3a23-4904-878f-6833adcfa2a4" containerName="cleanup"
Sep 08 16:00:08.458559 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.458446    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f5e8231-308e-473c-bacd-21c448e3a666" containerName="cleanup"
Sep 08 16:00:08.461643 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.461621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.464268 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.464245    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cf23f7aa-openshift-default-dockercfg-d84qz\""
Sep 08 16:00:08.473287 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.473266    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"]
Sep 08 16:00:08.570932 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.570825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-envoy\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.570932 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.570918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-data\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.571239 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.571026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istiod-ca-cert\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.571239 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.571111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgqq\" (UniqueName: \"kubernetes.io/projected/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-kube-api-access-8hgqq\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.571239 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.571148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-credential-socket\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.571239 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.571182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-workload-socket\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.571239 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.571199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-token\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.571239 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.571227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-workload-certs\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.571629 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.571249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-podinfo\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.672412 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.672369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istiod-ca-cert\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.672539 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.672459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgqq\" (UniqueName: \"kubernetes.io/projected/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-kube-api-access-8hgqq\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.672539 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.672491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-credential-socket\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.672539 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.672527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-workload-socket\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.672656 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.672550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-token\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.672656 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.672588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-workload-certs\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.672656 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.672621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-podinfo\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.672808 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.672654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-envoy\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.672808 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.672753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-data\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.672972 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.672923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-workload-socket\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.673104 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.673006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-workload-certs\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.673104 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.673088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-credential-socket\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.673201 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.673157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-data\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.673309 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.673292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istiod-ca-cert\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.675043 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.675021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-envoy\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.675163 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.675142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-podinfo\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.680851 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.680830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-token\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.680928 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.680835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgqq\" (UniqueName: \"kubernetes.io/projected/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-kube-api-access-8hgqq\") pod \"e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") " pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.773754 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.773697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:08.961926 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:08.961894    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"]
Sep 08 16:00:09.052244 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:09.052213    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"]
Sep 08 16:00:09.058147 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:00:09.057669    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8fc74fc_1e8e_4a97_af3f_a6b3a458db75.slice/crio-25f93043750b585c3dfc6af470035de3c7b8399d0e02b2edf901716f19d4eeea WatchSource:0}: Error finding container 25f93043750b585c3dfc6af470035de3c7b8399d0e02b2edf901716f19d4eeea: Status 404 returned error can't find the container with id 25f93043750b585c3dfc6af470035de3c7b8399d0e02b2edf901716f19d4eeea
Sep 08 16:00:09.060135 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:09.060097    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:00:09.060284 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:09.060179    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:00:09.060284 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:09.060224    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:00:09.090667 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:09.090637    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"]
Sep 08 16:00:09.090912 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:09.090858    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm" podUID="fd125bef-1cc2-455d-8cce-7663ee3ab48b" containerName="istio-proxy" containerID="cri-o://b7eeb5af2a8c11f540d15b9b8134c73babdd69a8162bca03b55eedfd1104ed84" gracePeriod=30
Sep 08 16:00:09.750068 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:09.750032    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs" event={"ID":"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75","Type":"ContainerStarted","Data":"f0a6d423c1f7414e7bf0a420f5db1e26ac5822e3f98a74f4223fe37ccf26c7fe"}
Sep 08 16:00:09.750460 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:09.750089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs" event={"ID":"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75","Type":"ContainerStarted","Data":"25f93043750b585c3dfc6af470035de3c7b8399d0e02b2edf901716f19d4eeea"}
Sep 08 16:00:09.770281 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:09.770235    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs" podStartSLOduration=1.770220994 podStartE2EDuration="1.770220994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:00:09.768583032 +0000 UTC m=+956.392275367" watchObservedRunningTime="2026-09-08 16:00:09.770220994 +0000 UTC m=+956.393913319"
Sep 08 16:00:09.774468 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:09.774445    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:10.779340 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:10.779310    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:11.760330 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:11.760294    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:11.761281 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:11.761263    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:12.788458 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:12.788429    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"]
Sep 08 16:00:12.791795 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:12.791770    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:12.795197 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:12.795067    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cf23f7aa-serving-cert\""
Sep 08 16:00:12.795197 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:12.795146    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 16:00:12.795197 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:12.795148    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmx7t\""
Sep 08 16:00:12.795825 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:12.795789    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:00:12.795947 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:12.795898    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 16:00:12.824411 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:12.824370    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"]
Sep 08 16:00:12.914652 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:12.914611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhc4\" (UniqueName: \"kubernetes.io/projected/316b66dc-44a2-49d3-a684-5d0331748315-kube-api-access-hfhc4\") pod \"maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:12.914828 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:12.914719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-metrics-certs\") pod \"maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:12.914828 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:12.914771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-maas-api-tls\") pod \"maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:12.914828 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:12.914810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316b66dc-44a2-49d3-a684-5d0331748315-openshift-service-ca\") pod \"maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:12.985485 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:12.985456    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:13.015847 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.015822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316b66dc-44a2-49d3-a684-5d0331748315-openshift-service-ca\") pod \"maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:13.015953 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.015869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhc4\" (UniqueName: \"kubernetes.io/projected/316b66dc-44a2-49d3-a684-5d0331748315-kube-api-access-hfhc4\") pod \"maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:13.015953 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.015922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-metrics-certs\") pod \"maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:13.016090 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.015963    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-maas-api-tls\") pod \"maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:13.016282 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:00:13.016264    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-serving-cert: secret "maas-api-e2e-mt-cf23f7aa-serving-cert" not found
Sep 08 16:00:13.016356 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:00:13.016317    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-maas-api-tls podName:316b66dc-44a2-49d3-a684-5d0331748315 nodeName:}" failed. No retries permitted until 2026-09-08 16:00:13.516304078 +0000 UTC m=+960.139996380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-maas-api-tls") pod "maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w" (UID: "316b66dc-44a2-49d3-a684-5d0331748315") : secret "maas-api-e2e-mt-cf23f7aa-serving-cert" not found
Sep 08 16:00:13.016597 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.016575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316b66dc-44a2-49d3-a684-5d0331748315-openshift-service-ca\") pod \"maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:13.018800 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.018774    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-metrics-certs\") pod \"maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:13.024334 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.024308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhc4\" (UniqueName: \"kubernetes.io/projected/316b66dc-44a2-49d3-a684-5d0331748315-kube-api-access-hfhc4\") pod \"maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:13.521319 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.521287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-maas-api-tls\") pod \"maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:13.523663 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.523643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-maas-api-tls\") pod \"maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:13.540717 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.540683    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"]
Sep 08 16:00:13.544262 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.544241    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"
Sep 08 16:00:13.546485 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.546466    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cf23f7aa-dockercfg-m96vt\""
Sep 08 16:00:13.546581 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.546506    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cf23f7aa\""
Sep 08 16:00:13.558345 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.558323    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"]
Sep 08 16:00:13.565009 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.564981    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"]
Sep 08 16:00:13.568297 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.568278    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"
Sep 08 16:00:13.580460 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.580441    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"]
Sep 08 16:00:13.621863 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.621828    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc931a5c-7713-4bc6-bb9f-25db71c033d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd\" (UID: \"bc931a5c-7713-4bc6-bb9f-25db71c033d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"
Sep 08 16:00:13.621978 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.621872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gxc2\" (UniqueName: \"kubernetes.io/projected/bc931a5c-7713-4bc6-bb9f-25db71c033d1-kube-api-access-6gxc2\") pod \"payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd\" (UID: \"bc931a5c-7713-4bc6-bb9f-25db71c033d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"
Sep 08 16:00:13.621978 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.621890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c9278c-3e43-4e62-b239-9cfcbce372ff-plugins-config-volume\") pod \"payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72\" (UID: \"c4c9278c-3e43-4e62-b239-9cfcbce372ff\") " pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"
Sep 08 16:00:13.621978 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.621934    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k44xh\" (UniqueName: \"kubernetes.io/projected/c4c9278c-3e43-4e62-b239-9cfcbce372ff-kube-api-access-k44xh\") pod \"payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72\" (UID: \"c4c9278c-3e43-4e62-b239-9cfcbce372ff\") " pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"
Sep 08 16:00:13.702897 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.702863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:13.723280 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.723251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc931a5c-7713-4bc6-bb9f-25db71c033d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd\" (UID: \"bc931a5c-7713-4bc6-bb9f-25db71c033d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"
Sep 08 16:00:13.723430 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.723288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gxc2\" (UniqueName: \"kubernetes.io/projected/bc931a5c-7713-4bc6-bb9f-25db71c033d1-kube-api-access-6gxc2\") pod \"payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd\" (UID: \"bc931a5c-7713-4bc6-bb9f-25db71c033d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"
Sep 08 16:00:13.723430 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.723311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c9278c-3e43-4e62-b239-9cfcbce372ff-plugins-config-volume\") pod \"payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72\" (UID: \"c4c9278c-3e43-4e62-b239-9cfcbce372ff\") " pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"
Sep 08 16:00:13.723430 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.723346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k44xh\" (UniqueName: \"kubernetes.io/projected/c4c9278c-3e43-4e62-b239-9cfcbce372ff-kube-api-access-k44xh\") pod \"payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72\" (UID: \"c4c9278c-3e43-4e62-b239-9cfcbce372ff\") " pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"
Sep 08 16:00:13.724133 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.724106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc931a5c-7713-4bc6-bb9f-25db71c033d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd\" (UID: \"bc931a5c-7713-4bc6-bb9f-25db71c033d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"
Sep 08 16:00:13.724234 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.724132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c9278c-3e43-4e62-b239-9cfcbce372ff-plugins-config-volume\") pod \"payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72\" (UID: \"c4c9278c-3e43-4e62-b239-9cfcbce372ff\") " pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"
Sep 08 16:00:13.731132 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.731112    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gxc2\" (UniqueName: \"kubernetes.io/projected/bc931a5c-7713-4bc6-bb9f-25db71c033d1-kube-api-access-6gxc2\") pod \"payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd\" (UID: \"bc931a5c-7713-4bc6-bb9f-25db71c033d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"
Sep 08 16:00:13.731206 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.731149    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k44xh\" (UniqueName: \"kubernetes.io/projected/c4c9278c-3e43-4e62-b239-9cfcbce372ff-kube-api-access-k44xh\") pod \"payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72\" (UID: \"c4c9278c-3e43-4e62-b239-9cfcbce372ff\") " pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"
Sep 08 16:00:13.856869 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.856803    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cf23f7aa-dockercfg-m96vt\""
Sep 08 16:00:13.865559 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.865537    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"
Sep 08 16:00:13.878713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.878693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"
Sep 08 16:00:13.985782 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:13.985751    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"]
Sep 08 16:00:13.986791 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:00:13.986757    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316b66dc_44a2_49d3_a684_5d0331748315.slice/crio-4c0427e122151c19ef61f68ac7adcfd9cc732bc45c4ac525d46da394fb6a42bb WatchSource:0}: Error finding container 4c0427e122151c19ef61f68ac7adcfd9cc732bc45c4ac525d46da394fb6a42bb: Status 404 returned error can't find the container with id 4c0427e122151c19ef61f68ac7adcfd9cc732bc45c4ac525d46da394fb6a42bb
Sep 08 16:00:14.141126 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.141102    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"]
Sep 08 16:00:14.174540 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.174519    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"]
Sep 08 16:00:14.186347 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:00:14.186315    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c9278c_3e43_4e62_b239_9cfcbce372ff.slice/crio-ca2d47ae98343aa0f8dde6dc37c4b8b5b4de7a01e3dd3d8acc6073efc6953fce WatchSource:0}: Error finding container ca2d47ae98343aa0f8dde6dc37c4b8b5b4de7a01e3dd3d8acc6073efc6953fce: Status 404 returned error can't find the container with id ca2d47ae98343aa0f8dde6dc37c4b8b5b4de7a01e3dd3d8acc6073efc6953fce
Sep 08 16:00:14.186542 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:00:14.186521    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc931a5c_7713_4bc6_bb9f_25db71c033d1.slice/crio-d13c479ca96e494a8733009ca6cd2b953f9c0c4b2b490973ff51f646a0ccf641 WatchSource:0}: Error finding container d13c479ca96e494a8733009ca6cd2b953f9c0c4b2b490973ff51f646a0ccf641: Status 404 returned error can't find the container with id d13c479ca96e494a8733009ca6cd2b953f9c0c4b2b490973ff51f646a0ccf641
Sep 08 16:00:14.342123 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.342096    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 16:00:14.438309 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.438279    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-token\") pod \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") "
Sep 08 16:00:14.438515 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.438351    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjq9t\" (UniqueName: \"kubernetes.io/projected/fd125bef-1cc2-455d-8cce-7663ee3ab48b-kube-api-access-zjq9t\") pod \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") "
Sep 08 16:00:14.438515 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.438387    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-workload-socket\") pod \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") "
Sep 08 16:00:14.438515 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.438454    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istiod-ca-cert\") pod \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") "
Sep 08 16:00:14.438515 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.438496    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-credential-socket\") pod \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") "
Sep 08 16:00:14.438736 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.438536    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-envoy\") pod \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") "
Sep 08 16:00:14.438736 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.438568    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-podinfo\") pod \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") "
Sep 08 16:00:14.438736 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.438598    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-data\") pod \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") "
Sep 08 16:00:14.438736 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.438633    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-workload-certs\") pod \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\" (UID: \"fd125bef-1cc2-455d-8cce-7663ee3ab48b\") "
Sep 08 16:00:14.438944 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.438737    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd125bef-1cc2-455d-8cce-7663ee3ab48b" (UID: "fd125bef-1cc2-455d-8cce-7663ee3ab48b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:14.438944 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.438750    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd125bef-1cc2-455d-8cce-7663ee3ab48b" (UID: "fd125bef-1cc2-455d-8cce-7663ee3ab48b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:14.438944 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.438855    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd125bef-1cc2-455d-8cce-7663ee3ab48b" (UID: "fd125bef-1cc2-455d-8cce-7663ee3ab48b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:00:14.439113 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.438988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd125bef-1cc2-455d-8cce-7663ee3ab48b" (UID: "fd125bef-1cc2-455d-8cce-7663ee3ab48b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:14.439113 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.439007    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd125bef-1cc2-455d-8cce-7663ee3ab48b" (UID: "fd125bef-1cc2-455d-8cce-7663ee3ab48b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:14.439220 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.439123    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-workload-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:14.439220 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.439141    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istiod-ca-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:14.439220 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.439155    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-credential-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:14.439220 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.439167    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-data\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:14.439220 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.439182    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-workload-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:14.440597 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.440572    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd125bef-1cc2-455d-8cce-7663ee3ab48b" (UID: "fd125bef-1cc2-455d-8cce-7663ee3ab48b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:00:14.440844 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.440818    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd125bef-1cc2-455d-8cce-7663ee3ab48b-kube-api-access-zjq9t" (OuterVolumeSpecName: "kube-api-access-zjq9t") pod "fd125bef-1cc2-455d-8cce-7663ee3ab48b" (UID: "fd125bef-1cc2-455d-8cce-7663ee3ab48b"). InnerVolumeSpecName "kube-api-access-zjq9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:00:14.441158 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.441143    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd125bef-1cc2-455d-8cce-7663ee3ab48b" (UID: "fd125bef-1cc2-455d-8cce-7663ee3ab48b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:14.441226 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.441203    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd125bef-1cc2-455d-8cce-7663ee3ab48b" (UID: "fd125bef-1cc2-455d-8cce-7663ee3ab48b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:00:14.540494 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.540470    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-envoy\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:14.540494 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.540492    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-podinfo\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:14.540625 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.540500    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd125bef-1cc2-455d-8cce-7663ee3ab48b-istio-token\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:14.540625 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.540509    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjq9t\" (UniqueName: \"kubernetes.io/projected/fd125bef-1cc2-455d-8cce-7663ee3ab48b-kube-api-access-zjq9t\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:14.779239 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.779141    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w" event={"ID":"316b66dc-44a2-49d3-a684-5d0331748315","Type":"ContainerStarted","Data":"4c0427e122151c19ef61f68ac7adcfd9cc732bc45c4ac525d46da394fb6a42bb"}
Sep 08 16:00:14.780578 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.780549    2581 generic.go:356] "Generic (PLEG): container finished" podID="fd125bef-1cc2-455d-8cce-7663ee3ab48b" containerID="b7eeb5af2a8c11f540d15b9b8134c73babdd69a8162bca03b55eedfd1104ed84" exitCode=0
Sep 08 16:00:14.780709 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.780599    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"
Sep 08 16:00:14.780709 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.780629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm" event={"ID":"fd125bef-1cc2-455d-8cce-7663ee3ab48b","Type":"ContainerDied","Data":"b7eeb5af2a8c11f540d15b9b8134c73babdd69a8162bca03b55eedfd1104ed84"}
Sep 08 16:00:14.780709 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.780697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm" event={"ID":"fd125bef-1cc2-455d-8cce-7663ee3ab48b","Type":"ContainerDied","Data":"e24406eca948e373fb3947061dc5ca59fd2017728b281f23f8224a21509b468d"}
Sep 08 16:00:14.780864 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.780732    2581 scope.go:117] "RemoveContainer" containerID="b7eeb5af2a8c11f540d15b9b8134c73babdd69a8162bca03b55eedfd1104ed84"
Sep 08 16:00:14.782894 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.782384    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72" event={"ID":"c4c9278c-3e43-4e62-b239-9cfcbce372ff","Type":"ContainerStarted","Data":"1221ab7038011681f2aa067ff73d125b2366ea42b77d3cdd484ccb435160afc2"}
Sep 08 16:00:14.783041 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.782916    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"
Sep 08 16:00:14.783041 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.782940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72" event={"ID":"c4c9278c-3e43-4e62-b239-9cfcbce372ff","Type":"ContainerStarted","Data":"ca2d47ae98343aa0f8dde6dc37c4b8b5b4de7a01e3dd3d8acc6073efc6953fce"}
Sep 08 16:00:14.784295 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.784269    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd" event={"ID":"bc931a5c-7713-4bc6-bb9f-25db71c033d1","Type":"ContainerStarted","Data":"59fb0798750d8be3e922d3a0ec89e44507248c23683474f458fbaba22eedad89"}
Sep 08 16:00:14.784422 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.784316    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd" event={"ID":"bc931a5c-7713-4bc6-bb9f-25db71c033d1","Type":"ContainerStarted","Data":"d13c479ca96e494a8733009ca6cd2b953f9c0c4b2b490973ff51f646a0ccf641"}
Sep 08 16:00:14.784422 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.784374    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"
Sep 08 16:00:14.792058 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.792042    2581 scope.go:117] "RemoveContainer" containerID="b7eeb5af2a8c11f540d15b9b8134c73babdd69a8162bca03b55eedfd1104ed84"
Sep 08 16:00:14.792312 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:00:14.792294    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7eeb5af2a8c11f540d15b9b8134c73babdd69a8162bca03b55eedfd1104ed84\": container with ID starting with b7eeb5af2a8c11f540d15b9b8134c73babdd69a8162bca03b55eedfd1104ed84 not found: ID does not exist" containerID="b7eeb5af2a8c11f540d15b9b8134c73babdd69a8162bca03b55eedfd1104ed84"
Sep 08 16:00:14.792380 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.792317    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7eeb5af2a8c11f540d15b9b8134c73babdd69a8162bca03b55eedfd1104ed84"} err="failed to get container status \"b7eeb5af2a8c11f540d15b9b8134c73babdd69a8162bca03b55eedfd1104ed84\": rpc error: code = NotFound desc = could not find container \"b7eeb5af2a8c11f540d15b9b8134c73babdd69a8162bca03b55eedfd1104ed84\": container with ID starting with b7eeb5af2a8c11f540d15b9b8134c73babdd69a8162bca03b55eedfd1104ed84 not found: ID does not exist"
Sep 08 16:00:14.802045 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.801852    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72" podStartSLOduration=1.801837017 podStartE2EDuration="1.801837017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:00:14.800804659 +0000 UTC m=+961.424496982" watchObservedRunningTime="2026-09-08 16:00:14.801837017 +0000 UTC m=+961.425529342"
Sep 08 16:00:14.818125 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.818085    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd" podStartSLOduration=1.818072336 podStartE2EDuration="1.818072336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:00:14.815794177 +0000 UTC m=+961.439486503" watchObservedRunningTime="2026-09-08 16:00:14.818072336 +0000 UTC m=+961.441764659"
Sep 08 16:00:14.829787 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.829767    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"]
Sep 08 16:00:14.835709 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:14.835688    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-f50ef0-openshift-default-67d6899448-5b4gm"]
Sep 08 16:00:15.792291 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:15.792202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w" event={"ID":"316b66dc-44a2-49d3-a684-5d0331748315","Type":"ContainerStarted","Data":"e3864ca4d945dbda0e8caf2184a541a653ecb4e2b066c78a0a0bc41cee967c83"}
Sep 08 16:00:15.811182 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:15.811132    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w" podStartSLOduration=2.379370462 podStartE2EDuration="3.811118878s" totalImagesPullingTime="1.431748416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:00:15.809418008 +0000 UTC m=+962.433110332" watchObservedRunningTime="2026-09-08 16:00:15.811118878 +0000 UTC m=+962.434811212"
Sep 08 16:00:15.901003 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:15.900974    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd125bef-1cc2-455d-8cce-7663ee3ab48b" path="/var/lib/kubelet/pods/fd125bef-1cc2-455d-8cce-7663ee3ab48b/volumes"
Sep 08 16:00:16.797826 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:16.797794    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:22.810124 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:22.810092    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:23.985999 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:23.985958    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j" podUID="0da03951-86b1-4c02-8db9-2beec8adbe6a" containerName="main" containerID="cri-o://4144d6d503882d81e3de4a0e97e7140eb1587739d853aa40963a63ffa46afce5" gracePeriod=45
Sep 08 16:00:24.258190 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.258165    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:24.334409 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.334364    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da03951-86b1-4c02-8db9-2beec8adbe6a-tls-certs\") pod \"0da03951-86b1-4c02-8db9-2beec8adbe6a\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") "
Sep 08 16:00:24.334564 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.334499    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn7wp\" (UniqueName: \"kubernetes.io/projected/0da03951-86b1-4c02-8db9-2beec8adbe6a-kube-api-access-kn7wp\") pod \"0da03951-86b1-4c02-8db9-2beec8adbe6a\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") "
Sep 08 16:00:24.334564 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.334544    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-dshm\") pod \"0da03951-86b1-4c02-8db9-2beec8adbe6a\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") "
Sep 08 16:00:24.334673 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.334620    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-tmp-dir\") pod \"0da03951-86b1-4c02-8db9-2beec8adbe6a\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") "
Sep 08 16:00:24.334673 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.334655    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-model-cache\") pod \"0da03951-86b1-4c02-8db9-2beec8adbe6a\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") "
Sep 08 16:00:24.334789 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.334687    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-home\") pod \"0da03951-86b1-4c02-8db9-2beec8adbe6a\" (UID: \"0da03951-86b1-4c02-8db9-2beec8adbe6a\") "
Sep 08 16:00:24.334862 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.334833    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0da03951-86b1-4c02-8db9-2beec8adbe6a" (UID: "0da03951-86b1-4c02-8db9-2beec8adbe6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:24.334949 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.334920    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "0da03951-86b1-4c02-8db9-2beec8adbe6a" (UID: "0da03951-86b1-4c02-8db9-2beec8adbe6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:24.335066 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.335039    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-home" (OuterVolumeSpecName: "home") pod "0da03951-86b1-4c02-8db9-2beec8adbe6a" (UID: "0da03951-86b1-4c02-8db9-2beec8adbe6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:24.335136 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.335115    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-tmp-dir\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:24.335173 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.335143    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-model-cache\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:24.336841 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.336817    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-dshm" (OuterVolumeSpecName: "dshm") pod "0da03951-86b1-4c02-8db9-2beec8adbe6a" (UID: "0da03951-86b1-4c02-8db9-2beec8adbe6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:00:24.336919 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.336840    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da03951-86b1-4c02-8db9-2beec8adbe6a-kube-api-access-kn7wp" (OuterVolumeSpecName: "kube-api-access-kn7wp") pod "0da03951-86b1-4c02-8db9-2beec8adbe6a" (UID: "0da03951-86b1-4c02-8db9-2beec8adbe6a"). InnerVolumeSpecName "kube-api-access-kn7wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:00:24.336919 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.336853    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da03951-86b1-4c02-8db9-2beec8adbe6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0da03951-86b1-4c02-8db9-2beec8adbe6a" (UID: "0da03951-86b1-4c02-8db9-2beec8adbe6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:00:24.436571 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.436542    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn7wp\" (UniqueName: \"kubernetes.io/projected/0da03951-86b1-4c02-8db9-2beec8adbe6a-kube-api-access-kn7wp\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:24.436571 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.436564    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-dshm\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:24.436571 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.436575    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da03951-86b1-4c02-8db9-2beec8adbe6a-home\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:24.436763 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.436583    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da03951-86b1-4c02-8db9-2beec8adbe6a-tls-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:24.844562 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.844530    2581 generic.go:356] "Generic (PLEG): container finished" podID="0da03951-86b1-4c02-8db9-2beec8adbe6a" containerID="4144d6d503882d81e3de4a0e97e7140eb1587739d853aa40963a63ffa46afce5" exitCode=0
Sep 08 16:00:24.844728 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.844571    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"
Sep 08 16:00:24.844728 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.844613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j" event={"ID":"0da03951-86b1-4c02-8db9-2beec8adbe6a","Type":"ContainerDied","Data":"4144d6d503882d81e3de4a0e97e7140eb1587739d853aa40963a63ffa46afce5"}
Sep 08 16:00:24.844728 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.844659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j" event={"ID":"0da03951-86b1-4c02-8db9-2beec8adbe6a","Type":"ContainerDied","Data":"b922ba4bf1c6eb1d4c45dea4e3c98dd4d23dea6108b8e5451794f1717929a312"}
Sep 08 16:00:24.844728 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.844679    2581 scope.go:117] "RemoveContainer" containerID="4144d6d503882d81e3de4a0e97e7140eb1587739d853aa40963a63ffa46afce5"
Sep 08 16:00:24.853793 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.853776    2581 scope.go:117] "RemoveContainer" containerID="4144d6d503882d81e3de4a0e97e7140eb1587739d853aa40963a63ffa46afce5"
Sep 08 16:00:24.854049 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:00:24.854031    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4144d6d503882d81e3de4a0e97e7140eb1587739d853aa40963a63ffa46afce5\": container with ID starting with 4144d6d503882d81e3de4a0e97e7140eb1587739d853aa40963a63ffa46afce5 not found: ID does not exist" containerID="4144d6d503882d81e3de4a0e97e7140eb1587739d853aa40963a63ffa46afce5"
Sep 08 16:00:24.854096 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.854057    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4144d6d503882d81e3de4a0e97e7140eb1587739d853aa40963a63ffa46afce5"} err="failed to get container status \"4144d6d503882d81e3de4a0e97e7140eb1587739d853aa40963a63ffa46afce5\": rpc error: code = NotFound desc = could not find container \"4144d6d503882d81e3de4a0e97e7140eb1587739d853aa40963a63ffa46afce5\": container with ID starting with 4144d6d503882d81e3de4a0e97e7140eb1587739d853aa40963a63ffa46afce5 not found: ID does not exist"
Sep 08 16:00:24.865669 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.865648    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"]
Sep 08 16:00:24.872143 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:24.872114    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-f50ef0/noten-model-f50ef0-kserve-6f95f69f75-tk26j"]
Sep 08 16:00:25.795542 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:25.795512    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"
Sep 08 16:00:25.902844 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:25.902813    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da03951-86b1-4c02-8db9-2beec8adbe6a" path="/var/lib/kubelet/pods/0da03951-86b1-4c02-8db9-2beec8adbe6a/volumes"
Sep 08 16:00:26.799590 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:26.799562    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"
Sep 08 16:00:30.887287 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:30.887251    2581 generic.go:356] "Generic (PLEG): container finished" podID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerID="f0a6d423c1f7414e7bf0a420f5db1e26ac5822e3f98a74f4223fe37ccf26c7fe" exitCode=0
Sep 08 16:00:30.887702 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:30.887320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs" event={"ID":"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75","Type":"ContainerDied","Data":"f0a6d423c1f7414e7bf0a420f5db1e26ac5822e3f98a74f4223fe37ccf26c7fe"}
Sep 08 16:00:30.887833 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:30.887814    2581 scope.go:117] "RemoveContainer" containerID="f0a6d423c1f7414e7bf0a420f5db1e26ac5822e3f98a74f4223fe37ccf26c7fe"
Sep 08 16:00:30.888895 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:30.888867    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:00:30.888990 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:30.888937    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:00:30.888990 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:30.888968    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:00:31.901941 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:31.901904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs" event={"ID":"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75","Type":"ContainerStarted","Data":"07246cb776a926c3af8252a60c2b5307946362fe49e10bc021508f44e8429c02"}
Sep 08 16:00:32.774718 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:32.774677    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:32.774929 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:32.774716    2581 patch_prober.go:28] interesting pod/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 08 16:00:32.774929 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:32.774767    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs" podUID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 08 16:00:33.774950 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:33.774912    2581 patch_prober.go:28] interesting pod/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 08 16:00:33.775448 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:33.774982    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs" podUID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 08 16:00:34.782415 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:34.780784    2581 patch_prober.go:28] interesting pod/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 08 16:00:34.782415 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:34.780859    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs" podUID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 08 16:00:35.775105 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:35.775058    2581 patch_prober.go:28] interesting pod/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 08 16:00:35.775311 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:35.775161    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs" podUID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 08 16:00:36.774322 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:36.774277    2581 patch_prober.go:28] interesting pod/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 08 16:00:36.774845 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:36.774342    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs" podUID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 08 16:00:37.778450 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:37.778412    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:37.778838 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:37.778684    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:37.779445 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:37.779424    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:00:43.299705 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.299668    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4"]
Sep 08 16:00:43.301702 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.301682    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ef1190-29814720-prxs4"]
Sep 08 16:00:43.324538 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.324506    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"]
Sep 08 16:00:43.324800 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.324770    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j" podUID="31151fb8-a8eb-4ef7-9da9-f268a1dd4aad" containerName="payload-processing" containerID="cri-o://c156ce399056ec098b3fd0e4032b46eb0d4586960b6ccca657ead7945a9a2571" gracePeriod=30
Sep 08 16:00:43.330055 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.330032    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"]
Sep 08 16:00:43.330898 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.330863    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9" podUID="a3568e2a-0e61-4004-a2bb-1686463cf0cd" containerName="payload-pre-processing" containerID="cri-o://8d57ab41e668588bb604c732cc99486be5f8de813dbc9bfe2e3cf6c0604a0366" gracePeriod=30
Sep 08 16:00:43.611753 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.611729    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"
Sep 08 16:00:43.614646 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.614628    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"
Sep 08 16:00:43.703165 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.703129    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsb2z\" (UniqueName: \"kubernetes.io/projected/31151fb8-a8eb-4ef7-9da9-f268a1dd4aad-kube-api-access-lsb2z\") pod \"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad\" (UID: \"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad\") "
Sep 08 16:00:43.703334 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.703176    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31151fb8-a8eb-4ef7-9da9-f268a1dd4aad-plugins-config-volume\") pod \"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad\" (UID: \"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad\") "
Sep 08 16:00:43.703334 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.703292    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6tw8\" (UniqueName: \"kubernetes.io/projected/a3568e2a-0e61-4004-a2bb-1686463cf0cd-kube-api-access-h6tw8\") pod \"a3568e2a-0e61-4004-a2bb-1686463cf0cd\" (UID: \"a3568e2a-0e61-4004-a2bb-1686463cf0cd\") "
Sep 08 16:00:43.703498 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.703331    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3568e2a-0e61-4004-a2bb-1686463cf0cd-plugins-config-volume\") pod \"a3568e2a-0e61-4004-a2bb-1686463cf0cd\" (UID: \"a3568e2a-0e61-4004-a2bb-1686463cf0cd\") "
Sep 08 16:00:43.703651 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.703618    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31151fb8-a8eb-4ef7-9da9-f268a1dd4aad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31151fb8-a8eb-4ef7-9da9-f268a1dd4aad" (UID: "31151fb8-a8eb-4ef7-9da9-f268a1dd4aad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:00:43.703774 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.703717    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31151fb8-a8eb-4ef7-9da9-f268a1dd4aad-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:43.703774 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.703716    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3568e2a-0e61-4004-a2bb-1686463cf0cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3568e2a-0e61-4004-a2bb-1686463cf0cd" (UID: "a3568e2a-0e61-4004-a2bb-1686463cf0cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:00:43.705334 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.705311    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31151fb8-a8eb-4ef7-9da9-f268a1dd4aad-kube-api-access-lsb2z" (OuterVolumeSpecName: "kube-api-access-lsb2z") pod "31151fb8-a8eb-4ef7-9da9-f268a1dd4aad" (UID: "31151fb8-a8eb-4ef7-9da9-f268a1dd4aad"). InnerVolumeSpecName "kube-api-access-lsb2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:00:43.705477 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.705337    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3568e2a-0e61-4004-a2bb-1686463cf0cd-kube-api-access-h6tw8" (OuterVolumeSpecName: "kube-api-access-h6tw8") pod "a3568e2a-0e61-4004-a2bb-1686463cf0cd" (UID: "a3568e2a-0e61-4004-a2bb-1686463cf0cd"). InnerVolumeSpecName "kube-api-access-h6tw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:00:43.804522 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.804492    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsb2z\" (UniqueName: \"kubernetes.io/projected/31151fb8-a8eb-4ef7-9da9-f268a1dd4aad-kube-api-access-lsb2z\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:43.804522 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.804517    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6tw8\" (UniqueName: \"kubernetes.io/projected/a3568e2a-0e61-4004-a2bb-1686463cf0cd-kube-api-access-h6tw8\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:43.804522 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.804527    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3568e2a-0e61-4004-a2bb-1686463cf0cd-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:43.904673 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.904643    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ee9f470-3a23-4904-878f-6833adcfa2a4" path="/var/lib/kubelet/pods/4ee9f470-3a23-4904-878f-6833adcfa2a4/volumes"
Sep 08 16:00:43.959420 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.959373    2581 generic.go:356] "Generic (PLEG): container finished" podID="a3568e2a-0e61-4004-a2bb-1686463cf0cd" containerID="8d57ab41e668588bb604c732cc99486be5f8de813dbc9bfe2e3cf6c0604a0366" exitCode=0
Sep 08 16:00:43.959420 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.959426    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"
Sep 08 16:00:43.959638 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.959460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9" event={"ID":"a3568e2a-0e61-4004-a2bb-1686463cf0cd","Type":"ContainerDied","Data":"8d57ab41e668588bb604c732cc99486be5f8de813dbc9bfe2e3cf6c0604a0366"}
Sep 08 16:00:43.959638 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.959512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9" event={"ID":"a3568e2a-0e61-4004-a2bb-1686463cf0cd","Type":"ContainerDied","Data":"5ac67ece6877921ea39564e413deb2bcd2ebcbe3a8a18cb4219ed2ed468d39f7"}
Sep 08 16:00:43.959638 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.959526    2581 scope.go:117] "RemoveContainer" containerID="8d57ab41e668588bb604c732cc99486be5f8de813dbc9bfe2e3cf6c0604a0366"
Sep 08 16:00:43.961268 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.961246    2581 generic.go:356] "Generic (PLEG): container finished" podID="31151fb8-a8eb-4ef7-9da9-f268a1dd4aad" containerID="c156ce399056ec098b3fd0e4032b46eb0d4586960b6ccca657ead7945a9a2571" exitCode=0
Sep 08 16:00:43.961379 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.961291    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"
Sep 08 16:00:43.961379 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.961351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j" event={"ID":"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad","Type":"ContainerDied","Data":"c156ce399056ec098b3fd0e4032b46eb0d4586960b6ccca657ead7945a9a2571"}
Sep 08 16:00:43.961513 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.961407    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j" event={"ID":"31151fb8-a8eb-4ef7-9da9-f268a1dd4aad","Type":"ContainerDied","Data":"e572db8c878bb4bdfa112634983de805c8c0321aa05665b164269e6545acc37a"}
Sep 08 16:00:43.968991 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.968971    2581 scope.go:117] "RemoveContainer" containerID="8d57ab41e668588bb604c732cc99486be5f8de813dbc9bfe2e3cf6c0604a0366"
Sep 08 16:00:43.969271 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:00:43.969249    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d57ab41e668588bb604c732cc99486be5f8de813dbc9bfe2e3cf6c0604a0366\": container with ID starting with 8d57ab41e668588bb604c732cc99486be5f8de813dbc9bfe2e3cf6c0604a0366 not found: ID does not exist" containerID="8d57ab41e668588bb604c732cc99486be5f8de813dbc9bfe2e3cf6c0604a0366"
Sep 08 16:00:43.969342 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.969279    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d57ab41e668588bb604c732cc99486be5f8de813dbc9bfe2e3cf6c0604a0366"} err="failed to get container status \"8d57ab41e668588bb604c732cc99486be5f8de813dbc9bfe2e3cf6c0604a0366\": rpc error: code = NotFound desc = could not find container \"8d57ab41e668588bb604c732cc99486be5f8de813dbc9bfe2e3cf6c0604a0366\": container with ID starting with 8d57ab41e668588bb604c732cc99486be5f8de813dbc9bfe2e3cf6c0604a0366 not found: ID does not exist"
Sep 08 16:00:43.969342 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.969296    2581 scope.go:117] "RemoveContainer" containerID="c156ce399056ec098b3fd0e4032b46eb0d4586960b6ccca657ead7945a9a2571"
Sep 08 16:00:43.977264 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.977245    2581 scope.go:117] "RemoveContainer" containerID="c156ce399056ec098b3fd0e4032b46eb0d4586960b6ccca657ead7945a9a2571"
Sep 08 16:00:43.977657 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:00:43.977625    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c156ce399056ec098b3fd0e4032b46eb0d4586960b6ccca657ead7945a9a2571\": container with ID starting with c156ce399056ec098b3fd0e4032b46eb0d4586960b6ccca657ead7945a9a2571 not found: ID does not exist" containerID="c156ce399056ec098b3fd0e4032b46eb0d4586960b6ccca657ead7945a9a2571"
Sep 08 16:00:43.977764 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.977661    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c156ce399056ec098b3fd0e4032b46eb0d4586960b6ccca657ead7945a9a2571"} err="failed to get container status \"c156ce399056ec098b3fd0e4032b46eb0d4586960b6ccca657ead7945a9a2571\": rpc error: code = NotFound desc = could not find container \"c156ce399056ec098b3fd0e4032b46eb0d4586960b6ccca657ead7945a9a2571\": container with ID starting with c156ce399056ec098b3fd0e4032b46eb0d4586960b6ccca657ead7945a9a2571 not found: ID does not exist"
Sep 08 16:00:43.980751 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.980370    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"]
Sep 08 16:00:43.985557 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.985537    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ef1190-696799dcbb-m8fz9"]
Sep 08 16:00:43.998530 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:43.998508    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"]
Sep 08 16:00:44.003317 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:44.003296    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ef1190-7f49f4ff7-rzb6j"]
Sep 08 16:00:45.900184 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:45.900151    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31151fb8-a8eb-4ef7-9da9-f268a1dd4aad" path="/var/lib/kubelet/pods/31151fb8-a8eb-4ef7-9da9-f268a1dd4aad/volumes"
Sep 08 16:00:45.900572 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:45.900543    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3568e2a-0e61-4004-a2bb-1686463cf0cd" path="/var/lib/kubelet/pods/a3568e2a-0e61-4004-a2bb-1686463cf0cd/volumes"
Sep 08 16:00:55.382475 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.382439    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"]
Sep 08 16:00:55.382947 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.382656    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w" podUID="316b66dc-44a2-49d3-a684-5d0331748315" containerName="maas-api" containerID="cri-o://e3864ca4d945dbda0e8caf2184a541a653ecb4e2b066c78a0a0bc41cee967c83" gracePeriod=30
Sep 08 16:00:55.431759 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.431730    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"]
Sep 08 16:00:55.431988 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.431957    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72" podUID="c4c9278c-3e43-4e62-b239-9cfcbce372ff" containerName="payload-processing" containerID="cri-o://1221ab7038011681f2aa067ff73d125b2366ea42b77d3cdd484ccb435160afc2" gracePeriod=30
Sep 08 16:00:55.438117 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.438082    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"]
Sep 08 16:00:55.438324 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.438291    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd" podUID="bc931a5c-7713-4bc6-bb9f-25db71c033d1" containerName="payload-pre-processing" containerID="cri-o://59fb0798750d8be3e922d3a0ec89e44507248c23683474f458fbaba22eedad89" gracePeriod=30
Sep 08 16:00:55.711780 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.711749    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:55.743452 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.743426    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"
Sep 08 16:00:55.746432 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.746411    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"
Sep 08 16:00:55.799001 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.798973    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c9278c-3e43-4e62-b239-9cfcbce372ff-plugins-config-volume\") pod \"c4c9278c-3e43-4e62-b239-9cfcbce372ff\" (UID: \"c4c9278c-3e43-4e62-b239-9cfcbce372ff\") "
Sep 08 16:00:55.799001 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.799006    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-metrics-certs\") pod \"316b66dc-44a2-49d3-a684-5d0331748315\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") "
Sep 08 16:00:55.799208 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.799041    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k44xh\" (UniqueName: \"kubernetes.io/projected/c4c9278c-3e43-4e62-b239-9cfcbce372ff-kube-api-access-k44xh\") pod \"c4c9278c-3e43-4e62-b239-9cfcbce372ff\" (UID: \"c4c9278c-3e43-4e62-b239-9cfcbce372ff\") "
Sep 08 16:00:55.799281 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.799262    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfhc4\" (UniqueName: \"kubernetes.io/projected/316b66dc-44a2-49d3-a684-5d0331748315-kube-api-access-hfhc4\") pod \"316b66dc-44a2-49d3-a684-5d0331748315\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") "
Sep 08 16:00:55.799340 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.799316    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-maas-api-tls\") pod \"316b66dc-44a2-49d3-a684-5d0331748315\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") "
Sep 08 16:00:55.799417 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.799341    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc931a5c-7713-4bc6-bb9f-25db71c033d1-plugins-config-volume\") pod \"bc931a5c-7713-4bc6-bb9f-25db71c033d1\" (UID: \"bc931a5c-7713-4bc6-bb9f-25db71c033d1\") "
Sep 08 16:00:55.799417 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.799384    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gxc2\" (UniqueName: \"kubernetes.io/projected/bc931a5c-7713-4bc6-bb9f-25db71c033d1-kube-api-access-6gxc2\") pod \"bc931a5c-7713-4bc6-bb9f-25db71c033d1\" (UID: \"bc931a5c-7713-4bc6-bb9f-25db71c033d1\") "
Sep 08 16:00:55.799528 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.799423    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316b66dc-44a2-49d3-a684-5d0331748315-openshift-service-ca\") pod \"316b66dc-44a2-49d3-a684-5d0331748315\" (UID: \"316b66dc-44a2-49d3-a684-5d0331748315\") "
Sep 08 16:00:55.799842 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.799786    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c9278c-3e43-4e62-b239-9cfcbce372ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4c9278c-3e43-4e62-b239-9cfcbce372ff" (UID: "c4c9278c-3e43-4e62-b239-9cfcbce372ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:00:55.799939 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.799867    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc931a5c-7713-4bc6-bb9f-25db71c033d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc931a5c-7713-4bc6-bb9f-25db71c033d1" (UID: "bc931a5c-7713-4bc6-bb9f-25db71c033d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:00:55.800008 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.799958    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316b66dc-44a2-49d3-a684-5d0331748315-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "316b66dc-44a2-49d3-a684-5d0331748315" (UID: "316b66dc-44a2-49d3-a684-5d0331748315"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:00:55.801382 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.801358    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c9278c-3e43-4e62-b239-9cfcbce372ff-kube-api-access-k44xh" (OuterVolumeSpecName: "kube-api-access-k44xh") pod "c4c9278c-3e43-4e62-b239-9cfcbce372ff" (UID: "c4c9278c-3e43-4e62-b239-9cfcbce372ff"). InnerVolumeSpecName "kube-api-access-k44xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:00:55.801812 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.801791    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "316b66dc-44a2-49d3-a684-5d0331748315" (UID: "316b66dc-44a2-49d3-a684-5d0331748315"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:00:55.801960 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.801931    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "316b66dc-44a2-49d3-a684-5d0331748315" (UID: "316b66dc-44a2-49d3-a684-5d0331748315"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:00:55.802028 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.801959    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc931a5c-7713-4bc6-bb9f-25db71c033d1-kube-api-access-6gxc2" (OuterVolumeSpecName: "kube-api-access-6gxc2") pod "bc931a5c-7713-4bc6-bb9f-25db71c033d1" (UID: "bc931a5c-7713-4bc6-bb9f-25db71c033d1"). InnerVolumeSpecName "kube-api-access-6gxc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:00:55.802028 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.801944    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316b66dc-44a2-49d3-a684-5d0331748315-kube-api-access-hfhc4" (OuterVolumeSpecName: "kube-api-access-hfhc4") pod "316b66dc-44a2-49d3-a684-5d0331748315" (UID: "316b66dc-44a2-49d3-a684-5d0331748315"). InnerVolumeSpecName "kube-api-access-hfhc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:00:55.900818 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.900793    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-maas-api-tls\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:55.900818 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.900817    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc931a5c-7713-4bc6-bb9f-25db71c033d1-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:55.900970 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.900831    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gxc2\" (UniqueName: \"kubernetes.io/projected/bc931a5c-7713-4bc6-bb9f-25db71c033d1-kube-api-access-6gxc2\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:55.900970 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.900844    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/316b66dc-44a2-49d3-a684-5d0331748315-openshift-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:55.900970 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.900856    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c9278c-3e43-4e62-b239-9cfcbce372ff-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:55.900970 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.900868    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/316b66dc-44a2-49d3-a684-5d0331748315-metrics-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:55.900970 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.900880    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k44xh\" (UniqueName: \"kubernetes.io/projected/c4c9278c-3e43-4e62-b239-9cfcbce372ff-kube-api-access-k44xh\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:55.900970 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:55.900892    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfhc4\" (UniqueName: \"kubernetes.io/projected/316b66dc-44a2-49d3-a684-5d0331748315-kube-api-access-hfhc4\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:00:56.022648 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.022585    2581 generic.go:356] "Generic (PLEG): container finished" podID="316b66dc-44a2-49d3-a684-5d0331748315" containerID="e3864ca4d945dbda0e8caf2184a541a653ecb4e2b066c78a0a0bc41cee967c83" exitCode=0
Sep 08 16:00:56.022648 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.022620    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"
Sep 08 16:00:56.022809 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.022665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w" event={"ID":"316b66dc-44a2-49d3-a684-5d0331748315","Type":"ContainerDied","Data":"e3864ca4d945dbda0e8caf2184a541a653ecb4e2b066c78a0a0bc41cee967c83"}
Sep 08 16:00:56.022809 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.022710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w" event={"ID":"316b66dc-44a2-49d3-a684-5d0331748315","Type":"ContainerDied","Data":"4c0427e122151c19ef61f68ac7adcfd9cc732bc45c4ac525d46da394fb6a42bb"}
Sep 08 16:00:56.022809 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.022734    2581 scope.go:117] "RemoveContainer" containerID="e3864ca4d945dbda0e8caf2184a541a653ecb4e2b066c78a0a0bc41cee967c83"
Sep 08 16:00:56.024239 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.024205    2581 generic.go:356] "Generic (PLEG): container finished" podID="c4c9278c-3e43-4e62-b239-9cfcbce372ff" containerID="1221ab7038011681f2aa067ff73d125b2366ea42b77d3cdd484ccb435160afc2" exitCode=0
Sep 08 16:00:56.024342 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.024269    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"
Sep 08 16:00:56.024342 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.024289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72" event={"ID":"c4c9278c-3e43-4e62-b239-9cfcbce372ff","Type":"ContainerDied","Data":"1221ab7038011681f2aa067ff73d125b2366ea42b77d3cdd484ccb435160afc2"}
Sep 08 16:00:56.024342 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.024337    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72" event={"ID":"c4c9278c-3e43-4e62-b239-9cfcbce372ff","Type":"ContainerDied","Data":"ca2d47ae98343aa0f8dde6dc37c4b8b5b4de7a01e3dd3d8acc6073efc6953fce"}
Sep 08 16:00:56.025805 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.025775    2581 generic.go:356] "Generic (PLEG): container finished" podID="bc931a5c-7713-4bc6-bb9f-25db71c033d1" containerID="59fb0798750d8be3e922d3a0ec89e44507248c23683474f458fbaba22eedad89" exitCode=0
Sep 08 16:00:56.025892 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.025807    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"
Sep 08 16:00:56.025892 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.025843    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd" event={"ID":"bc931a5c-7713-4bc6-bb9f-25db71c033d1","Type":"ContainerDied","Data":"59fb0798750d8be3e922d3a0ec89e44507248c23683474f458fbaba22eedad89"}
Sep 08 16:00:56.026022 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.025893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd" event={"ID":"bc931a5c-7713-4bc6-bb9f-25db71c033d1","Type":"ContainerDied","Data":"d13c479ca96e494a8733009ca6cd2b953f9c0c4b2b490973ff51f646a0ccf641"}
Sep 08 16:00:56.032944 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.032927    2581 scope.go:117] "RemoveContainer" containerID="e3864ca4d945dbda0e8caf2184a541a653ecb4e2b066c78a0a0bc41cee967c83"
Sep 08 16:00:56.033232 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:00:56.033215    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3864ca4d945dbda0e8caf2184a541a653ecb4e2b066c78a0a0bc41cee967c83\": container with ID starting with e3864ca4d945dbda0e8caf2184a541a653ecb4e2b066c78a0a0bc41cee967c83 not found: ID does not exist" containerID="e3864ca4d945dbda0e8caf2184a541a653ecb4e2b066c78a0a0bc41cee967c83"
Sep 08 16:00:56.033284 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.033241    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3864ca4d945dbda0e8caf2184a541a653ecb4e2b066c78a0a0bc41cee967c83"} err="failed to get container status \"e3864ca4d945dbda0e8caf2184a541a653ecb4e2b066c78a0a0bc41cee967c83\": rpc error: code = NotFound desc = could not find container \"e3864ca4d945dbda0e8caf2184a541a653ecb4e2b066c78a0a0bc41cee967c83\": container with ID starting with e3864ca4d945dbda0e8caf2184a541a653ecb4e2b066c78a0a0bc41cee967c83 not found: ID does not exist"
Sep 08 16:00:56.033284 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.033257    2581 scope.go:117] "RemoveContainer" containerID="1221ab7038011681f2aa067ff73d125b2366ea42b77d3cdd484ccb435160afc2"
Sep 08 16:00:56.040827 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.040809    2581 scope.go:117] "RemoveContainer" containerID="1221ab7038011681f2aa067ff73d125b2366ea42b77d3cdd484ccb435160afc2"
Sep 08 16:00:56.041075 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:00:56.041056    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1221ab7038011681f2aa067ff73d125b2366ea42b77d3cdd484ccb435160afc2\": container with ID starting with 1221ab7038011681f2aa067ff73d125b2366ea42b77d3cdd484ccb435160afc2 not found: ID does not exist" containerID="1221ab7038011681f2aa067ff73d125b2366ea42b77d3cdd484ccb435160afc2"
Sep 08 16:00:56.041124 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.041081    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1221ab7038011681f2aa067ff73d125b2366ea42b77d3cdd484ccb435160afc2"} err="failed to get container status \"1221ab7038011681f2aa067ff73d125b2366ea42b77d3cdd484ccb435160afc2\": rpc error: code = NotFound desc = could not find container \"1221ab7038011681f2aa067ff73d125b2366ea42b77d3cdd484ccb435160afc2\": container with ID starting with 1221ab7038011681f2aa067ff73d125b2366ea42b77d3cdd484ccb435160afc2 not found: ID does not exist"
Sep 08 16:00:56.041124 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.041097    2581 scope.go:117] "RemoveContainer" containerID="59fb0798750d8be3e922d3a0ec89e44507248c23683474f458fbaba22eedad89"
Sep 08 16:00:56.047080 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.047060    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"]
Sep 08 16:00:56.049057 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.049037    2581 scope.go:117] "RemoveContainer" containerID="59fb0798750d8be3e922d3a0ec89e44507248c23683474f458fbaba22eedad89"
Sep 08 16:00:56.049336 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:00:56.049314    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59fb0798750d8be3e922d3a0ec89e44507248c23683474f458fbaba22eedad89\": container with ID starting with 59fb0798750d8be3e922d3a0ec89e44507248c23683474f458fbaba22eedad89 not found: ID does not exist" containerID="59fb0798750d8be3e922d3a0ec89e44507248c23683474f458fbaba22eedad89"
Sep 08 16:00:56.049419 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.049344    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59fb0798750d8be3e922d3a0ec89e44507248c23683474f458fbaba22eedad89"} err="failed to get container status \"59fb0798750d8be3e922d3a0ec89e44507248c23683474f458fbaba22eedad89\": rpc error: code = NotFound desc = could not find container \"59fb0798750d8be3e922d3a0ec89e44507248c23683474f458fbaba22eedad89\": container with ID starting with 59fb0798750d8be3e922d3a0ec89e44507248c23683474f458fbaba22eedad89 not found: ID does not exist"
Sep 08 16:00:56.051069 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.051052    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf23f7aa-5f98bf5b5c-29x7w"]
Sep 08 16:00:56.068907 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.068884    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"]
Sep 08 16:00:56.074529 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.074498    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cf23f7aa-d75b8665d-9pwxd"]
Sep 08 16:00:56.089469 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.089448    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"]
Sep 08 16:00:56.094659 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:56.094641    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cf23f7aa-5cff84bccc-pdr72"]
Sep 08 16:00:57.900167 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:57.900131    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="316b66dc-44a2-49d3-a684-5d0331748315" path="/var/lib/kubelet/pods/316b66dc-44a2-49d3-a684-5d0331748315/volumes"
Sep 08 16:00:57.900695 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:57.900672    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc931a5c-7713-4bc6-bb9f-25db71c033d1" path="/var/lib/kubelet/pods/bc931a5c-7713-4bc6-bb9f-25db71c033d1/volumes"
Sep 08 16:00:57.901047 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:00:57.901029    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4c9278c-3e43-4e62-b239-9cfcbce372ff" path="/var/lib/kubelet/pods/c4c9278c-3e43-4e62-b239-9cfcbce372ff/volumes"
Sep 08 16:01:00.960806 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:00.960775    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8"]
Sep 08 16:01:00.963515 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:00.963479    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c96e2b-29814720-prxn8"]
Sep 08 16:01:01.900866 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:01.900824    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f5e8231-308e-473c-bacd-21c448e3a666" path="/var/lib/kubelet/pods/5f5e8231-308e-473c-bacd-21c448e3a666/volumes"
Sep 08 16:01:02.431649 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:02.431618    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"]
Sep 08 16:01:02.432033 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:02.431825    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs" podUID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerName="istio-proxy" containerID="cri-o://07246cb776a926c3af8252a60c2b5307946362fe49e10bc021508f44e8429c02" gracePeriod=30
Sep 08 16:01:07.682745 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.682719    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:01:07.718540 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.718469    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"]
Sep 08 16:01:07.718734 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.718708    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy" containerID="cri-o://afa32386171d57d9ee5112c07cb15317191d7f15be52983486a993e8e3643971" gracePeriod=30
Sep 08 16:01:07.800905 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.800881    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-token\") pod \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") "
Sep 08 16:01:07.801059 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.800932    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-envoy\") pod \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") "
Sep 08 16:01:07.801059 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.800982    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istiod-ca-cert\") pod \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") "
Sep 08 16:01:07.801059 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801010    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hgqq\" (UniqueName: \"kubernetes.io/projected/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-kube-api-access-8hgqq\") pod \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") "
Sep 08 16:01:07.801059 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801042    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-podinfo\") pod \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") "
Sep 08 16:01:07.801274 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801078    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-workload-socket\") pod \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") "
Sep 08 16:01:07.801274 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801103    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-workload-certs\") pod \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") "
Sep 08 16:01:07.801274 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801145    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-credential-socket\") pod \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") "
Sep 08 16:01:07.801274 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801169    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-data\") pod \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\" (UID: \"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75\") "
Sep 08 16:01:07.801524 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801380    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" (UID: "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:01:07.801524 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801444    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" (UID: "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:01:07.801524 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801459    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" (UID: "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:01:07.801689 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801562    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istiod-ca-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:07.801689 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801581    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-workload-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:07.801689 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801596    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-workload-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:07.801689 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801562    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" (UID: "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:01:07.801689 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.801643    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-data" (OuterVolumeSpecName: "istio-data") pod "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" (UID: "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:01:07.803211 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.803184    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" (UID: "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:01:07.803332 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.803311    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-token" (OuterVolumeSpecName: "istio-token") pod "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" (UID: "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:01:07.803461 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.803432    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-kube-api-access-8hgqq" (OuterVolumeSpecName: "kube-api-access-8hgqq") pod "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" (UID: "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75"). InnerVolumeSpecName "kube-api-access-8hgqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:01:07.803571 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.803555    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" (UID: "b8fc74fc-1e8e-4a97-af3f-a6b3a458db75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:01:07.902560 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.902532    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hgqq\" (UniqueName: \"kubernetes.io/projected/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-kube-api-access-8hgqq\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:07.902560 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.902558    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-podinfo\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:07.902716 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.902571    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-credential-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:07.902716 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.902583    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-data\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:07.902716 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.902594    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-token\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:07.902716 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:07.902604    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75-istio-envoy\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:08.092667 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.092596    2581 generic.go:356] "Generic (PLEG): container finished" podID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerID="07246cb776a926c3af8252a60c2b5307946362fe49e10bc021508f44e8429c02" exitCode=0
Sep 08 16:01:08.092667 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.092630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs" event={"ID":"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75","Type":"ContainerDied","Data":"07246cb776a926c3af8252a60c2b5307946362fe49e10bc021508f44e8429c02"}
Sep 08 16:01:08.092667 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.092646    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"
Sep 08 16:01:08.092667 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.092665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs" event={"ID":"b8fc74fc-1e8e-4a97-af3f-a6b3a458db75","Type":"ContainerDied","Data":"25f93043750b585c3dfc6af470035de3c7b8399d0e02b2edf901716f19d4eeea"}
Sep 08 16:01:08.092959 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.092685    2581 scope.go:117] "RemoveContainer" containerID="07246cb776a926c3af8252a60c2b5307946362fe49e10bc021508f44e8429c02"
Sep 08 16:01:08.101856 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.101824    2581 scope.go:117] "RemoveContainer" containerID="f0a6d423c1f7414e7bf0a420f5db1e26ac5822e3f98a74f4223fe37ccf26c7fe"
Sep 08 16:01:08.108807 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.108785    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"]
Sep 08 16:01:08.109688 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.109671    2581 scope.go:117] "RemoveContainer" containerID="07246cb776a926c3af8252a60c2b5307946362fe49e10bc021508f44e8429c02"
Sep 08 16:01:08.109945 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:01:08.109927    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07246cb776a926c3af8252a60c2b5307946362fe49e10bc021508f44e8429c02\": container with ID starting with 07246cb776a926c3af8252a60c2b5307946362fe49e10bc021508f44e8429c02 not found: ID does not exist" containerID="07246cb776a926c3af8252a60c2b5307946362fe49e10bc021508f44e8429c02"
Sep 08 16:01:08.109997 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.109952    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07246cb776a926c3af8252a60c2b5307946362fe49e10bc021508f44e8429c02"} err="failed to get container status \"07246cb776a926c3af8252a60c2b5307946362fe49e10bc021508f44e8429c02\": rpc error: code = NotFound desc = could not find container \"07246cb776a926c3af8252a60c2b5307946362fe49e10bc021508f44e8429c02\": container with ID starting with 07246cb776a926c3af8252a60c2b5307946362fe49e10bc021508f44e8429c02 not found: ID does not exist"
Sep 08 16:01:08.109997 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.109971    2581 scope.go:117] "RemoveContainer" containerID="f0a6d423c1f7414e7bf0a420f5db1e26ac5822e3f98a74f4223fe37ccf26c7fe"
Sep 08 16:01:08.110297 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:01:08.110272    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a6d423c1f7414e7bf0a420f5db1e26ac5822e3f98a74f4223fe37ccf26c7fe\": container with ID starting with f0a6d423c1f7414e7bf0a420f5db1e26ac5822e3f98a74f4223fe37ccf26c7fe not found: ID does not exist" containerID="f0a6d423c1f7414e7bf0a420f5db1e26ac5822e3f98a74f4223fe37ccf26c7fe"
Sep 08 16:01:08.110380 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.110304    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a6d423c1f7414e7bf0a420f5db1e26ac5822e3f98a74f4223fe37ccf26c7fe"} err="failed to get container status \"f0a6d423c1f7414e7bf0a420f5db1e26ac5822e3f98a74f4223fe37ccf26c7fe\": rpc error: code = NotFound desc = could not find container \"f0a6d423c1f7414e7bf0a420f5db1e26ac5822e3f98a74f4223fe37ccf26c7fe\": container with ID starting with f0a6d423c1f7414e7bf0a420f5db1e26ac5822e3f98a74f4223fe37ccf26c7fe not found: ID does not exist"
Sep 08 16:01:08.112328 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.112308    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cf23f7aa-openshift-default-bb6c5c784-4l9qs"]
Sep 08 16:01:08.560716 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.560683    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"]
Sep 08 16:01:08.561102 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561087    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerName="istio-proxy"
Sep 08 16:01:08.561184 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561105    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerName="istio-proxy"
Sep 08 16:01:08.561184 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561121    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3568e2a-0e61-4004-a2bb-1686463cf0cd" containerName="payload-pre-processing"
Sep 08 16:01:08.561184 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561130    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3568e2a-0e61-4004-a2bb-1686463cf0cd" containerName="payload-pre-processing"
Sep 08 16:01:08.561184 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561145    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da03951-86b1-4c02-8db9-2beec8adbe6a" containerName="main"
Sep 08 16:01:08.561184 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561154    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da03951-86b1-4c02-8db9-2beec8adbe6a" containerName="main"
Sep 08 16:01:08.561184 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561172    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc931a5c-7713-4bc6-bb9f-25db71c033d1" containerName="payload-pre-processing"
Sep 08 16:01:08.561184 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561181    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc931a5c-7713-4bc6-bb9f-25db71c033d1" containerName="payload-pre-processing"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561202    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31151fb8-a8eb-4ef7-9da9-f268a1dd4aad" containerName="payload-processing"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561211    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="31151fb8-a8eb-4ef7-9da9-f268a1dd4aad" containerName="payload-processing"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561221    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd125bef-1cc2-455d-8cce-7663ee3ab48b" containerName="istio-proxy"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561229    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd125bef-1cc2-455d-8cce-7663ee3ab48b" containerName="istio-proxy"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561239    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316b66dc-44a2-49d3-a684-5d0331748315" containerName="maas-api"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561245    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="316b66dc-44a2-49d3-a684-5d0331748315" containerName="maas-api"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561253    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4c9278c-3e43-4e62-b239-9cfcbce372ff" containerName="payload-processing"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561259    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4c9278c-3e43-4e62-b239-9cfcbce372ff" containerName="payload-processing"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561339    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="31151fb8-a8eb-4ef7-9da9-f268a1dd4aad" containerName="payload-processing"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561353    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd125bef-1cc2-455d-8cce-7663ee3ab48b" containerName="istio-proxy"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561364    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="316b66dc-44a2-49d3-a684-5d0331748315" containerName="maas-api"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561371    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4c9278c-3e43-4e62-b239-9cfcbce372ff" containerName="payload-processing"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561379    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc931a5c-7713-4bc6-bb9f-25db71c033d1" containerName="payload-pre-processing"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561384    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da03951-86b1-4c02-8db9-2beec8adbe6a" containerName="main"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561412    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerName="istio-proxy"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561420    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerName="istio-proxy"
Sep 08 16:01:08.561517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.561428    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3568e2a-0e61-4004-a2bb-1686463cf0cd" containerName="payload-pre-processing"
Sep 08 16:01:08.565834 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.565817    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.568006 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.567981    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-108124-openshift-default-dockercfg-7k4z9\""
Sep 08 16:01:08.575910 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.575887    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"]
Sep 08 16:01:08.607863 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.607838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-podinfo\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.607997 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.607878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d4b521-010e-4f98-9e55-efba77ed63c0-istiod-ca-cert\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.607997 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.607900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-token\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.607997 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.607962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-envoy\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.608147 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.608023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-data\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.608147 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.608101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-credential-socket\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.608147 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.608119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-workload-certs\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.608147 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.608139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-workload-socket\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.608286 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.608181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzt9\" (UniqueName: \"kubernetes.io/projected/49d4b521-010e-4f98-9e55-efba77ed63c0-kube-api-access-5gzt9\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.708890 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.708863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-credential-socket\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.708890 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.708895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-workload-certs\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.709350 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.708916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-workload-socket\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.709350 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.708934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzt9\" (UniqueName: \"kubernetes.io/projected/49d4b521-010e-4f98-9e55-efba77ed63c0-kube-api-access-5gzt9\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.709350 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.709034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-podinfo\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.709350 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.709095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d4b521-010e-4f98-9e55-efba77ed63c0-istiod-ca-cert\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.709350 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.709125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-token\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.709350 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.709174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-envoy\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.709350 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.709210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-data\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.709350 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.709346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-workload-certs\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.709792 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.709599    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-credential-socket\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.709792 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.709714    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-workload-socket\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.709792 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.709733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-data\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.710008 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.709988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d4b521-010e-4f98-9e55-efba77ed63c0-istiod-ca-cert\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.711887 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.711870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-envoy\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.712104 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.712085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-podinfo\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.738066 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.738044    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-token\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.738423 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.738380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzt9\" (UniqueName: \"kubernetes.io/projected/49d4b521-010e-4f98-9e55-efba77ed63c0-kube-api-access-5gzt9\") pod \"e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") " pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:08.877095 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:08.877034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:09.154304 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:09.154280    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"]
Sep 08 16:01:09.155984 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:01:09.155960    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d4b521_010e_4f98_9e55_efba77ed63c0.slice/crio-5102759b43742baaa1f7ff1c7743c9c2c6b98cd29bfeb6190fcdcab759b1dd8e WatchSource:0}: Error finding container 5102759b43742baaa1f7ff1c7743c9c2c6b98cd29bfeb6190fcdcab759b1dd8e: Status 404 returned error can't find the container with id 5102759b43742baaa1f7ff1c7743c9c2c6b98cd29bfeb6190fcdcab759b1dd8e
Sep 08 16:01:09.158113 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:09.158081    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:01:09.158174 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:09.158147    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:01:09.158215 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:09.158175    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:01:09.901890 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:09.901855    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" path="/var/lib/kubelet/pods/b8fc74fc-1e8e-4a97-af3f-a6b3a458db75/volumes"
Sep 08 16:01:10.105426 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:10.105371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" event={"ID":"49d4b521-010e-4f98-9e55-efba77ed63c0","Type":"ContainerStarted","Data":"8cc99d2d69c27e72d644e8afce08b80d2ffea00857aaa6dca0ee1ad3335b9121"}
Sep 08 16:01:10.105582 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:10.105444    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" event={"ID":"49d4b521-010e-4f98-9e55-efba77ed63c0","Type":"ContainerStarted","Data":"5102759b43742baaa1f7ff1c7743c9c2c6b98cd29bfeb6190fcdcab759b1dd8e"}
Sep 08 16:01:10.125549 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:10.125506    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" podStartSLOduration=2.125491275 podStartE2EDuration="2.125491275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:01:10.124517014 +0000 UTC m=+1016.748209339" watchObservedRunningTime="2026-09-08 16:01:10.125491275 +0000 UTC m=+1016.749183630"
Sep 08 16:01:10.878093 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:10.878058    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:10.882753 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:10.882727    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:11.110382 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:11.110350    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:11.111732 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:11.111663    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:12.965574 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:12.965547    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 16:01:13.048808 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.048750    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-workload-certs\") pod \"449bdd7a-7b16-469a-a5f4-5e45f834403e\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") "
Sep 08 16:01:13.048808 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.048788    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-podinfo\") pod \"449bdd7a-7b16-469a-a5f4-5e45f834403e\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") "
Sep 08 16:01:13.048808 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.048809    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-envoy\") pod \"449bdd7a-7b16-469a-a5f4-5e45f834403e\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") "
Sep 08 16:01:13.049040 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.048848    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-data\") pod \"449bdd7a-7b16-469a-a5f4-5e45f834403e\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") "
Sep 08 16:01:13.049040 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.048903    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449bdd7a-7b16-469a-a5f4-5e45f834403e-istiod-ca-cert\") pod \"449bdd7a-7b16-469a-a5f4-5e45f834403e\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") "
Sep 08 16:01:13.049040 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.048924    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-workload-socket\") pod \"449bdd7a-7b16-469a-a5f4-5e45f834403e\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") "
Sep 08 16:01:13.049040 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.048957    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-credential-socket\") pod \"449bdd7a-7b16-469a-a5f4-5e45f834403e\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") "
Sep 08 16:01:13.049040 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.048986    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-token\") pod \"449bdd7a-7b16-469a-a5f4-5e45f834403e\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") "
Sep 08 16:01:13.049040 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.049028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcvs2\" (UniqueName: \"kubernetes.io/projected/449bdd7a-7b16-469a-a5f4-5e45f834403e-kube-api-access-dcvs2\") pod \"449bdd7a-7b16-469a-a5f4-5e45f834403e\" (UID: \"449bdd7a-7b16-469a-a5f4-5e45f834403e\") "
Sep 08 16:01:13.049353 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.049042    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "449bdd7a-7b16-469a-a5f4-5e45f834403e" (UID: "449bdd7a-7b16-469a-a5f4-5e45f834403e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:01:13.049353 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.049224    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "449bdd7a-7b16-469a-a5f4-5e45f834403e" (UID: "449bdd7a-7b16-469a-a5f4-5e45f834403e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:01:13.049353 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.049298    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "449bdd7a-7b16-469a-a5f4-5e45f834403e" (UID: "449bdd7a-7b16-469a-a5f4-5e45f834403e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:01:13.049353 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.049329    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449bdd7a-7b16-469a-a5f4-5e45f834403e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "449bdd7a-7b16-469a-a5f4-5e45f834403e" (UID: "449bdd7a-7b16-469a-a5f4-5e45f834403e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:01:13.049353 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.049333    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-workload-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:13.049591 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.049370    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-workload-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:13.049591 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.049454    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-data" (OuterVolumeSpecName: "istio-data") pod "449bdd7a-7b16-469a-a5f4-5e45f834403e" (UID: "449bdd7a-7b16-469a-a5f4-5e45f834403e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:01:13.050989 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.050962    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "449bdd7a-7b16-469a-a5f4-5e45f834403e" (UID: "449bdd7a-7b16-469a-a5f4-5e45f834403e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:01:13.051085 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.051061    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449bdd7a-7b16-469a-a5f4-5e45f834403e-kube-api-access-dcvs2" (OuterVolumeSpecName: "kube-api-access-dcvs2") pod "449bdd7a-7b16-469a-a5f4-5e45f834403e" (UID: "449bdd7a-7b16-469a-a5f4-5e45f834403e"). InnerVolumeSpecName "kube-api-access-dcvs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:01:13.051439 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.051417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "449bdd7a-7b16-469a-a5f4-5e45f834403e" (UID: "449bdd7a-7b16-469a-a5f4-5e45f834403e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:01:13.051512 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.051460    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-token" (OuterVolumeSpecName: "istio-token") pod "449bdd7a-7b16-469a-a5f4-5e45f834403e" (UID: "449bdd7a-7b16-469a-a5f4-5e45f834403e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:01:13.121966 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.121943    2581 generic.go:356] "Generic (PLEG): container finished" podID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerID="afa32386171d57d9ee5112c07cb15317191d7f15be52983486a993e8e3643971" exitCode=0
Sep 08 16:01:13.122073 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.121991    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"
Sep 08 16:01:13.122073 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.122038    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" event={"ID":"449bdd7a-7b16-469a-a5f4-5e45f834403e","Type":"ContainerDied","Data":"afa32386171d57d9ee5112c07cb15317191d7f15be52983486a993e8e3643971"}
Sep 08 16:01:13.122191 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.122100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq" event={"ID":"449bdd7a-7b16-469a-a5f4-5e45f834403e","Type":"ContainerDied","Data":"2b0d37cf30e08be109664771b0faf1af8dd8b24b903ed8c799e1236928f8aee0"}
Sep 08 16:01:13.122191 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.122133    2581 scope.go:117] "RemoveContainer" containerID="afa32386171d57d9ee5112c07cb15317191d7f15be52983486a993e8e3643971"
Sep 08 16:01:13.131095 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.131077    2581 scope.go:117] "RemoveContainer" containerID="1ecb52f1746f9c7fa347801a086a0cfef810194efdff7861fb9bdee660582275"
Sep 08 16:01:13.138842 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.138827    2581 scope.go:117] "RemoveContainer" containerID="afa32386171d57d9ee5112c07cb15317191d7f15be52983486a993e8e3643971"
Sep 08 16:01:13.139099 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:01:13.139081    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa32386171d57d9ee5112c07cb15317191d7f15be52983486a993e8e3643971\": container with ID starting with afa32386171d57d9ee5112c07cb15317191d7f15be52983486a993e8e3643971 not found: ID does not exist" containerID="afa32386171d57d9ee5112c07cb15317191d7f15be52983486a993e8e3643971"
Sep 08 16:01:13.139149 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.139107    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa32386171d57d9ee5112c07cb15317191d7f15be52983486a993e8e3643971"} err="failed to get container status \"afa32386171d57d9ee5112c07cb15317191d7f15be52983486a993e8e3643971\": rpc error: code = NotFound desc = could not find container \"afa32386171d57d9ee5112c07cb15317191d7f15be52983486a993e8e3643971\": container with ID starting with afa32386171d57d9ee5112c07cb15317191d7f15be52983486a993e8e3643971 not found: ID does not exist"
Sep 08 16:01:13.139149 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.139122    2581 scope.go:117] "RemoveContainer" containerID="1ecb52f1746f9c7fa347801a086a0cfef810194efdff7861fb9bdee660582275"
Sep 08 16:01:13.139348 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:01:13.139331    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecb52f1746f9c7fa347801a086a0cfef810194efdff7861fb9bdee660582275\": container with ID starting with 1ecb52f1746f9c7fa347801a086a0cfef810194efdff7861fb9bdee660582275 not found: ID does not exist" containerID="1ecb52f1746f9c7fa347801a086a0cfef810194efdff7861fb9bdee660582275"
Sep 08 16:01:13.139419 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.139357    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecb52f1746f9c7fa347801a086a0cfef810194efdff7861fb9bdee660582275"} err="failed to get container status \"1ecb52f1746f9c7fa347801a086a0cfef810194efdff7861fb9bdee660582275\": rpc error: code = NotFound desc = could not find container \"1ecb52f1746f9c7fa347801a086a0cfef810194efdff7861fb9bdee660582275\": container with ID starting with 1ecb52f1746f9c7fa347801a086a0cfef810194efdff7861fb9bdee660582275 not found: ID does not exist"
Sep 08 16:01:13.142897 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.142876    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"]
Sep 08 16:01:13.146349 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.146330    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c96e2b-openshift-default-7c78b59d66-lj2mq"]
Sep 08 16:01:13.150679 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.150660    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449bdd7a-7b16-469a-a5f4-5e45f834403e-istiod-ca-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:13.150758 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.150682    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-credential-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:13.150758 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.150698    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-token\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:13.150758 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.150710    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcvs2\" (UniqueName: \"kubernetes.io/projected/449bdd7a-7b16-469a-a5f4-5e45f834403e-kube-api-access-dcvs2\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:13.150758 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.150724    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-podinfo\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:13.150758 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.150738    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-envoy\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:13.150758 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.150750    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449bdd7a-7b16-469a-a5f4-5e45f834403e-istio-data\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:01:13.196438 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.196412    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"]
Sep 08 16:01:13.196796 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.196780    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy"
Sep 08 16:01:13.196796 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.196796    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy"
Sep 08 16:01:13.196939 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.196805    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerName="istio-proxy"
Sep 08 16:01:13.196939 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.196810    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8fc74fc-1e8e-4a97-af3f-a6b3a458db75" containerName="istio-proxy"
Sep 08 16:01:13.196939 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.196821    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy"
Sep 08 16:01:13.196939 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.196826    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy"
Sep 08 16:01:13.196939 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.196888    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy"
Sep 08 16:01:13.196939 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.196895    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" containerName="istio-proxy"
Sep 08 16:01:13.201430 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.201409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.203867 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.203650    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmx7t\""
Sep 08 16:01:13.203867 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.203665    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 16:01:13.203867 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.203702    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-108124-serving-cert\""
Sep 08 16:01:13.203867 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.203666    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:01:13.203867 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.203670    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 16:01:13.211328 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.211305    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"]
Sep 08 16:01:13.353369 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.353301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.353369 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.353351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.353539 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.353426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.353539 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.353507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgddj\" (UniqueName: \"kubernetes.io/projected/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-kube-api-access-lgddj\") pod \"maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.454193 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.454156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.454308 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.454210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.454308 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.454237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.454308 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.454275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgddj\" (UniqueName: \"kubernetes.io/projected/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-kube-api-access-lgddj\") pod \"maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.454520 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:01:13.454417    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-serving-cert: secret "maas-api-e2e-shared-a-w3-108124-serving-cert" not found
Sep 08 16:01:13.454520 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:01:13.454492    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-maas-api-tls podName:f7c3105e-4fdb-4dd9-898c-1aebdabb6cff nodeName:}" failed. No retries permitted until 2026-09-08 16:01:13.954469968 +0000 UTC m=+1020.578162275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-maas-api-tls") pod "maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w" (UID: "f7c3105e-4fdb-4dd9-898c-1aebdabb6cff") : secret "maas-api-e2e-shared-a-w3-108124-serving-cert" not found
Sep 08 16:01:13.454809 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.454788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.456787 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.456767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.466281 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.466260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgddj\" (UniqueName: \"kubernetes.io/projected/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-kube-api-access-lgddj\") pod \"maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.785248 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.785218    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"]
Sep 08 16:01:13.795410 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.795366    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"
Sep 08 16:01:13.798060 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.798036    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-108124\""
Sep 08 16:01:13.798251 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.798101    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-108124-dockercfg-fzxqt\""
Sep 08 16:01:13.798346 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.798122    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"]
Sep 08 16:01:13.827220 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.827196    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"]
Sep 08 16:01:13.830651 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.830629    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"
Sep 08 16:01:13.839205 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.839182    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"]
Sep 08 16:01:13.857673 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.857650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8b1004-ff7e-449a-8351-e5b9d69ee84d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz\" (UID: \"9d8b1004-ff7e-449a-8351-e5b9d69ee84d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"
Sep 08 16:01:13.857782 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.857753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gp2\" (UniqueName: \"kubernetes.io/projected/9d8b1004-ff7e-449a-8351-e5b9d69ee84d-kube-api-access-22gp2\") pod \"payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz\" (UID: \"9d8b1004-ff7e-449a-8351-e5b9d69ee84d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"
Sep 08 16:01:13.901108 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.901083    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="449bdd7a-7b16-469a-a5f4-5e45f834403e" path="/var/lib/kubelet/pods/449bdd7a-7b16-469a-a5f4-5e45f834403e/volumes"
Sep 08 16:01:13.958771 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.958747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b96330b-5c01-48d5-acc0-422148a59dd3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx\" (UID: \"8b96330b-5c01-48d5-acc0-422148a59dd3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"
Sep 08 16:01:13.958875 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.958784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gp2\" (UniqueName: \"kubernetes.io/projected/9d8b1004-ff7e-449a-8351-e5b9d69ee84d-kube-api-access-22gp2\") pod \"payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz\" (UID: \"9d8b1004-ff7e-449a-8351-e5b9d69ee84d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"
Sep 08 16:01:13.958875 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.958860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8b1004-ff7e-449a-8351-e5b9d69ee84d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz\" (UID: \"9d8b1004-ff7e-449a-8351-e5b9d69ee84d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"
Sep 08 16:01:13.958953 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.958889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.958953 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.958905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwjnl\" (UniqueName: \"kubernetes.io/projected/8b96330b-5c01-48d5-acc0-422148a59dd3-kube-api-access-jwjnl\") pod \"payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx\" (UID: \"8b96330b-5c01-48d5-acc0-422148a59dd3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"
Sep 08 16:01:13.959542 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.959519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8b1004-ff7e-449a-8351-e5b9d69ee84d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz\" (UID: \"9d8b1004-ff7e-449a-8351-e5b9d69ee84d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"
Sep 08 16:01:13.961018 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.960999    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:13.966252 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:13.966227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gp2\" (UniqueName: \"kubernetes.io/projected/9d8b1004-ff7e-449a-8351-e5b9d69ee84d-kube-api-access-22gp2\") pod \"payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz\" (UID: \"9d8b1004-ff7e-449a-8351-e5b9d69ee84d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"
Sep 08 16:01:14.059623 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:14.059598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b96330b-5c01-48d5-acc0-422148a59dd3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx\" (UID: \"8b96330b-5c01-48d5-acc0-422148a59dd3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"
Sep 08 16:01:14.059725 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:14.059686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwjnl\" (UniqueName: \"kubernetes.io/projected/8b96330b-5c01-48d5-acc0-422148a59dd3-kube-api-access-jwjnl\") pod \"payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx\" (UID: \"8b96330b-5c01-48d5-acc0-422148a59dd3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"
Sep 08 16:01:14.060360 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:14.060341    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b96330b-5c01-48d5-acc0-422148a59dd3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx\" (UID: \"8b96330b-5c01-48d5-acc0-422148a59dd3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"
Sep 08 16:01:14.068456 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:14.068297    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwjnl\" (UniqueName: \"kubernetes.io/projected/8b96330b-5c01-48d5-acc0-422148a59dd3-kube-api-access-jwjnl\") pod \"payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx\" (UID: \"8b96330b-5c01-48d5-acc0-422148a59dd3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"
Sep 08 16:01:14.106811 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:14.106784    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"
Sep 08 16:01:14.116549 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:14.116531    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmx7t\""
Sep 08 16:01:14.124082 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:14.124060    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:14.140783 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:14.140761    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"
Sep 08 16:01:14.384276 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:14.384240    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"]
Sep 08 16:01:14.387226 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:01:14.387195    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8b1004_ff7e_449a_8351_e5b9d69ee84d.slice/crio-93c6d83d1b31da666fe3e7112726cdd0c71c22342be1575943b1d376aca42808 WatchSource:0}: Error finding container 93c6d83d1b31da666fe3e7112726cdd0c71c22342be1575943b1d376aca42808: Status 404 returned error can't find the container with id 93c6d83d1b31da666fe3e7112726cdd0c71c22342be1575943b1d376aca42808
Sep 08 16:01:14.411104 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:14.411080    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"]
Sep 08 16:01:14.420566 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:01:14.420539    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c3105e_4fdb_4dd9_898c_1aebdabb6cff.slice/crio-a8ae926f34f6eea1e506c2ec41694b6c2a5601899d7b3d7f55339aa3718e73b1 WatchSource:0}: Error finding container a8ae926f34f6eea1e506c2ec41694b6c2a5601899d7b3d7f55339aa3718e73b1: Status 404 returned error can't find the container with id a8ae926f34f6eea1e506c2ec41694b6c2a5601899d7b3d7f55339aa3718e73b1
Sep 08 16:01:14.421911 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:14.421893    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:01:14.431031 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:14.430876    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"]
Sep 08 16:01:14.440331 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:01:14.440308    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b96330b_5c01_48d5_acc0_422148a59dd3.slice/crio-155f1cca28a3d65ba6a4147fba098ddd2a907dffaa2639344b3d4d8751d1c721 WatchSource:0}: Error finding container 155f1cca28a3d65ba6a4147fba098ddd2a907dffaa2639344b3d4d8751d1c721: Status 404 returned error can't find the container with id 155f1cca28a3d65ba6a4147fba098ddd2a907dffaa2639344b3d4d8751d1c721
Sep 08 16:01:15.134166 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:15.134129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx" event={"ID":"8b96330b-5c01-48d5-acc0-422148a59dd3","Type":"ContainerStarted","Data":"92d1f6377ae412ed546f669089e83eee7af64a51fad363a589f5dd398d42a7a8"}
Sep 08 16:01:15.134602 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:15.134188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx" event={"ID":"8b96330b-5c01-48d5-acc0-422148a59dd3","Type":"ContainerStarted","Data":"155f1cca28a3d65ba6a4147fba098ddd2a907dffaa2639344b3d4d8751d1c721"}
Sep 08 16:01:15.134602 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:15.134214    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"
Sep 08 16:01:15.135722 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:15.135693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz" event={"ID":"9d8b1004-ff7e-449a-8351-e5b9d69ee84d","Type":"ContainerStarted","Data":"5e45d77acf6601c26a035077887c0b0f6acdb5b9dbaad12be248cd2cf83b9a0a"}
Sep 08 16:01:15.135847 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:15.135742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz" event={"ID":"9d8b1004-ff7e-449a-8351-e5b9d69ee84d","Type":"ContainerStarted","Data":"93c6d83d1b31da666fe3e7112726cdd0c71c22342be1575943b1d376aca42808"}
Sep 08 16:01:15.135847 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:15.135771    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"
Sep 08 16:01:15.136932 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:15.136904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w" event={"ID":"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff","Type":"ContainerStarted","Data":"a8ae926f34f6eea1e506c2ec41694b6c2a5601899d7b3d7f55339aa3718e73b1"}
Sep 08 16:01:15.156987 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:15.156944    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx" podStartSLOduration=2.156930515 podStartE2EDuration="2.156930515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:01:15.156488211 +0000 UTC m=+1021.780180538" watchObservedRunningTime="2026-09-08 16:01:15.156930515 +0000 UTC m=+1021.780622875"
Sep 08 16:01:15.176425 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:15.176373    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz" podStartSLOduration=2.176359349 podStartE2EDuration="2.176359349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:01:15.174918893 +0000 UTC m=+1021.798611236" watchObservedRunningTime="2026-09-08 16:01:15.176359349 +0000 UTC m=+1021.800051674"
Sep 08 16:01:17.150112 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:17.150074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w" event={"ID":"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff","Type":"ContainerStarted","Data":"575b3d10a79052fc191b5016963929b8937de53e4e1ec5b14e0e76de60b066f9"}
Sep 08 16:01:17.150536 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:17.150255    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:17.170009 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:17.169970    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w" podStartSLOduration=2.143758175 podStartE2EDuration="4.169958687s" totalImagesPullingTime="2.026200512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:01:17.169221161 +0000 UTC m=+1023.792913485" watchObservedRunningTime="2026-09-08 16:01:17.169958687 +0000 UTC m=+1023.793651010"
Sep 08 16:01:21.694964 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.694925    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"]
Sep 08 16:01:21.698467 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.698448    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"
Sep 08 16:01:21.700916 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.700897    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-13d3a5\""
Sep 08 16:01:21.709272 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.709248    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"]
Sep 08 16:01:21.748528 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.748508    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"]
Sep 08 16:01:21.755644 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.755619    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"
Sep 08 16:01:21.757732 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.757713    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-13d3a5-dockercfg-v6kjc\""
Sep 08 16:01:21.762271 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.762247    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"]
Sep 08 16:01:21.836913 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.836876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86\" (UID: \"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"
Sep 08 16:01:21.837073 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.836948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99nc7\" (UniqueName: \"kubernetes.io/projected/50d45a50-6e8d-456d-b7ec-28fe71254fc7-kube-api-access-99nc7\") pod \"payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7\" (UID: \"50d45a50-6e8d-456d-b7ec-28fe71254fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"
Sep 08 16:01:21.837073 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.836976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b47pz\" (UniqueName: \"kubernetes.io/projected/8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4-kube-api-access-b47pz\") pod \"payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86\" (UID: \"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"
Sep 08 16:01:21.837154 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.837077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d45a50-6e8d-456d-b7ec-28fe71254fc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7\" (UID: \"50d45a50-6e8d-456d-b7ec-28fe71254fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"
Sep 08 16:01:21.938307 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.938278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d45a50-6e8d-456d-b7ec-28fe71254fc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7\" (UID: \"50d45a50-6e8d-456d-b7ec-28fe71254fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"
Sep 08 16:01:21.938512 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.938333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86\" (UID: \"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"
Sep 08 16:01:21.938512 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.938409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99nc7\" (UniqueName: \"kubernetes.io/projected/50d45a50-6e8d-456d-b7ec-28fe71254fc7-kube-api-access-99nc7\") pod \"payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7\" (UID: \"50d45a50-6e8d-456d-b7ec-28fe71254fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"
Sep 08 16:01:21.938512 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.938431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b47pz\" (UniqueName: \"kubernetes.io/projected/8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4-kube-api-access-b47pz\") pod \"payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86\" (UID: \"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"
Sep 08 16:01:21.939024 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.939001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d45a50-6e8d-456d-b7ec-28fe71254fc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7\" (UID: \"50d45a50-6e8d-456d-b7ec-28fe71254fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"
Sep 08 16:01:21.939091 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.939014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86\" (UID: \"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"
Sep 08 16:01:21.946575 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.946510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99nc7\" (UniqueName: \"kubernetes.io/projected/50d45a50-6e8d-456d-b7ec-28fe71254fc7-kube-api-access-99nc7\") pod \"payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7\" (UID: \"50d45a50-6e8d-456d-b7ec-28fe71254fc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"
Sep 08 16:01:21.946575 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:21.946547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47pz\" (UniqueName: \"kubernetes.io/projected/8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4-kube-api-access-b47pz\") pod \"payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86\" (UID: \"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"
Sep 08 16:01:22.009120 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:22.009080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"
Sep 08 16:01:22.068128 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:22.067876    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"
Sep 08 16:01:22.296976 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:22.296946    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"]
Sep 08 16:01:22.297792 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:01:22.297762    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d45a50_6e8d_456d_b7ec_28fe71254fc7.slice/crio-bdc0112596f04f0350a95a88f5ed4bcf5560691fe12f36d3a4c94e1c54cf344f WatchSource:0}: Error finding container bdc0112596f04f0350a95a88f5ed4bcf5560691fe12f36d3a4c94e1c54cf344f: Status 404 returned error can't find the container with id bdc0112596f04f0350a95a88f5ed4bcf5560691fe12f36d3a4c94e1c54cf344f
Sep 08 16:01:22.384414 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:22.383805    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"]
Sep 08 16:01:22.395511 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:01:22.394911    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0fabcd_4cbd_4917_9c4b_e674ddfd3cb4.slice/crio-825ad6855a40c69ba01e200764cfb94e25ac26ae123e6251e1717511fb74fafe WatchSource:0}: Error finding container 825ad6855a40c69ba01e200764cfb94e25ac26ae123e6251e1717511fb74fafe: Status 404 returned error can't find the container with id 825ad6855a40c69ba01e200764cfb94e25ac26ae123e6251e1717511fb74fafe
Sep 08 16:01:23.162369 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:23.162347    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:01:23.208660 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:23.208621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86" event={"ID":"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4","Type":"ContainerStarted","Data":"75aeba122fd4a8b703960b162a9ef5aed0d2eefd1b3c753a73073f75d127e7da"}
Sep 08 16:01:23.208841 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:23.208687    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86" event={"ID":"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4","Type":"ContainerStarted","Data":"825ad6855a40c69ba01e200764cfb94e25ac26ae123e6251e1717511fb74fafe"}
Sep 08 16:01:23.208908 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:23.208854    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"
Sep 08 16:01:23.210864 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:23.210819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7" event={"ID":"50d45a50-6e8d-456d-b7ec-28fe71254fc7","Type":"ContainerStarted","Data":"689ff002b6610534526ce6c30e5f0eda0b3792fe320e6960c7ca0b9c714dd9a3"}
Sep 08 16:01:23.210992 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:23.210870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7" event={"ID":"50d45a50-6e8d-456d-b7ec-28fe71254fc7","Type":"ContainerStarted","Data":"bdc0112596f04f0350a95a88f5ed4bcf5560691fe12f36d3a4c94e1c54cf344f"}
Sep 08 16:01:23.210992 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:23.210898    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"
Sep 08 16:01:23.225789 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:23.225718    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86" podStartSLOduration=2.225700447 podStartE2EDuration="2.225700447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:01:23.223936084 +0000 UTC m=+1029.847628409" watchObservedRunningTime="2026-09-08 16:01:23.225700447 +0000 UTC m=+1029.849392788"
Sep 08 16:01:23.243296 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:23.243258    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7" podStartSLOduration=2.243244508 podStartE2EDuration="2.243244508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:01:23.242362014 +0000 UTC m=+1029.866054339" watchObservedRunningTime="2026-09-08 16:01:23.243244508 +0000 UTC m=+1029.866936842"
Sep 08 16:01:26.112419 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:26.112362    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 08 16:01:26.112872 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:26.112429    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 08 16:01:26.145285 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:26.145262    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"
Sep 08 16:01:26.145420 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:26.145328    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"
Sep 08 16:01:26.227612 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:26.227539    2581 generic.go:356] "Generic (PLEG): container finished" podID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerID="8cc99d2d69c27e72d644e8afce08b80d2ffea00857aaa6dca0ee1ad3335b9121" exitCode=0
Sep 08 16:01:26.227760 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:26.227621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" event={"ID":"49d4b521-010e-4f98-9e55-efba77ed63c0","Type":"ContainerDied","Data":"8cc99d2d69c27e72d644e8afce08b80d2ffea00857aaa6dca0ee1ad3335b9121"}
Sep 08 16:01:26.228161 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:26.228140    2581 scope.go:117] "RemoveContainer" containerID="8cc99d2d69c27e72d644e8afce08b80d2ffea00857aaa6dca0ee1ad3335b9121"
Sep 08 16:01:26.229467 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:26.229437    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:01:26.229586 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:26.229519    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:01:26.229586 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:26.229568    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:01:27.238570 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:27.238534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" event={"ID":"49d4b521-010e-4f98-9e55-efba77ed63c0","Type":"ContainerStarted","Data":"c46cdef1f9ba6ae01ff0640da2b7bb769896a7e4612d682c6ba4fcaedcdca750"}
Sep 08 16:01:27.878173 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:27.878137    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:27.878593 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:27.878562    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 08 16:01:27.878804 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:27.878776    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 08 16:01:28.878216 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:28.878176    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 08 16:01:28.878743 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:28.878248    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 08 16:01:29.878240 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:29.878197    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 08 16:01:29.878849 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:29.878814    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 08 16:01:30.877817 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:30.877781    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 08 16:01:30.878506 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:30.878087    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 08 16:01:31.878236 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:31.878191    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 08 16:01:31.878464 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:31.878271    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 08 16:01:32.881004 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:32.880973    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:32.881359 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:32.881248    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:32.881898 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:32.881881    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:01:34.218172 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:34.218145    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"
Sep 08 16:01:34.218619 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:34.218218    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"
Sep 08 16:01:43.378429 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.378381    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"]
Sep 08 16:01:43.381874 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.381853    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"
Sep 08 16:01:43.383980 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.383952    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-a09aa5\""
Sep 08 16:01:43.384100 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.384042    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-a09aa5-dockercfg-4lxz8\""
Sep 08 16:01:43.392729 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.392707    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"]
Sep 08 16:01:43.402711 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.402688    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"]
Sep 08 16:01:43.406196 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.406178    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"
Sep 08 16:01:43.420089 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.420069    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"]
Sep 08 16:01:43.528785 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.528751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2pph\" (UniqueName: \"kubernetes.io/projected/e98fd537-6f64-425e-8ee1-6930ff87564c-kube-api-access-g2pph\") pod \"payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp\" (UID: \"e98fd537-6f64-425e-8ee1-6930ff87564c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"
Sep 08 16:01:43.528950 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.528789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd2nx\" (UniqueName: \"kubernetes.io/projected/9665892f-5c67-4c10-8263-bbde957aa053-kube-api-access-dd2nx\") pod \"payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv\" (UID: \"9665892f-5c67-4c10-8263-bbde957aa053\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"
Sep 08 16:01:43.528950 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.528823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9665892f-5c67-4c10-8263-bbde957aa053-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv\" (UID: \"9665892f-5c67-4c10-8263-bbde957aa053\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"
Sep 08 16:01:43.528950 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.528935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98fd537-6f64-425e-8ee1-6930ff87564c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp\" (UID: \"e98fd537-6f64-425e-8ee1-6930ff87564c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"
Sep 08 16:01:43.629946 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.629860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2pph\" (UniqueName: \"kubernetes.io/projected/e98fd537-6f64-425e-8ee1-6930ff87564c-kube-api-access-g2pph\") pod \"payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp\" (UID: \"e98fd537-6f64-425e-8ee1-6930ff87564c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"
Sep 08 16:01:43.629946 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.629900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd2nx\" (UniqueName: \"kubernetes.io/projected/9665892f-5c67-4c10-8263-bbde957aa053-kube-api-access-dd2nx\") pod \"payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv\" (UID: \"9665892f-5c67-4c10-8263-bbde957aa053\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"
Sep 08 16:01:43.630167 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.630025    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9665892f-5c67-4c10-8263-bbde957aa053-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv\" (UID: \"9665892f-5c67-4c10-8263-bbde957aa053\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"
Sep 08 16:01:43.630167 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.630097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98fd537-6f64-425e-8ee1-6930ff87564c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp\" (UID: \"e98fd537-6f64-425e-8ee1-6930ff87564c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"
Sep 08 16:01:43.630807 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.630790    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9665892f-5c67-4c10-8263-bbde957aa053-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv\" (UID: \"9665892f-5c67-4c10-8263-bbde957aa053\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"
Sep 08 16:01:43.630865 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.630807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98fd537-6f64-425e-8ee1-6930ff87564c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp\" (UID: \"e98fd537-6f64-425e-8ee1-6930ff87564c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"
Sep 08 16:01:43.637702 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.637675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd2nx\" (UniqueName: \"kubernetes.io/projected/9665892f-5c67-4c10-8263-bbde957aa053-kube-api-access-dd2nx\") pod \"payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv\" (UID: \"9665892f-5c67-4c10-8263-bbde957aa053\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"
Sep 08 16:01:43.637787 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.637736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2pph\" (UniqueName: \"kubernetes.io/projected/e98fd537-6f64-425e-8ee1-6930ff87564c-kube-api-access-g2pph\") pod \"payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp\" (UID: \"e98fd537-6f64-425e-8ee1-6930ff87564c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"
Sep 08 16:01:43.692524 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.692503    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"
Sep 08 16:01:43.719745 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.719720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"
Sep 08 16:01:43.968538 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:43.968513    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"]
Sep 08 16:01:43.970790 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:01:43.970749    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9665892f_5c67_4c10_8263_bbde957aa053.slice/crio-a837a9e888426a265339356f7f2f80a98477c01e72a5f0e06128ec323928d74c WatchSource:0}: Error finding container a837a9e888426a265339356f7f2f80a98477c01e72a5f0e06128ec323928d74c: Status 404 returned error can't find the container with id a837a9e888426a265339356f7f2f80a98477c01e72a5f0e06128ec323928d74c
Sep 08 16:01:44.002338 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:44.002316    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"]
Sep 08 16:01:44.009423 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:01:44.009385    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98fd537_6f64_425e_8ee1_6930ff87564c.slice/crio-69337d3855cfa7a019948ad4080cc05672d32cb446cfeb05fbeb197a8d26370d WatchSource:0}: Error finding container 69337d3855cfa7a019948ad4080cc05672d32cb446cfeb05fbeb197a8d26370d: Status 404 returned error can't find the container with id 69337d3855cfa7a019948ad4080cc05672d32cb446cfeb05fbeb197a8d26370d
Sep 08 16:01:44.324765 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:44.324670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp" event={"ID":"e98fd537-6f64-425e-8ee1-6930ff87564c","Type":"ContainerStarted","Data":"66949c58c31f55b86c4c28d9bf1047dfdfd5dcbc8a8ddfb8e8151d8022dc747d"}
Sep 08 16:01:44.324929 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:44.324782    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"
Sep 08 16:01:44.324929 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:44.324811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp" event={"ID":"e98fd537-6f64-425e-8ee1-6930ff87564c","Type":"ContainerStarted","Data":"69337d3855cfa7a019948ad4080cc05672d32cb446cfeb05fbeb197a8d26370d"}
Sep 08 16:01:44.326350 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:44.326326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv" event={"ID":"9665892f-5c67-4c10-8263-bbde957aa053","Type":"ContainerStarted","Data":"f0803866168f0c89c55213d4db51c61508db94dc808e621efbdcca1f1c0a8f1b"}
Sep 08 16:01:44.326488 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:44.326369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv" event={"ID":"9665892f-5c67-4c10-8263-bbde957aa053","Type":"ContainerStarted","Data":"a837a9e888426a265339356f7f2f80a98477c01e72a5f0e06128ec323928d74c"}
Sep 08 16:01:44.326488 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:44.326439    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"
Sep 08 16:01:44.341213 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:44.341175    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp" podStartSLOduration=1.341162114 podStartE2EDuration="1.341162114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:01:44.339611755 +0000 UTC m=+1050.963304080" watchObservedRunningTime="2026-09-08 16:01:44.341162114 +0000 UTC m=+1050.964854437"
Sep 08 16:01:44.358496 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:44.358444    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv" podStartSLOduration=1.358430788 podStartE2EDuration="1.358430788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:01:44.355527729 +0000 UTC m=+1050.979220053" watchObservedRunningTime="2026-09-08 16:01:44.358430788 +0000 UTC m=+1050.982123113"
Sep 08 16:01:50.016179 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.016143    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"]
Sep 08 16:01:50.020987 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.020966    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"
Sep 08 16:01:50.023134 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.023107    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-e69292\""
Sep 08 16:01:50.023211 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.023160    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-e69292-dockercfg-96758\""
Sep 08 16:01:50.030433 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.030407    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"]
Sep 08 16:01:50.053420 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.053361    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"]
Sep 08 16:01:50.057455 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.057432    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"
Sep 08 16:01:50.063435 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.063409    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"]
Sep 08 16:01:50.087478 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.087455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht772\" (UniqueName: \"kubernetes.io/projected/5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4-kube-api-access-ht772\") pod \"payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn\" (UID: \"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"
Sep 08 16:01:50.087596 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.087489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e3923a-caa4-46b3-8f0b-1abaf2750ee4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr\" (UID: \"40e3923a-caa4-46b3-8f0b-1abaf2750ee4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"
Sep 08 16:01:50.087596 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.087526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn\" (UID: \"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"
Sep 08 16:01:50.087684 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.087642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fbz5\" (UniqueName: \"kubernetes.io/projected/40e3923a-caa4-46b3-8f0b-1abaf2750ee4-kube-api-access-4fbz5\") pod \"payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr\" (UID: \"40e3923a-caa4-46b3-8f0b-1abaf2750ee4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"
Sep 08 16:01:50.188345 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.188311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn\" (UID: \"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"
Sep 08 16:01:50.188553 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.188483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fbz5\" (UniqueName: \"kubernetes.io/projected/40e3923a-caa4-46b3-8f0b-1abaf2750ee4-kube-api-access-4fbz5\") pod \"payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr\" (UID: \"40e3923a-caa4-46b3-8f0b-1abaf2750ee4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"
Sep 08 16:01:50.188651 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.188632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht772\" (UniqueName: \"kubernetes.io/projected/5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4-kube-api-access-ht772\") pod \"payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn\" (UID: \"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"
Sep 08 16:01:50.188713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.188666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e3923a-caa4-46b3-8f0b-1abaf2750ee4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr\" (UID: \"40e3923a-caa4-46b3-8f0b-1abaf2750ee4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"
Sep 08 16:01:50.189063 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.189041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn\" (UID: \"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"
Sep 08 16:01:50.189344 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.189323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e3923a-caa4-46b3-8f0b-1abaf2750ee4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr\" (UID: \"40e3923a-caa4-46b3-8f0b-1abaf2750ee4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"
Sep 08 16:01:50.196342 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.196321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fbz5\" (UniqueName: \"kubernetes.io/projected/40e3923a-caa4-46b3-8f0b-1abaf2750ee4-kube-api-access-4fbz5\") pod \"payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr\" (UID: \"40e3923a-caa4-46b3-8f0b-1abaf2750ee4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"
Sep 08 16:01:50.196461 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.196323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht772\" (UniqueName: \"kubernetes.io/projected/5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4-kube-api-access-ht772\") pod \"payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn\" (UID: \"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"
Sep 08 16:01:50.332652 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.332558    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"
Sep 08 16:01:50.369045 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.369014    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"
Sep 08 16:01:50.629320 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.629287    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"]
Sep 08 16:01:50.631360 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:01:50.631330    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf9e265_2f80_4e3e_bea3_baf64e6a7eb4.slice/crio-5d673ac781b34a7945b1c2fc3913dc1e30a13aa79184357442368f824edfe6c6 WatchSource:0}: Error finding container 5d673ac781b34a7945b1c2fc3913dc1e30a13aa79184357442368f824edfe6c6: Status 404 returned error can't find the container with id 5d673ac781b34a7945b1c2fc3913dc1e30a13aa79184357442368f824edfe6c6
Sep 08 16:01:50.661898 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:50.661879    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"]
Sep 08 16:01:50.669455 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:01:50.669433    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e3923a_caa4_46b3_8f0b_1abaf2750ee4.slice/crio-81001e04a7af3b7d0bf46110315cba6e1d678ff6c37343676e34a2530548a3b9 WatchSource:0}: Error finding container 81001e04a7af3b7d0bf46110315cba6e1d678ff6c37343676e34a2530548a3b9: Status 404 returned error can't find the container with id 81001e04a7af3b7d0bf46110315cba6e1d678ff6c37343676e34a2530548a3b9
Sep 08 16:01:51.362874 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:51.362835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr" event={"ID":"40e3923a-caa4-46b3-8f0b-1abaf2750ee4","Type":"ContainerStarted","Data":"3afa7b2237e0608771f4e80c5ab62bebc28a21c74070f43ab1b902e5a0dceaee"}
Sep 08 16:01:51.363315 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:51.362906    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"
Sep 08 16:01:51.363315 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:51.362930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr" event={"ID":"40e3923a-caa4-46b3-8f0b-1abaf2750ee4","Type":"ContainerStarted","Data":"81001e04a7af3b7d0bf46110315cba6e1d678ff6c37343676e34a2530548a3b9"}
Sep 08 16:01:51.364598 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:51.364565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn" event={"ID":"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4","Type":"ContainerStarted","Data":"8c0a55d237b554f3465162afbfcc8cc9407f42ae0b1312ea9eba5fc261326e4a"}
Sep 08 16:01:51.364724 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:51.364621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn" event={"ID":"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4","Type":"ContainerStarted","Data":"5d673ac781b34a7945b1c2fc3913dc1e30a13aa79184357442368f824edfe6c6"}
Sep 08 16:01:51.364724 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:51.364651    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"
Sep 08 16:01:51.382445 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:51.382368    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr" podStartSLOduration=1.3823525970000001 podStartE2EDuration="1.382352597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:01:51.380490562 +0000 UTC m=+1058.004182909" watchObservedRunningTime="2026-09-08 16:01:51.382352597 +0000 UTC m=+1058.006044921"
Sep 08 16:01:51.400272 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:51.400187    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn" podStartSLOduration=1.400175119 podStartE2EDuration="1.400175119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:01:51.39836188 +0000 UTC m=+1058.022054208" watchObservedRunningTime="2026-09-08 16:01:51.400175119 +0000 UTC m=+1058.023867442"
Sep 08 16:01:55.333946 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:55.333911    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"
Sep 08 16:01:55.334355 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:01:55.334178    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"
Sep 08 16:02:02.372634 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:02.372599    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"
Sep 08 16:02:02.373061 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:02.372831    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"
Sep 08 16:02:14.088977 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:14.088948    2581 scope.go:117] "RemoveContainer" containerID="f040af2ea3e41123133d5031b6e47043acb9d501559a74e2a3fcac471fbee8f0"
Sep 08 16:02:32.576523 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.576491    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"]
Sep 08 16:02:32.580513 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.580489    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:32.583456 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.583427    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-510a51a5-serving-cert\""
Sep 08 16:02:32.591923 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.591178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-maas-api-tls\") pod \"maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:32.591923 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.591236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-openshift-service-ca\") pod \"maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:32.591923 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.591265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4m2z\" (UniqueName: \"kubernetes.io/projected/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-kube-api-access-b4m2z\") pod \"maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:32.591923 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.591380    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-metrics-certs\") pod \"maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:32.591923 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.591783    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"]
Sep 08 16:02:32.692089 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.692055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-maas-api-tls\") pod \"maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:32.692274 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.692114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-openshift-service-ca\") pod \"maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:32.692274 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.692140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4m2z\" (UniqueName: \"kubernetes.io/projected/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-kube-api-access-b4m2z\") pod \"maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:32.692274 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:02:32.692205    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-serving-cert: secret "maas-api-e2e-mt-510a51a5-serving-cert" not found
Sep 08 16:02:32.692274 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.692223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-metrics-certs\") pod \"maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:32.692474 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:02:32.692296    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-maas-api-tls podName:b3d8f4ec-a8ce-4437-a03a-9e1936cca655 nodeName:}" failed. No retries permitted until 2026-09-08 16:02:33.192273826 +0000 UTC m=+1099.815966148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-maas-api-tls") pod "maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69" (UID: "b3d8f4ec-a8ce-4437-a03a-9e1936cca655") : secret "maas-api-e2e-mt-510a51a5-serving-cert" not found
Sep 08 16:02:32.692817 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.692798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-openshift-service-ca\") pod \"maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:32.695045 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.695021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-metrics-certs\") pod \"maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:32.704261 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:32.704229    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4m2z\" (UniqueName: \"kubernetes.io/projected/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-kube-api-access-b4m2z\") pod \"maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:33.197018 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:33.196975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-maas-api-tls\") pod \"maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:33.199377 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:33.199353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-maas-api-tls\") pod \"maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:33.495451 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:33.495341    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:33.771555 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:33.771475    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"]
Sep 08 16:02:33.773926 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:02:33.773899    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d8f4ec_a8ce_4437_a03a_9e1936cca655.slice/crio-33895fb7d01373b4310ab42b9cd76b5d5092300870cd2e81dedb11d09d2fe766 WatchSource:0}: Error finding container 33895fb7d01373b4310ab42b9cd76b5d5092300870cd2e81dedb11d09d2fe766: Status 404 returned error can't find the container with id 33895fb7d01373b4310ab42b9cd76b5d5092300870cd2e81dedb11d09d2fe766
Sep 08 16:02:34.593336 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:34.593236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69" event={"ID":"b3d8f4ec-a8ce-4437-a03a-9e1936cca655","Type":"ContainerStarted","Data":"21cbffe0bfd2129b5dd10170db111d7bb96013bc7d4f92635685e4097fa3039a"}
Sep 08 16:02:34.593336 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:34.593299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69" event={"ID":"b3d8f4ec-a8ce-4437-a03a-9e1936cca655","Type":"ContainerStarted","Data":"33895fb7d01373b4310ab42b9cd76b5d5092300870cd2e81dedb11d09d2fe766"}
Sep 08 16:02:34.593336 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:34.593329    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:02:34.609686 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:34.609641    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69" podStartSLOduration=2.183506013 podStartE2EDuration="2.609622344s" totalImagesPullingTime="426.116331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:02:34.608383142 +0000 UTC m=+1101.232075466" watchObservedRunningTime="2026-09-08 16:02:34.609622344 +0000 UTC m=+1101.233314667"
Sep 08 16:02:40.604364 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:02:40.604334    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:03:57.265539 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.265504    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"]
Sep 08 16:03:57.266162 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.265706    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69" podUID="b3d8f4ec-a8ce-4437-a03a-9e1936cca655" containerName="maas-api" containerID="cri-o://21cbffe0bfd2129b5dd10170db111d7bb96013bc7d4f92635685e4097fa3039a" gracePeriod=30
Sep 08 16:03:57.509322 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.509298    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:03:57.574868 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.574806    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4m2z\" (UniqueName: \"kubernetes.io/projected/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-kube-api-access-b4m2z\") pod \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") "
Sep 08 16:03:57.574868 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.574846    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-maas-api-tls\") pod \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") "
Sep 08 16:03:57.575027 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.574878    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-metrics-certs\") pod \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") "
Sep 08 16:03:57.575027 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.574935    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-openshift-service-ca\") pod \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\" (UID: \"b3d8f4ec-a8ce-4437-a03a-9e1936cca655\") "
Sep 08 16:03:57.575371 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.575337    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b3d8f4ec-a8ce-4437-a03a-9e1936cca655" (UID: "b3d8f4ec-a8ce-4437-a03a-9e1936cca655"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:03:57.576936 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.576907    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b3d8f4ec-a8ce-4437-a03a-9e1936cca655" (UID: "b3d8f4ec-a8ce-4437-a03a-9e1936cca655"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:03:57.576936 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.576922    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b3d8f4ec-a8ce-4437-a03a-9e1936cca655" (UID: "b3d8f4ec-a8ce-4437-a03a-9e1936cca655"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:03:57.577073 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.576981    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-kube-api-access-b4m2z" (OuterVolumeSpecName: "kube-api-access-b4m2z") pod "b3d8f4ec-a8ce-4437-a03a-9e1936cca655" (UID: "b3d8f4ec-a8ce-4437-a03a-9e1936cca655"). InnerVolumeSpecName "kube-api-access-b4m2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:03:57.675636 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.675609    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-openshift-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:03:57.675636 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.675635    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4m2z\" (UniqueName: \"kubernetes.io/projected/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-kube-api-access-b4m2z\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:03:57.675816 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.675645    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-maas-api-tls\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:03:57.675816 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:57.675653    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d8f4ec-a8ce-4437-a03a-9e1936cca655-metrics-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:03:58.029264 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:58.029227    2581 generic.go:356] "Generic (PLEG): container finished" podID="b3d8f4ec-a8ce-4437-a03a-9e1936cca655" containerID="21cbffe0bfd2129b5dd10170db111d7bb96013bc7d4f92635685e4097fa3039a" exitCode=0
Sep 08 16:03:58.029455 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:58.029282    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"
Sep 08 16:03:58.029455 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:58.029314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69" event={"ID":"b3d8f4ec-a8ce-4437-a03a-9e1936cca655","Type":"ContainerDied","Data":"21cbffe0bfd2129b5dd10170db111d7bb96013bc7d4f92635685e4097fa3039a"}
Sep 08 16:03:58.029455 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:58.029375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69" event={"ID":"b3d8f4ec-a8ce-4437-a03a-9e1936cca655","Type":"ContainerDied","Data":"33895fb7d01373b4310ab42b9cd76b5d5092300870cd2e81dedb11d09d2fe766"}
Sep 08 16:03:58.029455 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:58.029420    2581 scope.go:117] "RemoveContainer" containerID="21cbffe0bfd2129b5dd10170db111d7bb96013bc7d4f92635685e4097fa3039a"
Sep 08 16:03:58.040980 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:58.040941    2581 scope.go:117] "RemoveContainer" containerID="21cbffe0bfd2129b5dd10170db111d7bb96013bc7d4f92635685e4097fa3039a"
Sep 08 16:03:58.041277 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:03:58.041257    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21cbffe0bfd2129b5dd10170db111d7bb96013bc7d4f92635685e4097fa3039a\": container with ID starting with 21cbffe0bfd2129b5dd10170db111d7bb96013bc7d4f92635685e4097fa3039a not found: ID does not exist" containerID="21cbffe0bfd2129b5dd10170db111d7bb96013bc7d4f92635685e4097fa3039a"
Sep 08 16:03:58.041349 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:58.041288    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21cbffe0bfd2129b5dd10170db111d7bb96013bc7d4f92635685e4097fa3039a"} err="failed to get container status \"21cbffe0bfd2129b5dd10170db111d7bb96013bc7d4f92635685e4097fa3039a\": rpc error: code = NotFound desc = could not find container \"21cbffe0bfd2129b5dd10170db111d7bb96013bc7d4f92635685e4097fa3039a\": container with ID starting with 21cbffe0bfd2129b5dd10170db111d7bb96013bc7d4f92635685e4097fa3039a not found: ID does not exist"
Sep 08 16:03:58.044302 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:58.044282    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"]
Sep 08 16:03:58.051117 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:58.051097    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-510a51a5-59cc75b55f-d5q69"]
Sep 08 16:03:59.900844 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:03:59.900811    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3d8f4ec-a8ce-4437-a03a-9e1936cca655" path="/var/lib/kubelet/pods/b3d8f4ec-a8ce-4437-a03a-9e1936cca655/volumes"
Sep 08 16:04:13.907933 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:04:13.907901    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 16:04:13.909769 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:04:13.909742    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 16:06:01.878377 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:01.878339    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"]
Sep 08 16:06:01.879073 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:01.879048    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d8f4ec-a8ce-4437-a03a-9e1936cca655" containerName="maas-api"
Sep 08 16:06:01.879073 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:01.879075    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d8f4ec-a8ce-4437-a03a-9e1936cca655" containerName="maas-api"
Sep 08 16:06:01.879240 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:01.879178    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3d8f4ec-a8ce-4437-a03a-9e1936cca655" containerName="maas-api"
Sep 08 16:06:01.882807 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:01.882773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:01.885235 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:01.885213    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5214de-serving-cert\""
Sep 08 16:06:01.905227 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:01.905186    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"]
Sep 08 16:06:01.907709 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:01.907680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0d14f0-713d-45ec-814d-a0a25a256a9e-maas-api-tls\") pod \"maas-api-e2e-api-a-5214de-5fc4848f76-h8g48\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:01.907878 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:01.907766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0d14f0-713d-45ec-814d-a0a25a256a9e-metrics-certs\") pod \"maas-api-e2e-api-a-5214de-5fc4848f76-h8g48\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:01.907878 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:01.907823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p56kd\" (UniqueName: \"kubernetes.io/projected/7b0d14f0-713d-45ec-814d-a0a25a256a9e-kube-api-access-p56kd\") pod \"maas-api-e2e-api-a-5214de-5fc4848f76-h8g48\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:01.907878 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:01.907856    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0d14f0-713d-45ec-814d-a0a25a256a9e-openshift-service-ca\") pod \"maas-api-e2e-api-a-5214de-5fc4848f76-h8g48\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:02.009144 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.009117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0d14f0-713d-45ec-814d-a0a25a256a9e-maas-api-tls\") pod \"maas-api-e2e-api-a-5214de-5fc4848f76-h8g48\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:02.009254 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.009171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0d14f0-713d-45ec-814d-a0a25a256a9e-metrics-certs\") pod \"maas-api-e2e-api-a-5214de-5fc4848f76-h8g48\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:02.009254 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.009199    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p56kd\" (UniqueName: \"kubernetes.io/projected/7b0d14f0-713d-45ec-814d-a0a25a256a9e-kube-api-access-p56kd\") pod \"maas-api-e2e-api-a-5214de-5fc4848f76-h8g48\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:02.009254 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.009219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0d14f0-713d-45ec-814d-a0a25a256a9e-openshift-service-ca\") pod \"maas-api-e2e-api-a-5214de-5fc4848f76-h8g48\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:02.009921 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.009900    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0d14f0-713d-45ec-814d-a0a25a256a9e-openshift-service-ca\") pod \"maas-api-e2e-api-a-5214de-5fc4848f76-h8g48\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:02.011604 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.011584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0d14f0-713d-45ec-814d-a0a25a256a9e-maas-api-tls\") pod \"maas-api-e2e-api-a-5214de-5fc4848f76-h8g48\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:02.011685 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.011617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0d14f0-713d-45ec-814d-a0a25a256a9e-metrics-certs\") pod \"maas-api-e2e-api-a-5214de-5fc4848f76-h8g48\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:02.022414 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.022371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p56kd\" (UniqueName: \"kubernetes.io/projected/7b0d14f0-713d-45ec-814d-a0a25a256a9e-kube-api-access-p56kd\") pod \"maas-api-e2e-api-a-5214de-5fc4848f76-h8g48\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:02.200415 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.200365    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:02.474585 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.474558    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"]
Sep 08 16:06:02.476654 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:06:02.476620    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0d14f0_713d_45ec_814d_a0a25a256a9e.slice/crio-d6cd50626ab31b62cd0f8fb4cb094e49628b9590ce0abc82a9e9d810b8debe32 WatchSource:0}: Error finding container d6cd50626ab31b62cd0f8fb4cb094e49628b9590ce0abc82a9e9d810b8debe32: Status 404 returned error can't find the container with id d6cd50626ab31b62cd0f8fb4cb094e49628b9590ce0abc82a9e9d810b8debe32
Sep 08 16:06:02.602791 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.602761    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"]
Sep 08 16:06:02.607489 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.607469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"
Sep 08 16:06:02.609750 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.609721    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5214de-dockercfg-gl5rc\""
Sep 08 16:06:02.609859 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.609721    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5214de\""
Sep 08 16:06:02.623645 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.623617    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"]
Sep 08 16:06:02.674077 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.674049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48" event={"ID":"7b0d14f0-713d-45ec-814d-a0a25a256a9e","Type":"ContainerStarted","Data":"d6cd50626ab31b62cd0f8fb4cb094e49628b9590ce0abc82a9e9d810b8debe32"}
Sep 08 16:06:02.714621 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.714595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9zv9\" (UniqueName: \"kubernetes.io/projected/3518251d-ddfc-4c6d-a17f-a7fed5c9418b-kube-api-access-m9zv9\") pod \"payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8\" (UID: \"3518251d-ddfc-4c6d-a17f-a7fed5c9418b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"
Sep 08 16:06:02.714726 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.714664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3518251d-ddfc-4c6d-a17f-a7fed5c9418b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8\" (UID: \"3518251d-ddfc-4c6d-a17f-a7fed5c9418b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"
Sep 08 16:06:02.815213 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.815135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9zv9\" (UniqueName: \"kubernetes.io/projected/3518251d-ddfc-4c6d-a17f-a7fed5c9418b-kube-api-access-m9zv9\") pod \"payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8\" (UID: \"3518251d-ddfc-4c6d-a17f-a7fed5c9418b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"
Sep 08 16:06:02.815364 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.815222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3518251d-ddfc-4c6d-a17f-a7fed5c9418b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8\" (UID: \"3518251d-ddfc-4c6d-a17f-a7fed5c9418b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"
Sep 08 16:06:02.815926 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.815906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3518251d-ddfc-4c6d-a17f-a7fed5c9418b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8\" (UID: \"3518251d-ddfc-4c6d-a17f-a7fed5c9418b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"
Sep 08 16:06:02.823252 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.823233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9zv9\" (UniqueName: \"kubernetes.io/projected/3518251d-ddfc-4c6d-a17f-a7fed5c9418b-kube-api-access-m9zv9\") pod \"payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8\" (UID: \"3518251d-ddfc-4c6d-a17f-a7fed5c9418b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"
Sep 08 16:06:02.918564 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:02.918538    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"
Sep 08 16:06:03.189594 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:03.189564    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"]
Sep 08 16:06:03.191457 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:06:03.191432    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3518251d_ddfc_4c6d_a17f_a7fed5c9418b.slice/crio-8d3a357faa664d6621ba5fee6ea33a3b3317128c3cc119f1628b68907e81677a WatchSource:0}: Error finding container 8d3a357faa664d6621ba5fee6ea33a3b3317128c3cc119f1628b68907e81677a: Status 404 returned error can't find the container with id 8d3a357faa664d6621ba5fee6ea33a3b3317128c3cc119f1628b68907e81677a
Sep 08 16:06:03.680945 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:03.680908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8" event={"ID":"3518251d-ddfc-4c6d-a17f-a7fed5c9418b","Type":"ContainerStarted","Data":"1ed682afbecd8b2f74642a1827a3307c858209ccb76d9fabf9c26ab6c39f87de"}
Sep 08 16:06:03.681138 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:03.680983    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"
Sep 08 16:06:03.681138 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:03.681006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8" event={"ID":"3518251d-ddfc-4c6d-a17f-a7fed5c9418b","Type":"ContainerStarted","Data":"8d3a357faa664d6621ba5fee6ea33a3b3317128c3cc119f1628b68907e81677a"}
Sep 08 16:06:03.696245 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:03.696203    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8" podStartSLOduration=1.696188335 podStartE2EDuration="1.696188335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:06:03.695040916 +0000 UTC m=+1310.318733241" watchObservedRunningTime="2026-09-08 16:06:03.696188335 +0000 UTC m=+1310.319880689"
Sep 08 16:06:05.694945 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:05.694905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48" event={"ID":"7b0d14f0-713d-45ec-814d-a0a25a256a9e","Type":"ContainerStarted","Data":"05733455e11e499da8eb7b4275345219c5e9e5cfea3b4162e09517ea35e22055"}
Sep 08 16:06:05.695413 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:05.695253    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:05.714670 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:05.714626    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48" podStartSLOduration=2.172693153 podStartE2EDuration="4.714612048s" totalImagesPullingTime="2.541918895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:06:05.712296547 +0000 UTC m=+1312.335988888" watchObservedRunningTime="2026-09-08 16:06:05.714612048 +0000 UTC m=+1312.338304419"
Sep 08 16:06:11.707400 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:11.707368    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:06:14.135537 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:14.135508    2581 scope.go:117] "RemoveContainer" containerID="01dd5c0cba5c9ff8571068f536bde815f0aaa567172816f02c1991f0de555799"
Sep 08 16:06:14.144130 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:14.144113    2581 scope.go:117] "RemoveContainer" containerID="87b5f097dfa69fba669765e59beb5f8628b892f04b7671791c6a9c5e698187d6"
Sep 08 16:06:14.688765 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:06:14.688734    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"
Sep 08 16:07:01.170613 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.170582    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"]
Sep 08 16:07:01.174447 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.174426    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.178010 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.177986    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a09aa5\"/\"rate-test-model-a09aa5-kserve-self-signed-certs\""
Sep 08 16:07:01.178010 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.177999    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a09aa5\"/\"default-dockercfg-shzhl\""
Sep 08 16:07:01.178184 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.178000    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a09aa5\"/\"kube-root-ca.crt\""
Sep 08 16:07:01.178184 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.177986    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a09aa5\"/\"openshift-service-ca.crt\""
Sep 08 16:07:01.184710 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.184686    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"]
Sep 08 16:07:01.256369 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.256333    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-tmp-dir\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.256579 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.256451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-model-cache\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.256579 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.256482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh769\" (UniqueName: \"kubernetes.io/projected/6ff9f21a-cd71-44ab-b5af-06384696284f-kube-api-access-fh769\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.256579 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.256549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff9f21a-cd71-44ab-b5af-06384696284f-tls-certs\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.256708 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.256615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-home\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.256708 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.256659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-dshm\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.357265 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.357228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-model-cache\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.357265 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.357268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh769\" (UniqueName: \"kubernetes.io/projected/6ff9f21a-cd71-44ab-b5af-06384696284f-kube-api-access-fh769\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.357583 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.357289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff9f21a-cd71-44ab-b5af-06384696284f-tls-certs\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.357583 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.357331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-home\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.357583 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.357356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-dshm\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.357583 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.357384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-tmp-dir\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.357763 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.357713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-model-cache\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.357763 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.357738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-home\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.357842 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.357786    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-tmp-dir\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.359744 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.359727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-dshm\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.359854 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.359838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff9f21a-cd71-44ab-b5af-06384696284f-tls-certs\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.368093 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.368066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh769\" (UniqueName: \"kubernetes.io/projected/6ff9f21a-cd71-44ab-b5af-06384696284f-kube-api-access-fh769\") pod \"rate-test-model-a09aa5-kserve-7bd44b5799-qtzww\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") " pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.485157 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.485078    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:01.760183 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.760154    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"]
Sep 08 16:07:01.762430 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:07:01.762385    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ff9f21a_cd71_44ab_b5af_06384696284f.slice/crio-80053a875c372e8627c8a34d1fc9ed65ceb353001fc26a3d2ffc025181fe969e WatchSource:0}: Error finding container 80053a875c372e8627c8a34d1fc9ed65ceb353001fc26a3d2ffc025181fe969e: Status 404 returned error can't find the container with id 80053a875c372e8627c8a34d1fc9ed65ceb353001fc26a3d2ffc025181fe969e
Sep 08 16:07:01.988124 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.988088    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww" event={"ID":"6ff9f21a-cd71-44ab-b5af-06384696284f","Type":"ContainerStarted","Data":"2172d061704b9dc325c5905d0fe8dfcdc166eab81458747da57eacdb4eaf904d"}
Sep 08 16:07:01.988124 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:01.988131    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww" event={"ID":"6ff9f21a-cd71-44ab-b5af-06384696284f","Type":"ContainerStarted","Data":"80053a875c372e8627c8a34d1fc9ed65ceb353001fc26a3d2ffc025181fe969e"}
Sep 08 16:07:02.006659 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:02.006610    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww" podStartSLOduration=1.006595173 podStartE2EDuration="1.006595173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:07:02.005067044 +0000 UTC m=+1368.628759368" watchObservedRunningTime="2026-09-08 16:07:02.006595173 +0000 UTC m=+1368.630287496"
Sep 08 16:07:11.485812 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:11.485779    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:11.486192 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:11.485822    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:11.498584 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:11.498552    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:12.052011 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.051981    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:07:12.473906 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.473871    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"]
Sep 08 16:07:12.474156 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.474129    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48" podUID="7b0d14f0-713d-45ec-814d-a0a25a256a9e" containerName="maas-api" containerID="cri-o://05733455e11e499da8eb7b4275345219c5e9e5cfea3b4162e09517ea35e22055" gracePeriod=30
Sep 08 16:07:12.531855 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.531814    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"]
Sep 08 16:07:12.532293 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.532051    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8" podUID="3518251d-ddfc-4c6d-a17f-a7fed5c9418b" containerName="payload-pre-processing" containerID="cri-o://1ed682afbecd8b2f74642a1827a3307c858209ccb76d9fabf9c26ab6c39f87de" gracePeriod=30
Sep 08 16:07:12.781305 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.781275    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:07:12.822512 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.822481    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"
Sep 08 16:07:12.870013 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.869987    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0d14f0-713d-45ec-814d-a0a25a256a9e-metrics-certs\") pod \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") "
Sep 08 16:07:12.870172 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.870040    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p56kd\" (UniqueName: \"kubernetes.io/projected/7b0d14f0-713d-45ec-814d-a0a25a256a9e-kube-api-access-p56kd\") pod \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") "
Sep 08 16:07:12.870172 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.870067    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0d14f0-713d-45ec-814d-a0a25a256a9e-openshift-service-ca\") pod \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") "
Sep 08 16:07:12.870172 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.870094    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0d14f0-713d-45ec-814d-a0a25a256a9e-maas-api-tls\") pod \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\" (UID: \"7b0d14f0-713d-45ec-814d-a0a25a256a9e\") "
Sep 08 16:07:12.870459 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.870432    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0d14f0-713d-45ec-814d-a0a25a256a9e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b0d14f0-713d-45ec-814d-a0a25a256a9e" (UID: "7b0d14f0-713d-45ec-814d-a0a25a256a9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:07:12.872178 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.872147    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0d14f0-713d-45ec-814d-a0a25a256a9e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7b0d14f0-713d-45ec-814d-a0a25a256a9e" (UID: "7b0d14f0-713d-45ec-814d-a0a25a256a9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:07:12.872284 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.872212    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0d14f0-713d-45ec-814d-a0a25a256a9e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b0d14f0-713d-45ec-814d-a0a25a256a9e" (UID: "7b0d14f0-713d-45ec-814d-a0a25a256a9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:07:12.872284 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.872216    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0d14f0-713d-45ec-814d-a0a25a256a9e-kube-api-access-p56kd" (OuterVolumeSpecName: "kube-api-access-p56kd") pod "7b0d14f0-713d-45ec-814d-a0a25a256a9e" (UID: "7b0d14f0-713d-45ec-814d-a0a25a256a9e"). InnerVolumeSpecName "kube-api-access-p56kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:07:12.971364 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.971325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3518251d-ddfc-4c6d-a17f-a7fed5c9418b-plugins-config-volume\") pod \"3518251d-ddfc-4c6d-a17f-a7fed5c9418b\" (UID: \"3518251d-ddfc-4c6d-a17f-a7fed5c9418b\") "
Sep 08 16:07:12.971535 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.971384    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9zv9\" (UniqueName: \"kubernetes.io/projected/3518251d-ddfc-4c6d-a17f-a7fed5c9418b-kube-api-access-m9zv9\") pod \"3518251d-ddfc-4c6d-a17f-a7fed5c9418b\" (UID: \"3518251d-ddfc-4c6d-a17f-a7fed5c9418b\") "
Sep 08 16:07:12.971721 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.971700    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0d14f0-713d-45ec-814d-a0a25a256a9e-metrics-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:07:12.971764 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.971734    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p56kd\" (UniqueName: \"kubernetes.io/projected/7b0d14f0-713d-45ec-814d-a0a25a256a9e-kube-api-access-p56kd\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:07:12.971764 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.971749    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0d14f0-713d-45ec-814d-a0a25a256a9e-openshift-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:07:12.971833 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.971764    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0d14f0-713d-45ec-814d-a0a25a256a9e-maas-api-tls\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:07:12.971833 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.971699    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3518251d-ddfc-4c6d-a17f-a7fed5c9418b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3518251d-ddfc-4c6d-a17f-a7fed5c9418b" (UID: "3518251d-ddfc-4c6d-a17f-a7fed5c9418b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:07:12.973466 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:12.973449    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3518251d-ddfc-4c6d-a17f-a7fed5c9418b-kube-api-access-m9zv9" (OuterVolumeSpecName: "kube-api-access-m9zv9") pod "3518251d-ddfc-4c6d-a17f-a7fed5c9418b" (UID: "3518251d-ddfc-4c6d-a17f-a7fed5c9418b"). InnerVolumeSpecName "kube-api-access-m9zv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:07:13.047776 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.047680    2581 generic.go:356] "Generic (PLEG): container finished" podID="3518251d-ddfc-4c6d-a17f-a7fed5c9418b" containerID="1ed682afbecd8b2f74642a1827a3307c858209ccb76d9fabf9c26ab6c39f87de" exitCode=0
Sep 08 16:07:13.047776 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.047723    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"
Sep 08 16:07:13.047776 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.047757    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8" event={"ID":"3518251d-ddfc-4c6d-a17f-a7fed5c9418b","Type":"ContainerDied","Data":"1ed682afbecd8b2f74642a1827a3307c858209ccb76d9fabf9c26ab6c39f87de"}
Sep 08 16:07:13.048049 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.047806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8" event={"ID":"3518251d-ddfc-4c6d-a17f-a7fed5c9418b","Type":"ContainerDied","Data":"8d3a357faa664d6621ba5fee6ea33a3b3317128c3cc119f1628b68907e81677a"}
Sep 08 16:07:13.048049 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.047834    2581 scope.go:117] "RemoveContainer" containerID="1ed682afbecd8b2f74642a1827a3307c858209ccb76d9fabf9c26ab6c39f87de"
Sep 08 16:07:13.049078 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.049058    2581 generic.go:356] "Generic (PLEG): container finished" podID="7b0d14f0-713d-45ec-814d-a0a25a256a9e" containerID="05733455e11e499da8eb7b4275345219c5e9e5cfea3b4162e09517ea35e22055" exitCode=0
Sep 08 16:07:13.049209 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.049098    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"
Sep 08 16:07:13.049209 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.049128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48" event={"ID":"7b0d14f0-713d-45ec-814d-a0a25a256a9e","Type":"ContainerDied","Data":"05733455e11e499da8eb7b4275345219c5e9e5cfea3b4162e09517ea35e22055"}
Sep 08 16:07:13.049209 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.049166    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48" event={"ID":"7b0d14f0-713d-45ec-814d-a0a25a256a9e","Type":"ContainerDied","Data":"d6cd50626ab31b62cd0f8fb4cb094e49628b9590ce0abc82a9e9d810b8debe32"}
Sep 08 16:07:13.057578 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.057482    2581 scope.go:117] "RemoveContainer" containerID="1ed682afbecd8b2f74642a1827a3307c858209ccb76d9fabf9c26ab6c39f87de"
Sep 08 16:07:13.057836 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:07:13.057815    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed682afbecd8b2f74642a1827a3307c858209ccb76d9fabf9c26ab6c39f87de\": container with ID starting with 1ed682afbecd8b2f74642a1827a3307c858209ccb76d9fabf9c26ab6c39f87de not found: ID does not exist" containerID="1ed682afbecd8b2f74642a1827a3307c858209ccb76d9fabf9c26ab6c39f87de"
Sep 08 16:07:13.057919 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.057844    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed682afbecd8b2f74642a1827a3307c858209ccb76d9fabf9c26ab6c39f87de"} err="failed to get container status \"1ed682afbecd8b2f74642a1827a3307c858209ccb76d9fabf9c26ab6c39f87de\": rpc error: code = NotFound desc = could not find container \"1ed682afbecd8b2f74642a1827a3307c858209ccb76d9fabf9c26ab6c39f87de\": container with ID starting with 1ed682afbecd8b2f74642a1827a3307c858209ccb76d9fabf9c26ab6c39f87de not found: ID does not exist"
Sep 08 16:07:13.057919 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.057859    2581 scope.go:117] "RemoveContainer" containerID="05733455e11e499da8eb7b4275345219c5e9e5cfea3b4162e09517ea35e22055"
Sep 08 16:07:13.066045 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.066030    2581 scope.go:117] "RemoveContainer" containerID="05733455e11e499da8eb7b4275345219c5e9e5cfea3b4162e09517ea35e22055"
Sep 08 16:07:13.066296 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:07:13.066279    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05733455e11e499da8eb7b4275345219c5e9e5cfea3b4162e09517ea35e22055\": container with ID starting with 05733455e11e499da8eb7b4275345219c5e9e5cfea3b4162e09517ea35e22055 not found: ID does not exist" containerID="05733455e11e499da8eb7b4275345219c5e9e5cfea3b4162e09517ea35e22055"
Sep 08 16:07:13.066343 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.066302    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05733455e11e499da8eb7b4275345219c5e9e5cfea3b4162e09517ea35e22055"} err="failed to get container status \"05733455e11e499da8eb7b4275345219c5e9e5cfea3b4162e09517ea35e22055\": rpc error: code = NotFound desc = could not find container \"05733455e11e499da8eb7b4275345219c5e9e5cfea3b4162e09517ea35e22055\": container with ID starting with 05733455e11e499da8eb7b4275345219c5e9e5cfea3b4162e09517ea35e22055 not found: ID does not exist"
Sep 08 16:07:13.072978 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.072884    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3518251d-ddfc-4c6d-a17f-a7fed5c9418b-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:07:13.072978 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.072909    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9zv9\" (UniqueName: \"kubernetes.io/projected/3518251d-ddfc-4c6d-a17f-a7fed5c9418b-kube-api-access-m9zv9\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:07:13.075263 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.075238    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"]
Sep 08 16:07:13.078870 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.078850    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5214de-5fc4848f76-h8g48"]
Sep 08 16:07:13.092756 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.092738    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"]
Sep 08 16:07:13.096851 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.096829    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5214de-565865cd6-nl2q8"]
Sep 08 16:07:13.901564 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.901533    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3518251d-ddfc-4c6d-a17f-a7fed5c9418b" path="/var/lib/kubelet/pods/3518251d-ddfc-4c6d-a17f-a7fed5c9418b/volumes"
Sep 08 16:07:13.901923 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:13.901909    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b0d14f0-713d-45ec-814d-a0a25a256a9e" path="/var/lib/kubelet/pods/7b0d14f0-713d-45ec-814d-a0a25a256a9e/volumes"
Sep 08 16:07:45.988718 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:45.988677    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"]
Sep 08 16:07:52.352997 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.352959    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"]
Sep 08 16:07:52.353560 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.353222    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86" podUID="8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4" containerName="payload-processing" containerID="cri-o://75aeba122fd4a8b703960b162a9ef5aed0d2eefd1b3c753a73073f75d127e7da" gracePeriod=30
Sep 08 16:07:52.357134 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.357110    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"]
Sep 08 16:07:52.358342 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.358238    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7" podUID="50d45a50-6e8d-456d-b7ec-28fe71254fc7" containerName="payload-pre-processing" containerID="cri-o://689ff002b6610534526ce6c30e5f0eda0b3792fe320e6960c7ca0b9c714dd9a3" gracePeriod=30
Sep 08 16:07:52.654904 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.654877    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"
Sep 08 16:07:52.657652 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.657633    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"
Sep 08 16:07:52.720768 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.720736    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99nc7\" (UniqueName: \"kubernetes.io/projected/50d45a50-6e8d-456d-b7ec-28fe71254fc7-kube-api-access-99nc7\") pod \"50d45a50-6e8d-456d-b7ec-28fe71254fc7\" (UID: \"50d45a50-6e8d-456d-b7ec-28fe71254fc7\") "
Sep 08 16:07:52.720930 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.720774    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b47pz\" (UniqueName: \"kubernetes.io/projected/8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4-kube-api-access-b47pz\") pod \"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4\" (UID: \"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4\") "
Sep 08 16:07:52.720930 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.720805    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d45a50-6e8d-456d-b7ec-28fe71254fc7-plugins-config-volume\") pod \"50d45a50-6e8d-456d-b7ec-28fe71254fc7\" (UID: \"50d45a50-6e8d-456d-b7ec-28fe71254fc7\") "
Sep 08 16:07:52.720930 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.720892    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4-plugins-config-volume\") pod \"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4\" (UID: \"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4\") "
Sep 08 16:07:52.721253 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.721229    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d45a50-6e8d-456d-b7ec-28fe71254fc7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50d45a50-6e8d-456d-b7ec-28fe71254fc7" (UID: "50d45a50-6e8d-456d-b7ec-28fe71254fc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:07:52.721338 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.721253    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4" (UID: "8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:07:52.722934 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.722913    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d45a50-6e8d-456d-b7ec-28fe71254fc7-kube-api-access-99nc7" (OuterVolumeSpecName: "kube-api-access-99nc7") pod "50d45a50-6e8d-456d-b7ec-28fe71254fc7" (UID: "50d45a50-6e8d-456d-b7ec-28fe71254fc7"). InnerVolumeSpecName "kube-api-access-99nc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:07:52.723014 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.722940    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4-kube-api-access-b47pz" (OuterVolumeSpecName: "kube-api-access-b47pz") pod "8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4" (UID: "8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4"). InnerVolumeSpecName "kube-api-access-b47pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:07:52.821885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.821851    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99nc7\" (UniqueName: \"kubernetes.io/projected/50d45a50-6e8d-456d-b7ec-28fe71254fc7-kube-api-access-99nc7\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:07:52.821885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.821884    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b47pz\" (UniqueName: \"kubernetes.io/projected/8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4-kube-api-access-b47pz\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:07:52.821885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.821894    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d45a50-6e8d-456d-b7ec-28fe71254fc7-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:07:52.822077 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:52.821903    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:07:53.253260 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.253223    2581 generic.go:356] "Generic (PLEG): container finished" podID="8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4" containerID="75aeba122fd4a8b703960b162a9ef5aed0d2eefd1b3c753a73073f75d127e7da" exitCode=0
Sep 08 16:07:53.253483 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.253270    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"
Sep 08 16:07:53.253483 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.253306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86" event={"ID":"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4","Type":"ContainerDied","Data":"75aeba122fd4a8b703960b162a9ef5aed0d2eefd1b3c753a73073f75d127e7da"}
Sep 08 16:07:53.253483 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.253371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86" event={"ID":"8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4","Type":"ContainerDied","Data":"825ad6855a40c69ba01e200764cfb94e25ac26ae123e6251e1717511fb74fafe"}
Sep 08 16:07:53.253483 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.253422    2581 scope.go:117] "RemoveContainer" containerID="75aeba122fd4a8b703960b162a9ef5aed0d2eefd1b3c753a73073f75d127e7da"
Sep 08 16:07:53.254572 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.254539    2581 generic.go:356] "Generic (PLEG): container finished" podID="50d45a50-6e8d-456d-b7ec-28fe71254fc7" containerID="689ff002b6610534526ce6c30e5f0eda0b3792fe320e6960c7ca0b9c714dd9a3" exitCode=0
Sep 08 16:07:53.254648 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.254586    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"
Sep 08 16:07:53.254648 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.254597    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7" event={"ID":"50d45a50-6e8d-456d-b7ec-28fe71254fc7","Type":"ContainerDied","Data":"689ff002b6610534526ce6c30e5f0eda0b3792fe320e6960c7ca0b9c714dd9a3"}
Sep 08 16:07:53.254648 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.254627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7" event={"ID":"50d45a50-6e8d-456d-b7ec-28fe71254fc7","Type":"ContainerDied","Data":"bdc0112596f04f0350a95a88f5ed4bcf5560691fe12f36d3a4c94e1c54cf344f"}
Sep 08 16:07:53.263469 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.263454    2581 scope.go:117] "RemoveContainer" containerID="75aeba122fd4a8b703960b162a9ef5aed0d2eefd1b3c753a73073f75d127e7da"
Sep 08 16:07:53.263772 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:07:53.263753    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75aeba122fd4a8b703960b162a9ef5aed0d2eefd1b3c753a73073f75d127e7da\": container with ID starting with 75aeba122fd4a8b703960b162a9ef5aed0d2eefd1b3c753a73073f75d127e7da not found: ID does not exist" containerID="75aeba122fd4a8b703960b162a9ef5aed0d2eefd1b3c753a73073f75d127e7da"
Sep 08 16:07:53.263826 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.263780    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75aeba122fd4a8b703960b162a9ef5aed0d2eefd1b3c753a73073f75d127e7da"} err="failed to get container status \"75aeba122fd4a8b703960b162a9ef5aed0d2eefd1b3c753a73073f75d127e7da\": rpc error: code = NotFound desc = could not find container \"75aeba122fd4a8b703960b162a9ef5aed0d2eefd1b3c753a73073f75d127e7da\": container with ID starting with 75aeba122fd4a8b703960b162a9ef5aed0d2eefd1b3c753a73073f75d127e7da not found: ID does not exist"
Sep 08 16:07:53.263826 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.263800    2581 scope.go:117] "RemoveContainer" containerID="689ff002b6610534526ce6c30e5f0eda0b3792fe320e6960c7ca0b9c714dd9a3"
Sep 08 16:07:53.272299 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.272282    2581 scope.go:117] "RemoveContainer" containerID="689ff002b6610534526ce6c30e5f0eda0b3792fe320e6960c7ca0b9c714dd9a3"
Sep 08 16:07:53.272568 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:07:53.272546    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"689ff002b6610534526ce6c30e5f0eda0b3792fe320e6960c7ca0b9c714dd9a3\": container with ID starting with 689ff002b6610534526ce6c30e5f0eda0b3792fe320e6960c7ca0b9c714dd9a3 not found: ID does not exist" containerID="689ff002b6610534526ce6c30e5f0eda0b3792fe320e6960c7ca0b9c714dd9a3"
Sep 08 16:07:53.272617 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.272576    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"689ff002b6610534526ce6c30e5f0eda0b3792fe320e6960c7ca0b9c714dd9a3"} err="failed to get container status \"689ff002b6610534526ce6c30e5f0eda0b3792fe320e6960c7ca0b9c714dd9a3\": rpc error: code = NotFound desc = could not find container \"689ff002b6610534526ce6c30e5f0eda0b3792fe320e6960c7ca0b9c714dd9a3\": container with ID starting with 689ff002b6610534526ce6c30e5f0eda0b3792fe320e6960c7ca0b9c714dd9a3 not found: ID does not exist"
Sep 08 16:07:53.278136 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.278113    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"]
Sep 08 16:07:53.280175 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.280157    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-13d3a5-65b57c7c6c-lqd86"]
Sep 08 16:07:53.288632 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.288613    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"]
Sep 08 16:07:53.296285 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.296259    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-13d3a5-59bbcd58c8-fx2t7"]
Sep 08 16:07:53.901988 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.901944    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50d45a50-6e8d-456d-b7ec-28fe71254fc7" path="/var/lib/kubelet/pods/50d45a50-6e8d-456d-b7ec-28fe71254fc7/volumes"
Sep 08 16:07:53.902552 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:07:53.902531    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4" path="/var/lib/kubelet/pods/8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4/volumes"
Sep 08 16:08:01.008815 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.008753    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww" podUID="6ff9f21a-cd71-44ab-b5af-06384696284f" containerName="main" containerID="cri-o://2172d061704b9dc325c5905d0fe8dfcdc166eab81458747da57eacdb4eaf904d" gracePeriod=45
Sep 08 16:08:01.271275 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.271223    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:08:01.298191 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.298147    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ff9f21a-cd71-44ab-b5af-06384696284f" containerID="2172d061704b9dc325c5905d0fe8dfcdc166eab81458747da57eacdb4eaf904d" exitCode=0
Sep 08 16:08:01.298191 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.298179    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"
Sep 08 16:08:01.298373 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.298226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww" event={"ID":"6ff9f21a-cd71-44ab-b5af-06384696284f","Type":"ContainerDied","Data":"2172d061704b9dc325c5905d0fe8dfcdc166eab81458747da57eacdb4eaf904d"}
Sep 08 16:08:01.298373 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.298286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww" event={"ID":"6ff9f21a-cd71-44ab-b5af-06384696284f","Type":"ContainerDied","Data":"80053a875c372e8627c8a34d1fc9ed65ceb353001fc26a3d2ffc025181fe969e"}
Sep 08 16:08:01.298373 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.298312    2581 scope.go:117] "RemoveContainer" containerID="2172d061704b9dc325c5905d0fe8dfcdc166eab81458747da57eacdb4eaf904d"
Sep 08 16:08:01.308540 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.308516    2581 scope.go:117] "RemoveContainer" containerID="2172d061704b9dc325c5905d0fe8dfcdc166eab81458747da57eacdb4eaf904d"
Sep 08 16:08:01.308798 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:08:01.308783    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2172d061704b9dc325c5905d0fe8dfcdc166eab81458747da57eacdb4eaf904d\": container with ID starting with 2172d061704b9dc325c5905d0fe8dfcdc166eab81458747da57eacdb4eaf904d not found: ID does not exist" containerID="2172d061704b9dc325c5905d0fe8dfcdc166eab81458747da57eacdb4eaf904d"
Sep 08 16:08:01.308873 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.308805    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2172d061704b9dc325c5905d0fe8dfcdc166eab81458747da57eacdb4eaf904d"} err="failed to get container status \"2172d061704b9dc325c5905d0fe8dfcdc166eab81458747da57eacdb4eaf904d\": rpc error: code = NotFound desc = could not find container \"2172d061704b9dc325c5905d0fe8dfcdc166eab81458747da57eacdb4eaf904d\": container with ID starting with 2172d061704b9dc325c5905d0fe8dfcdc166eab81458747da57eacdb4eaf904d not found: ID does not exist"
Sep 08 16:08:01.396294 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.396270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh769\" (UniqueName: \"kubernetes.io/projected/6ff9f21a-cd71-44ab-b5af-06384696284f-kube-api-access-fh769\") pod \"6ff9f21a-cd71-44ab-b5af-06384696284f\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") "
Sep 08 16:08:01.396418 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.396303    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-model-cache\") pod \"6ff9f21a-cd71-44ab-b5af-06384696284f\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") "
Sep 08 16:08:01.396418 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.396352    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-home\") pod \"6ff9f21a-cd71-44ab-b5af-06384696284f\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") "
Sep 08 16:08:01.396418 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.396375    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-dshm\") pod \"6ff9f21a-cd71-44ab-b5af-06384696284f\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") "
Sep 08 16:08:01.396565 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.396457    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff9f21a-cd71-44ab-b5af-06384696284f-tls-certs\") pod \"6ff9f21a-cd71-44ab-b5af-06384696284f\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") "
Sep 08 16:08:01.396565 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.396489    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-tmp-dir\") pod \"6ff9f21a-cd71-44ab-b5af-06384696284f\" (UID: \"6ff9f21a-cd71-44ab-b5af-06384696284f\") "
Sep 08 16:08:01.396645 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.396626    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-home" (OuterVolumeSpecName: "home") pod "6ff9f21a-cd71-44ab-b5af-06384696284f" (UID: "6ff9f21a-cd71-44ab-b5af-06384696284f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:08:01.396722 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.396634    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ff9f21a-cd71-44ab-b5af-06384696284f" (UID: "6ff9f21a-cd71-44ab-b5af-06384696284f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:08:01.396927 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.396863    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ff9f21a-cd71-44ab-b5af-06384696284f" (UID: "6ff9f21a-cd71-44ab-b5af-06384696284f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:08:01.396927 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.396867    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-model-cache\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:08:01.396927 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.396908    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-home\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:08:01.398780 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.398754    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-dshm" (OuterVolumeSpecName: "dshm") pod "6ff9f21a-cd71-44ab-b5af-06384696284f" (UID: "6ff9f21a-cd71-44ab-b5af-06384696284f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:08:01.398864 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.398805    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff9f21a-cd71-44ab-b5af-06384696284f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ff9f21a-cd71-44ab-b5af-06384696284f" (UID: "6ff9f21a-cd71-44ab-b5af-06384696284f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:08:01.398907 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.398875    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff9f21a-cd71-44ab-b5af-06384696284f-kube-api-access-fh769" (OuterVolumeSpecName: "kube-api-access-fh769") pod "6ff9f21a-cd71-44ab-b5af-06384696284f" (UID: "6ff9f21a-cd71-44ab-b5af-06384696284f"). InnerVolumeSpecName "kube-api-access-fh769". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:08:01.497821 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.497792    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-tmp-dir\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:08:01.497821 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.497819    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh769\" (UniqueName: \"kubernetes.io/projected/6ff9f21a-cd71-44ab-b5af-06384696284f-kube-api-access-fh769\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:08:01.497951 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.497833    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff9f21a-cd71-44ab-b5af-06384696284f-dshm\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:08:01.497951 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.497846    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff9f21a-cd71-44ab-b5af-06384696284f-tls-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:08:01.619016 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.618994    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"]
Sep 08 16:08:01.622525 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.622504    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a09aa5/rate-test-model-a09aa5-kserve-7bd44b5799-qtzww"]
Sep 08 16:08:01.901371 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:01.901343    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ff9f21a-cd71-44ab-b5af-06384696284f" path="/var/lib/kubelet/pods/6ff9f21a-cd71-44ab-b5af-06384696284f/volumes"
Sep 08 16:08:10.183285 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.183245    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"]
Sep 08 16:08:10.188102 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.188075    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"]
Sep 08 16:08:10.188347 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.188316    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn" podUID="5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4" containerName="payload-pre-processing" containerID="cri-o://8c0a55d237b554f3465162afbfcc8cc9407f42ae0b1312ea9eba5fc261326e4a" gracePeriod=30
Sep 08 16:08:10.344517 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.344484    2581 generic.go:356] "Generic (PLEG): container finished" podID="5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4" containerID="8c0a55d237b554f3465162afbfcc8cc9407f42ae0b1312ea9eba5fc261326e4a" exitCode=0
Sep 08 16:08:10.344691 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.344563    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn" event={"ID":"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4","Type":"ContainerDied","Data":"8c0a55d237b554f3465162afbfcc8cc9407f42ae0b1312ea9eba5fc261326e4a"}
Sep 08 16:08:10.344737 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.344724    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr" podUID="40e3923a-caa4-46b3-8f0b-1abaf2750ee4" containerName="payload-processing" containerID="cri-o://3afa7b2237e0608771f4e80c5ab62bebc28a21c74070f43ab1b902e5a0dceaee" gracePeriod=30
Sep 08 16:08:10.472109 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.472081    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"
Sep 08 16:08:10.578294 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.578268    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4-plugins-config-volume\") pod \"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4\" (UID: \"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4\") "
Sep 08 16:08:10.578473 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.578347    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht772\" (UniqueName: \"kubernetes.io/projected/5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4-kube-api-access-ht772\") pod \"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4\" (UID: \"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4\") "
Sep 08 16:08:10.578759 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.578724    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4" (UID: "5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:08:10.580929 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.580906    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4-kube-api-access-ht772" (OuterVolumeSpecName: "kube-api-access-ht772") pod "5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4" (UID: "5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4"). InnerVolumeSpecName "kube-api-access-ht772". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:08:10.601741 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.601722    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"
Sep 08 16:08:10.678993 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.678970    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fbz5\" (UniqueName: \"kubernetes.io/projected/40e3923a-caa4-46b3-8f0b-1abaf2750ee4-kube-api-access-4fbz5\") pod \"40e3923a-caa4-46b3-8f0b-1abaf2750ee4\" (UID: \"40e3923a-caa4-46b3-8f0b-1abaf2750ee4\") "
Sep 08 16:08:10.679108 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.679026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e3923a-caa4-46b3-8f0b-1abaf2750ee4-plugins-config-volume\") pod \"40e3923a-caa4-46b3-8f0b-1abaf2750ee4\" (UID: \"40e3923a-caa4-46b3-8f0b-1abaf2750ee4\") "
Sep 08 16:08:10.679285 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.679269    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht772\" (UniqueName: \"kubernetes.io/projected/5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4-kube-api-access-ht772\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:08:10.679331 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.679295    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:08:10.679414 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.679380    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e3923a-caa4-46b3-8f0b-1abaf2750ee4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40e3923a-caa4-46b3-8f0b-1abaf2750ee4" (UID: "40e3923a-caa4-46b3-8f0b-1abaf2750ee4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:08:10.681027 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.681007    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e3923a-caa4-46b3-8f0b-1abaf2750ee4-kube-api-access-4fbz5" (OuterVolumeSpecName: "kube-api-access-4fbz5") pod "40e3923a-caa4-46b3-8f0b-1abaf2750ee4" (UID: "40e3923a-caa4-46b3-8f0b-1abaf2750ee4"). InnerVolumeSpecName "kube-api-access-4fbz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:08:10.780013 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.779946    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fbz5\" (UniqueName: \"kubernetes.io/projected/40e3923a-caa4-46b3-8f0b-1abaf2750ee4-kube-api-access-4fbz5\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:08:10.780013 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:10.779969    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e3923a-caa4-46b3-8f0b-1abaf2750ee4-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:08:11.350858 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.350830    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"
Sep 08 16:08:11.351305 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.350863    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn" event={"ID":"5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4","Type":"ContainerDied","Data":"5d673ac781b34a7945b1c2fc3913dc1e30a13aa79184357442368f824edfe6c6"}
Sep 08 16:08:11.351305 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.350932    2581 scope.go:117] "RemoveContainer" containerID="8c0a55d237b554f3465162afbfcc8cc9407f42ae0b1312ea9eba5fc261326e4a"
Sep 08 16:08:11.352212 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.352184    2581 generic.go:356] "Generic (PLEG): container finished" podID="40e3923a-caa4-46b3-8f0b-1abaf2750ee4" containerID="3afa7b2237e0608771f4e80c5ab62bebc28a21c74070f43ab1b902e5a0dceaee" exitCode=0
Sep 08 16:08:11.352286 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.352233    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"
Sep 08 16:08:11.352335 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.352275    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr" event={"ID":"40e3923a-caa4-46b3-8f0b-1abaf2750ee4","Type":"ContainerDied","Data":"3afa7b2237e0608771f4e80c5ab62bebc28a21c74070f43ab1b902e5a0dceaee"}
Sep 08 16:08:11.352335 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.352324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr" event={"ID":"40e3923a-caa4-46b3-8f0b-1abaf2750ee4","Type":"ContainerDied","Data":"81001e04a7af3b7d0bf46110315cba6e1d678ff6c37343676e34a2530548a3b9"}
Sep 08 16:08:11.361137 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.361117    2581 scope.go:117] "RemoveContainer" containerID="3afa7b2237e0608771f4e80c5ab62bebc28a21c74070f43ab1b902e5a0dceaee"
Sep 08 16:08:11.369238 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.369222    2581 scope.go:117] "RemoveContainer" containerID="3afa7b2237e0608771f4e80c5ab62bebc28a21c74070f43ab1b902e5a0dceaee"
Sep 08 16:08:11.369515 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:08:11.369498    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afa7b2237e0608771f4e80c5ab62bebc28a21c74070f43ab1b902e5a0dceaee\": container with ID starting with 3afa7b2237e0608771f4e80c5ab62bebc28a21c74070f43ab1b902e5a0dceaee not found: ID does not exist" containerID="3afa7b2237e0608771f4e80c5ab62bebc28a21c74070f43ab1b902e5a0dceaee"
Sep 08 16:08:11.369572 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.369524    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afa7b2237e0608771f4e80c5ab62bebc28a21c74070f43ab1b902e5a0dceaee"} err="failed to get container status \"3afa7b2237e0608771f4e80c5ab62bebc28a21c74070f43ab1b902e5a0dceaee\": rpc error: code = NotFound desc = could not find container \"3afa7b2237e0608771f4e80c5ab62bebc28a21c74070f43ab1b902e5a0dceaee\": container with ID starting with 3afa7b2237e0608771f4e80c5ab62bebc28a21c74070f43ab1b902e5a0dceaee not found: ID does not exist"
Sep 08 16:08:11.404889 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.404863    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"]
Sep 08 16:08:11.409782 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.409764    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e69292-58685cf858-5jdpn"]
Sep 08 16:08:11.421242 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.421218    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"]
Sep 08 16:08:11.425482 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.425459    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e69292-86895ffcf7-lntvr"]
Sep 08 16:08:11.900783 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.900749    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40e3923a-caa4-46b3-8f0b-1abaf2750ee4" path="/var/lib/kubelet/pods/40e3923a-caa4-46b3-8f0b-1abaf2750ee4/volumes"
Sep 08 16:08:11.901123 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:11.901107    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4" path="/var/lib/kubelet/pods/5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4/volumes"
Sep 08 16:08:20.853267 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853229    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"]
Sep 08 16:08:20.853699 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853665    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d45a50-6e8d-456d-b7ec-28fe71254fc7" containerName="payload-pre-processing"
Sep 08 16:08:20.853699 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853679    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d45a50-6e8d-456d-b7ec-28fe71254fc7" containerName="payload-pre-processing"
Sep 08 16:08:20.853699 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853689    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3518251d-ddfc-4c6d-a17f-a7fed5c9418b" containerName="payload-pre-processing"
Sep 08 16:08:20.853699 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853694    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3518251d-ddfc-4c6d-a17f-a7fed5c9418b" containerName="payload-pre-processing"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853705    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4" containerName="payload-processing"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853711    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4" containerName="payload-processing"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853718    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4" containerName="payload-pre-processing"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853723    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4" containerName="payload-pre-processing"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853732    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0d14f0-713d-45ec-814d-a0a25a256a9e" containerName="maas-api"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853737    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0d14f0-713d-45ec-814d-a0a25a256a9e" containerName="maas-api"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853744    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ff9f21a-cd71-44ab-b5af-06384696284f" containerName="main"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853749    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ff9f21a-cd71-44ab-b5af-06384696284f" containerName="main"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853761    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e3923a-caa4-46b3-8f0b-1abaf2750ee4" containerName="payload-processing"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853766    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e3923a-caa4-46b3-8f0b-1abaf2750ee4" containerName="payload-processing"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853821    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf9e265-2f80-4e3e-bea3-baf64e6a7eb4" containerName="payload-pre-processing"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853829    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ff9f21a-cd71-44ab-b5af-06384696284f" containerName="main"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853835    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d45a50-6e8d-456d-b7ec-28fe71254fc7" containerName="payload-pre-processing"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853841    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f0fabcd-4cbd-4917-9c4b-e674ddfd3cb4" containerName="payload-processing"
Sep 08 16:08:20.853843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853849    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3518251d-ddfc-4c6d-a17f-a7fed5c9418b" containerName="payload-pre-processing"
Sep 08 16:08:20.854353 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853855    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b0d14f0-713d-45ec-814d-a0a25a256a9e" containerName="maas-api"
Sep 08 16:08:20.854353 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.853861    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e3923a-caa4-46b3-8f0b-1abaf2750ee4" containerName="payload-processing"
Sep 08 16:08:20.858570 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.858552    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:20.861615 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.861594    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7a7297-serving-cert\""
Sep 08 16:08:20.871737 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.871712    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"]
Sep 08 16:08:20.965272 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.965239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2482a9f4-045d-48ee-8b16-28f42f4ebc13-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:20.965427 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.965277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:20.965427 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.965310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl75t\" (UniqueName: \"kubernetes.io/projected/2482a9f4-045d-48ee-8b16-28f42f4ebc13-kube-api-access-zl75t\") pod \"maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:20.965427 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:20.965387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-metrics-certs\") pod \"maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:21.065883 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.065846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-metrics-certs\") pod \"maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:21.066056 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.065939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2482a9f4-045d-48ee-8b16-28f42f4ebc13-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:21.066056 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.065958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:21.066056 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.065984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl75t\" (UniqueName: \"kubernetes.io/projected/2482a9f4-045d-48ee-8b16-28f42f4ebc13-kube-api-access-zl75t\") pod \"maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:21.066184 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:08:21.066108    2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-serving-cert: secret "maas-api-e2e-ipp-b-7a7297-serving-cert" not found
Sep 08 16:08:21.066222 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:08:21.066199    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-maas-api-tls podName:2482a9f4-045d-48ee-8b16-28f42f4ebc13 nodeName:}" failed. No retries permitted until 2026-09-08 16:08:21.566177609 +0000 UTC m=+1448.189869917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-maas-api-tls") pod "maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt" (UID: "2482a9f4-045d-48ee-8b16-28f42f4ebc13") : secret "maas-api-e2e-ipp-b-7a7297-serving-cert" not found
Sep 08 16:08:21.066752 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.066727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2482a9f4-045d-48ee-8b16-28f42f4ebc13-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:21.068379 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.068356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-metrics-certs\") pod \"maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:21.079397 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.079370    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl75t\" (UniqueName: \"kubernetes.io/projected/2482a9f4-045d-48ee-8b16-28f42f4ebc13-kube-api-access-zl75t\") pod \"maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:21.572252 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.572221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:21.574685 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.574662    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:21.641403 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.641358    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"]
Sep 08 16:08:21.645121 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.645099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"
Sep 08 16:08:21.647356 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.647331    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7a7297\""
Sep 08 16:08:21.647504 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.647430    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7a7297-dockercfg-2mr7n\""
Sep 08 16:08:21.657090 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.657063    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"]
Sep 08 16:08:21.692717 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.692680    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"]
Sep 08 16:08:21.696224 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.696199    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"
Sep 08 16:08:21.710676 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.710654    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"]
Sep 08 16:08:21.771586 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.771550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:21.774897 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.774564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76v5w\" (UniqueName: \"kubernetes.io/projected/ccce6db9-8fad-43a8-97fc-8b731216feaf-kube-api-access-76v5w\") pod \"payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs\" (UID: \"ccce6db9-8fad-43a8-97fc-8b731216feaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"
Sep 08 16:08:21.774897 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.774648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccce6db9-8fad-43a8-97fc-8b731216feaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs\" (UID: \"ccce6db9-8fad-43a8-97fc-8b731216feaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"
Sep 08 16:08:21.875123 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.875090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccce6db9-8fad-43a8-97fc-8b731216feaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs\" (UID: \"ccce6db9-8fad-43a8-97fc-8b731216feaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"
Sep 08 16:08:21.875567 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.875252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76v5w\" (UniqueName: \"kubernetes.io/projected/ccce6db9-8fad-43a8-97fc-8b731216feaf-kube-api-access-76v5w\") pod \"payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs\" (UID: \"ccce6db9-8fad-43a8-97fc-8b731216feaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"
Sep 08 16:08:21.875567 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.875298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd554\" (UniqueName: \"kubernetes.io/projected/d672182b-42d5-4333-9830-c853a1e798c5-kube-api-access-sd554\") pod \"payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l\" (UID: \"d672182b-42d5-4333-9830-c853a1e798c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"
Sep 08 16:08:21.875567 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.875365    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d672182b-42d5-4333-9830-c853a1e798c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l\" (UID: \"d672182b-42d5-4333-9830-c853a1e798c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"
Sep 08 16:08:21.875849 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.875827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccce6db9-8fad-43a8-97fc-8b731216feaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs\" (UID: \"ccce6db9-8fad-43a8-97fc-8b731216feaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"
Sep 08 16:08:21.883714 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.883695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76v5w\" (UniqueName: \"kubernetes.io/projected/ccce6db9-8fad-43a8-97fc-8b731216feaf-kube-api-access-76v5w\") pod \"payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs\" (UID: \"ccce6db9-8fad-43a8-97fc-8b731216feaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"
Sep 08 16:08:21.957084 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.957059    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"
Sep 08 16:08:21.976497 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.976465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd554\" (UniqueName: \"kubernetes.io/projected/d672182b-42d5-4333-9830-c853a1e798c5-kube-api-access-sd554\") pod \"payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l\" (UID: \"d672182b-42d5-4333-9830-c853a1e798c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"
Sep 08 16:08:21.976631 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.976538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d672182b-42d5-4333-9830-c853a1e798c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l\" (UID: \"d672182b-42d5-4333-9830-c853a1e798c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"
Sep 08 16:08:21.977470 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.977446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d672182b-42d5-4333-9830-c853a1e798c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l\" (UID: \"d672182b-42d5-4333-9830-c853a1e798c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"
Sep 08 16:08:21.985991 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:21.985963    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd554\" (UniqueName: \"kubernetes.io/projected/d672182b-42d5-4333-9830-c853a1e798c5-kube-api-access-sd554\") pod \"payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l\" (UID: \"d672182b-42d5-4333-9830-c853a1e798c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"
Sep 08 16:08:22.006880 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.006851    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"
Sep 08 16:08:22.047669 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.047640    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"]
Sep 08 16:08:22.054743 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.054721    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:08:22.230046 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.229946    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"]
Sep 08 16:08:22.232755 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:08:22.232723    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccce6db9_8fad_43a8_97fc_8b731216feaf.slice/crio-24fda8c15462dc51f2dcdc3c73ba248470783a898de16b11bc6895bdd32ea163 WatchSource:0}: Error finding container 24fda8c15462dc51f2dcdc3c73ba248470783a898de16b11bc6895bdd32ea163: Status 404 returned error can't find the container with id 24fda8c15462dc51f2dcdc3c73ba248470783a898de16b11bc6895bdd32ea163
Sep 08 16:08:22.292308 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.292287    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"]
Sep 08 16:08:22.295345 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:08:22.295319    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd672182b_42d5_4333_9830_c853a1e798c5.slice/crio-340e9f9617d909002f73ac1ab83d1a6e30d576b1cf6e24e3d70951af5ddd4f3b WatchSource:0}: Error finding container 340e9f9617d909002f73ac1ab83d1a6e30d576b1cf6e24e3d70951af5ddd4f3b: Status 404 returned error can't find the container with id 340e9f9617d909002f73ac1ab83d1a6e30d576b1cf6e24e3d70951af5ddd4f3b
Sep 08 16:08:22.418380 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.418338    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt" event={"ID":"2482a9f4-045d-48ee-8b16-28f42f4ebc13","Type":"ContainerStarted","Data":"bb84e9f3661026698916c0659c65a793212d52ea99b5689572742d8d6d847264"}
Sep 08 16:08:22.420125 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.420095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l" event={"ID":"d672182b-42d5-4333-9830-c853a1e798c5","Type":"ContainerStarted","Data":"3aa10240ffda3f3c2b4f099f254dd0e9267da51e97f3f83ca6742e7198429b1e"}
Sep 08 16:08:22.420277 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.420147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l" event={"ID":"d672182b-42d5-4333-9830-c853a1e798c5","Type":"ContainerStarted","Data":"340e9f9617d909002f73ac1ab83d1a6e30d576b1cf6e24e3d70951af5ddd4f3b"}
Sep 08 16:08:22.420336 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.420296    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"
Sep 08 16:08:22.421954 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.421926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs" event={"ID":"ccce6db9-8fad-43a8-97fc-8b731216feaf","Type":"ContainerStarted","Data":"d1ae863c4dcf209d08b0bbacd60432fb067d5ef38feff6fbe408af732191a03b"}
Sep 08 16:08:22.422109 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.421977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs" event={"ID":"ccce6db9-8fad-43a8-97fc-8b731216feaf","Type":"ContainerStarted","Data":"24fda8c15462dc51f2dcdc3c73ba248470783a898de16b11bc6895bdd32ea163"}
Sep 08 16:08:22.422109 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.422013    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"
Sep 08 16:08:22.439335 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.439288    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l" podStartSLOduration=1.439276029 podStartE2EDuration="1.439276029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:08:22.437572713 +0000 UTC m=+1449.061265037" watchObservedRunningTime="2026-09-08 16:08:22.439276029 +0000 UTC m=+1449.062968396"
Sep 08 16:08:22.455851 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:22.455811    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs" podStartSLOduration=1.4557971410000001 podStartE2EDuration="1.455797141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:08:22.454506015 +0000 UTC m=+1449.078198339" watchObservedRunningTime="2026-09-08 16:08:22.455797141 +0000 UTC m=+1449.079489467"
Sep 08 16:08:23.430466 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:23.430425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt" event={"ID":"2482a9f4-045d-48ee-8b16-28f42f4ebc13","Type":"ContainerStarted","Data":"5e39d0066666f030b65d993effcf94c4f3c2bd47cd5a0df4b1f63fccf46067de"}
Sep 08 16:08:23.430965 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:23.430761    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:23.450486 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:23.450440    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt" podStartSLOduration=3.045509807 podStartE2EDuration="3.450425064s" totalImagesPullingTime="404.915257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:08:23.449007878 +0000 UTC m=+1450.072700203" watchObservedRunningTime="2026-09-08 16:08:23.450425064 +0000 UTC m=+1450.074117444"
Sep 08 16:08:29.441196 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:29.441159    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:08:33.433101 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:33.433066    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"
Sep 08 16:08:33.433506 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:33.433139    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"
Sep 08 16:08:49.870301 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:08:49.870254    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l_d672182b-42d5-4333-9830-c853a1e798c5/payload-processing/0.log"
Sep 08 16:09:13.940970 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:13.940940    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 16:09:13.941510 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:13.940983    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 16:09:30.884049 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:30.884014    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l_d672182b-42d5-4333-9830-c853a1e798c5/payload-processing/0.log"
Sep 08 16:09:31.448326 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.448291    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"]
Sep 08 16:09:31.452073 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.452053    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.454411 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.454376    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-a2951a-openshift-default-dockercfg-d2vjg\""
Sep 08 16:09:31.465160 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.465135    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"]
Sep 08 16:09:31.601499 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.601468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-envoy\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.601665 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.601513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-data\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.601665 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.601583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-workload-certs\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.601755 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.601662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d40d8bd-69bd-482a-91c6-098992fe5324-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.601755 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.601708    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-podinfo\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.601755 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.601731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqhpz\" (UniqueName: \"kubernetes.io/projected/3d40d8bd-69bd-482a-91c6-098992fe5324-kube-api-access-pqhpz\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.601904 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.601810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-workload-socket\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.601904 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.601840    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-token\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.601904 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.601868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-credential-socket\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.703260 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.703167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-podinfo\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.703260 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.703212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqhpz\" (UniqueName: \"kubernetes.io/projected/3d40d8bd-69bd-482a-91c6-098992fe5324-kube-api-access-pqhpz\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.703528 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.703342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-workload-socket\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.703528 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.703418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-token\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.703528 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.703451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-credential-socket\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.703691 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.703548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-envoy\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.703691 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.703583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-data\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.703691 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.703635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-workload-certs\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.703860 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.703696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d40d8bd-69bd-482a-91c6-098992fe5324-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.704100 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.704036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-credential-socket\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.704100 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.704057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-workload-socket\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.704265 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.704115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-workload-certs\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.704265 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.704208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-data\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.704528 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.704508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d40d8bd-69bd-482a-91c6-098992fe5324-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.705876 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.705857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-envoy\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.705928 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.705859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-podinfo\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.712131 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.712110    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqhpz\" (UniqueName: \"kubernetes.io/projected/3d40d8bd-69bd-482a-91c6-098992fe5324-kube-api-access-pqhpz\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.712518 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.712502    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-token\") pod \"e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") " pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:31.762930 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:31.762898    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:32.044423 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:32.044369    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"]
Sep 08 16:09:32.046301 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:09:32.046273    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d40d8bd_69bd_482a_91c6_098992fe5324.slice/crio-0cefedb88da84693b32568aa5ec7c597857cf033b5daa678de71f1d7722331ca WatchSource:0}: Error finding container 0cefedb88da84693b32568aa5ec7c597857cf033b5daa678de71f1d7722331ca: Status 404 returned error can't find the container with id 0cefedb88da84693b32568aa5ec7c597857cf033b5daa678de71f1d7722331ca
Sep 08 16:09:32.048791 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:32.048748    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:09:32.048878 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:32.048851    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:09:32.048918 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:32.048900    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:09:32.781918 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:32.781871    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" event={"ID":"3d40d8bd-69bd-482a-91c6-098992fe5324","Type":"ContainerStarted","Data":"7c004e47e61dad21a5561e6cbe8d1366e4ccbb38de490aa30f460becb0500eee"}
Sep 08 16:09:32.782120 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:32.781935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" event={"ID":"3d40d8bd-69bd-482a-91c6-098992fe5324","Type":"ContainerStarted","Data":"0cefedb88da84693b32568aa5ec7c597857cf033b5daa678de71f1d7722331ca"}
Sep 08 16:09:32.810202 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:32.810146    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" podStartSLOduration=1.810128727 podStartE2EDuration="1.810128727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:09:32.807813794 +0000 UTC m=+1519.431506131" watchObservedRunningTime="2026-09-08 16:09:32.810128727 +0000 UTC m=+1519.433821050"
Sep 08 16:09:33.764019 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:33.763977    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:33.770132 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:33.770099    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:33.787418 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:33.787367    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:33.788184 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:33.788162    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:36.700912 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:36.700837    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"]
Sep 08 16:09:36.704710 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:36.704693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"
Sep 08 16:09:36.709252 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:36.709229    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-a2951a-dockercfg-74gzf\""
Sep 08 16:09:36.710146 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:36.710125    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-a2951a\""
Sep 08 16:09:36.722443 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:36.722420    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"]
Sep 08 16:09:36.849473 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:36.849441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72zxd\" (UniqueName: \"kubernetes.io/projected/b4cd44f9-5de2-40a8-9909-c988977619a8-kube-api-access-72zxd\") pod \"payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f\" (UID: \"b4cd44f9-5de2-40a8-9909-c988977619a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"
Sep 08 16:09:36.849646 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:36.849503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4cd44f9-5de2-40a8-9909-c988977619a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f\" (UID: \"b4cd44f9-5de2-40a8-9909-c988977619a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"
Sep 08 16:09:36.950508 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:36.950479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72zxd\" (UniqueName: \"kubernetes.io/projected/b4cd44f9-5de2-40a8-9909-c988977619a8-kube-api-access-72zxd\") pod \"payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f\" (UID: \"b4cd44f9-5de2-40a8-9909-c988977619a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"
Sep 08 16:09:36.950654 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:36.950528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4cd44f9-5de2-40a8-9909-c988977619a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f\" (UID: \"b4cd44f9-5de2-40a8-9909-c988977619a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"
Sep 08 16:09:36.951172 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:36.951129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4cd44f9-5de2-40a8-9909-c988977619a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f\" (UID: \"b4cd44f9-5de2-40a8-9909-c988977619a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"
Sep 08 16:09:36.958885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:36.958863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72zxd\" (UniqueName: \"kubernetes.io/projected/b4cd44f9-5de2-40a8-9909-c988977619a8-kube-api-access-72zxd\") pod \"payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f\" (UID: \"b4cd44f9-5de2-40a8-9909-c988977619a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"
Sep 08 16:09:37.015795 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:37.015765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"
Sep 08 16:09:37.286742 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:37.286717    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"]
Sep 08 16:09:37.289366 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:09:37.289336    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cd44f9_5de2_40a8_9909_c988977619a8.slice/crio-929d2e911125d9145cc0a8ea2b5dcb00a07175d1e52f3266c0d712be7bbe1d63 WatchSource:0}: Error finding container 929d2e911125d9145cc0a8ea2b5dcb00a07175d1e52f3266c0d712be7bbe1d63: Status 404 returned error can't find the container with id 929d2e911125d9145cc0a8ea2b5dcb00a07175d1e52f3266c0d712be7bbe1d63
Sep 08 16:09:37.810841 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:37.810808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f" event={"ID":"b4cd44f9-5de2-40a8-9909-c988977619a8","Type":"ContainerStarted","Data":"4ede40cc381031d8fac9a5c81805f68d094bbaa03c436d9a4614ba1c9531081a"}
Sep 08 16:09:37.811229 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:37.810859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f" event={"ID":"b4cd44f9-5de2-40a8-9909-c988977619a8","Type":"ContainerStarted","Data":"929d2e911125d9145cc0a8ea2b5dcb00a07175d1e52f3266c0d712be7bbe1d63"}
Sep 08 16:09:37.811229 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:37.810901    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"
Sep 08 16:09:37.831018 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:37.830968    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f" podStartSLOduration=1.8309504909999998 podStartE2EDuration="1.830950491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:09:37.828572192 +0000 UTC m=+1524.452264544" watchObservedRunningTime="2026-09-08 16:09:37.830950491 +0000 UTC m=+1524.454642818"
Sep 08 16:09:48.818464 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:48.818432    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"
Sep 08 16:09:54.900555 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:54.900518    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerID="7c004e47e61dad21a5561e6cbe8d1366e4ccbb38de490aa30f460becb0500eee" exitCode=0
Sep 08 16:09:54.900946 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:54.900588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" event={"ID":"3d40d8bd-69bd-482a-91c6-098992fe5324","Type":"ContainerDied","Data":"7c004e47e61dad21a5561e6cbe8d1366e4ccbb38de490aa30f460becb0500eee"}
Sep 08 16:09:54.900946 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:54.900937    2581 scope.go:117] "RemoveContainer" containerID="7c004e47e61dad21a5561e6cbe8d1366e4ccbb38de490aa30f460becb0500eee"
Sep 08 16:09:54.902106 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:54.902080    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:09:54.902173 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:54.902147    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:09:54.902210 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:54.902179    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:09:55.907742 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:55.907702    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" event={"ID":"3d40d8bd-69bd-482a-91c6-098992fe5324","Type":"ContainerStarted","Data":"231c7df305d847e7430b96803c98683df84f8b087d1d4764115c10dfeca2b687"}
Sep 08 16:09:56.763988 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:56.763951    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:09:56.764172 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:56.764156    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 08 16:09:56.764224 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:56.764200    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 08 16:09:57.763860 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:57.763820    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 08 16:09:57.764348 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:57.763885    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 08 16:09:58.763994 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:58.763958    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 08 16:09:58.764385 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:58.764013    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 08 16:09:59.763498 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:59.763459    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 08 16:09:59.763676 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:09:59.763522    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 08 16:10:00.763684 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:00.763647    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Sep 08 16:10:00.764057 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:00.763703    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Sep 08 16:10:01.767791 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:01.767761    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:10:01.768288 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:01.768034    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:10:01.768761 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:01.768745    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:10:06.525070 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.525038    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"]
Sep 08 16:10:06.525486 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.525254    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f" podUID="b4cd44f9-5de2-40a8-9909-c988977619a8" containerName="payload-processing" containerID="cri-o://4ede40cc381031d8fac9a5c81805f68d094bbaa03c436d9a4614ba1c9531081a" gracePeriod=30
Sep 08 16:10:06.784702 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.784642    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"
Sep 08 16:10:06.821788 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.821760    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72zxd\" (UniqueName: \"kubernetes.io/projected/b4cd44f9-5de2-40a8-9909-c988977619a8-kube-api-access-72zxd\") pod \"b4cd44f9-5de2-40a8-9909-c988977619a8\" (UID: \"b4cd44f9-5de2-40a8-9909-c988977619a8\") "
Sep 08 16:10:06.821935 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.821807    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4cd44f9-5de2-40a8-9909-c988977619a8-plugins-config-volume\") pod \"b4cd44f9-5de2-40a8-9909-c988977619a8\" (UID: \"b4cd44f9-5de2-40a8-9909-c988977619a8\") "
Sep 08 16:10:06.822283 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.822250    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4cd44f9-5de2-40a8-9909-c988977619a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4cd44f9-5de2-40a8-9909-c988977619a8" (UID: "b4cd44f9-5de2-40a8-9909-c988977619a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:10:06.823832 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.823807    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cd44f9-5de2-40a8-9909-c988977619a8-kube-api-access-72zxd" (OuterVolumeSpecName: "kube-api-access-72zxd") pod "b4cd44f9-5de2-40a8-9909-c988977619a8" (UID: "b4cd44f9-5de2-40a8-9909-c988977619a8"). InnerVolumeSpecName "kube-api-access-72zxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:10:06.923478 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.923452    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72zxd\" (UniqueName: \"kubernetes.io/projected/b4cd44f9-5de2-40a8-9909-c988977619a8-kube-api-access-72zxd\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:10:06.923478 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.923475    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4cd44f9-5de2-40a8-9909-c988977619a8-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:10:06.961714 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.961686    2581 generic.go:356] "Generic (PLEG): container finished" podID="b4cd44f9-5de2-40a8-9909-c988977619a8" containerID="4ede40cc381031d8fac9a5c81805f68d094bbaa03c436d9a4614ba1c9531081a" exitCode=0
Sep 08 16:10:06.961806 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.961731    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"
Sep 08 16:10:06.961806 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.961739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f" event={"ID":"b4cd44f9-5de2-40a8-9909-c988977619a8","Type":"ContainerDied","Data":"4ede40cc381031d8fac9a5c81805f68d094bbaa03c436d9a4614ba1c9531081a"}
Sep 08 16:10:06.961806 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.961777    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f" event={"ID":"b4cd44f9-5de2-40a8-9909-c988977619a8","Type":"ContainerDied","Data":"929d2e911125d9145cc0a8ea2b5dcb00a07175d1e52f3266c0d712be7bbe1d63"}
Sep 08 16:10:06.961806 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.961798    2581 scope.go:117] "RemoveContainer" containerID="4ede40cc381031d8fac9a5c81805f68d094bbaa03c436d9a4614ba1c9531081a"
Sep 08 16:10:06.972960 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.972939    2581 scope.go:117] "RemoveContainer" containerID="4ede40cc381031d8fac9a5c81805f68d094bbaa03c436d9a4614ba1c9531081a"
Sep 08 16:10:06.973241 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:10:06.973222    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ede40cc381031d8fac9a5c81805f68d094bbaa03c436d9a4614ba1c9531081a\": container with ID starting with 4ede40cc381031d8fac9a5c81805f68d094bbaa03c436d9a4614ba1c9531081a not found: ID does not exist" containerID="4ede40cc381031d8fac9a5c81805f68d094bbaa03c436d9a4614ba1c9531081a"
Sep 08 16:10:06.973291 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.973252    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ede40cc381031d8fac9a5c81805f68d094bbaa03c436d9a4614ba1c9531081a"} err="failed to get container status \"4ede40cc381031d8fac9a5c81805f68d094bbaa03c436d9a4614ba1c9531081a\": rpc error: code = NotFound desc = could not find container \"4ede40cc381031d8fac9a5c81805f68d094bbaa03c436d9a4614ba1c9531081a\": container with ID starting with 4ede40cc381031d8fac9a5c81805f68d094bbaa03c436d9a4614ba1c9531081a not found: ID does not exist"
Sep 08 16:10:06.984819 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.984799    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"]
Sep 08 16:10:06.989164 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:06.989144    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a2951a-865fd7685-9nb5f"]
Sep 08 16:10:07.901029 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:07.900995    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4cd44f9-5de2-40a8-9909-c988977619a8" path="/var/lib/kubelet/pods/b4cd44f9-5de2-40a8-9909-c988977619a8/volumes"
Sep 08 16:10:13.213428 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:13.213368    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"]
Sep 08 16:10:13.213860 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:13.213607    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerName="istio-proxy" containerID="cri-o://231c7df305d847e7430b96803c98683df84f8b087d1d4764115c10dfeca2b687" gracePeriod=30
Sep 08 16:10:16.770513 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:16.770472    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 08 16:10:16.770917 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:16.770536    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:10:18.467026 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.467004    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:10:18.522477 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.522376    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-workload-socket\") pod \"3d40d8bd-69bd-482a-91c6-098992fe5324\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") "
Sep 08 16:10:18.522625 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.522532    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-workload-certs\") pod \"3d40d8bd-69bd-482a-91c6-098992fe5324\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") "
Sep 08 16:10:18.522625 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.522565    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d40d8bd-69bd-482a-91c6-098992fe5324-istiod-ca-cert\") pod \"3d40d8bd-69bd-482a-91c6-098992fe5324\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") "
Sep 08 16:10:18.522625 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.522593    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-podinfo\") pod \"3d40d8bd-69bd-482a-91c6-098992fe5324\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") "
Sep 08 16:10:18.522625 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.522619    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqhpz\" (UniqueName: \"kubernetes.io/projected/3d40d8bd-69bd-482a-91c6-098992fe5324-kube-api-access-pqhpz\") pod \"3d40d8bd-69bd-482a-91c6-098992fe5324\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") "
Sep 08 16:10:18.522847 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.522651    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-credential-socket\") pod \"3d40d8bd-69bd-482a-91c6-098992fe5324\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") "
Sep 08 16:10:18.522847 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.522660    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d40d8bd-69bd-482a-91c6-098992fe5324" (UID: "3d40d8bd-69bd-482a-91c6-098992fe5324"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:10:18.522847 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.522681    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-token\") pod \"3d40d8bd-69bd-482a-91c6-098992fe5324\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") "
Sep 08 16:10:18.522847 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.522715    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-envoy\") pod \"3d40d8bd-69bd-482a-91c6-098992fe5324\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") "
Sep 08 16:10:18.522847 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.522742    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-data\") pod \"3d40d8bd-69bd-482a-91c6-098992fe5324\" (UID: \"3d40d8bd-69bd-482a-91c6-098992fe5324\") "
Sep 08 16:10:18.523102 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.522824    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d40d8bd-69bd-482a-91c6-098992fe5324" (UID: "3d40d8bd-69bd-482a-91c6-098992fe5324"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:10:18.523164 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.523101    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-workload-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:10:18.523164 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.523003    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d40d8bd-69bd-482a-91c6-098992fe5324-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d40d8bd-69bd-482a-91c6-098992fe5324" (UID: "3d40d8bd-69bd-482a-91c6-098992fe5324"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:10:18.523164 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.523093    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d40d8bd-69bd-482a-91c6-098992fe5324" (UID: "3d40d8bd-69bd-482a-91c6-098992fe5324"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:10:18.523164 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.523121    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-workload-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:10:18.523452 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.523243    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d40d8bd-69bd-482a-91c6-098992fe5324" (UID: "3d40d8bd-69bd-482a-91c6-098992fe5324"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:10:18.524860 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.524825    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d40d8bd-69bd-482a-91c6-098992fe5324" (UID: "3d40d8bd-69bd-482a-91c6-098992fe5324"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:10:18.524963 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.524874    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d40d8bd-69bd-482a-91c6-098992fe5324" (UID: "3d40d8bd-69bd-482a-91c6-098992fe5324"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:10:18.524963 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.524880    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d40d8bd-69bd-482a-91c6-098992fe5324" (UID: "3d40d8bd-69bd-482a-91c6-098992fe5324"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:10:18.525042 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.524970    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d40d8bd-69bd-482a-91c6-098992fe5324-kube-api-access-pqhpz" (OuterVolumeSpecName: "kube-api-access-pqhpz") pod "3d40d8bd-69bd-482a-91c6-098992fe5324" (UID: "3d40d8bd-69bd-482a-91c6-098992fe5324"). InnerVolumeSpecName "kube-api-access-pqhpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:10:18.624648 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.624620    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d40d8bd-69bd-482a-91c6-098992fe5324-istiod-ca-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:10:18.624648 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.624644    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-podinfo\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:10:18.624810 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.624654    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqhpz\" (UniqueName: \"kubernetes.io/projected/3d40d8bd-69bd-482a-91c6-098992fe5324-kube-api-access-pqhpz\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:10:18.624810 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.624665    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-credential-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:10:18.624810 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.624673    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-token\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:10:18.624810 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.624681    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-envoy\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:10:18.624810 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:18.624688    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d40d8bd-69bd-482a-91c6-098992fe5324-istio-data\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:10:19.026560 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:19.026519    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerID="231c7df305d847e7430b96803c98683df84f8b087d1d4764115c10dfeca2b687" exitCode=0
Sep 08 16:10:19.026726 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:19.026571    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"
Sep 08 16:10:19.026726 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:19.026607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" event={"ID":"3d40d8bd-69bd-482a-91c6-098992fe5324","Type":"ContainerDied","Data":"231c7df305d847e7430b96803c98683df84f8b087d1d4764115c10dfeca2b687"}
Sep 08 16:10:19.026726 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:19.026662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx" event={"ID":"3d40d8bd-69bd-482a-91c6-098992fe5324","Type":"ContainerDied","Data":"0cefedb88da84693b32568aa5ec7c597857cf033b5daa678de71f1d7722331ca"}
Sep 08 16:10:19.026726 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:19.026684    2581 scope.go:117] "RemoveContainer" containerID="231c7df305d847e7430b96803c98683df84f8b087d1d4764115c10dfeca2b687"
Sep 08 16:10:19.036482 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:19.036465    2581 scope.go:117] "RemoveContainer" containerID="7c004e47e61dad21a5561e6cbe8d1366e4ccbb38de490aa30f460becb0500eee"
Sep 08 16:10:19.044343 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:19.044326    2581 scope.go:117] "RemoveContainer" containerID="231c7df305d847e7430b96803c98683df84f8b087d1d4764115c10dfeca2b687"
Sep 08 16:10:19.044615 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:10:19.044596    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231c7df305d847e7430b96803c98683df84f8b087d1d4764115c10dfeca2b687\": container with ID starting with 231c7df305d847e7430b96803c98683df84f8b087d1d4764115c10dfeca2b687 not found: ID does not exist" containerID="231c7df305d847e7430b96803c98683df84f8b087d1d4764115c10dfeca2b687"
Sep 08 16:10:19.044657 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:19.044622    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231c7df305d847e7430b96803c98683df84f8b087d1d4764115c10dfeca2b687"} err="failed to get container status \"231c7df305d847e7430b96803c98683df84f8b087d1d4764115c10dfeca2b687\": rpc error: code = NotFound desc = could not find container \"231c7df305d847e7430b96803c98683df84f8b087d1d4764115c10dfeca2b687\": container with ID starting with 231c7df305d847e7430b96803c98683df84f8b087d1d4764115c10dfeca2b687 not found: ID does not exist"
Sep 08 16:10:19.044657 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:19.044637    2581 scope.go:117] "RemoveContainer" containerID="7c004e47e61dad21a5561e6cbe8d1366e4ccbb38de490aa30f460becb0500eee"
Sep 08 16:10:19.044875 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:10:19.044860    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c004e47e61dad21a5561e6cbe8d1366e4ccbb38de490aa30f460becb0500eee\": container with ID starting with 7c004e47e61dad21a5561e6cbe8d1366e4ccbb38de490aa30f460becb0500eee not found: ID does not exist" containerID="7c004e47e61dad21a5561e6cbe8d1366e4ccbb38de490aa30f460becb0500eee"
Sep 08 16:10:19.044926 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:19.044879    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c004e47e61dad21a5561e6cbe8d1366e4ccbb38de490aa30f460becb0500eee"} err="failed to get container status \"7c004e47e61dad21a5561e6cbe8d1366e4ccbb38de490aa30f460becb0500eee\": rpc error: code = NotFound desc = could not find container \"7c004e47e61dad21a5561e6cbe8d1366e4ccbb38de490aa30f460becb0500eee\": container with ID starting with 7c004e47e61dad21a5561e6cbe8d1366e4ccbb38de490aa30f460becb0500eee not found: ID does not exist"
Sep 08 16:10:19.050083 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:19.050058    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"]
Sep 08 16:10:19.054524 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:19.054504    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a2951a-openshift-default-f8469c998-xmkjx"]
Sep 08 16:10:19.901084 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:10:19.901042    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" path="/var/lib/kubelet/pods/3d40d8bd-69bd-482a-91c6-098992fe5324/volumes"
Sep 08 16:11:31.864775 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:31.864735    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"]
Sep 08 16:11:31.865370 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:31.865007    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt" podUID="2482a9f4-045d-48ee-8b16-28f42f4ebc13" containerName="maas-api" containerID="cri-o://5e39d0066666f030b65d993effcf94c4f3c2bd47cd5a0df4b1f63fccf46067de" gracePeriod=30
Sep 08 16:11:31.915182 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:31.915150    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"]
Sep 08 16:11:31.915487 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:31.915427    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l" podUID="d672182b-42d5-4333-9830-c853a1e798c5" containerName="payload-processing" containerID="cri-o://3aa10240ffda3f3c2b4f099f254dd0e9267da51e97f3f83ca6742e7198429b1e" gracePeriod=30
Sep 08 16:11:31.925794 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:31.925766    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"]
Sep 08 16:11:31.926019 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:31.925983    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs" podUID="ccce6db9-8fad-43a8-97fc-8b731216feaf" containerName="payload-pre-processing" containerID="cri-o://d1ae863c4dcf209d08b0bbacd60432fb067d5ef38feff6fbe408af732191a03b" gracePeriod=30
Sep 08 16:11:32.179219 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.179191    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"
Sep 08 16:11:32.218358 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.218336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"
Sep 08 16:11:32.228574 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.228552    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:11:32.344585 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.344557    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-metrics-certs\") pod \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") "
Sep 08 16:11:32.344752 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.344620    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccce6db9-8fad-43a8-97fc-8b731216feaf-plugins-config-volume\") pod \"ccce6db9-8fad-43a8-97fc-8b731216feaf\" (UID: \"ccce6db9-8fad-43a8-97fc-8b731216feaf\") "
Sep 08 16:11:32.344752 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.344646    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-maas-api-tls\") pod \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") "
Sep 08 16:11:32.344752 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.344664    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2482a9f4-045d-48ee-8b16-28f42f4ebc13-openshift-service-ca\") pod \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") "
Sep 08 16:11:32.344985 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.344754    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76v5w\" (UniqueName: \"kubernetes.io/projected/ccce6db9-8fad-43a8-97fc-8b731216feaf-kube-api-access-76v5w\") pod \"ccce6db9-8fad-43a8-97fc-8b731216feaf\" (UID: \"ccce6db9-8fad-43a8-97fc-8b731216feaf\") "
Sep 08 16:11:32.344985 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.344781    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl75t\" (UniqueName: \"kubernetes.io/projected/2482a9f4-045d-48ee-8b16-28f42f4ebc13-kube-api-access-zl75t\") pod \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\" (UID: \"2482a9f4-045d-48ee-8b16-28f42f4ebc13\") "
Sep 08 16:11:32.344985 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.344822    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d672182b-42d5-4333-9830-c853a1e798c5-plugins-config-volume\") pod \"d672182b-42d5-4333-9830-c853a1e798c5\" (UID: \"d672182b-42d5-4333-9830-c853a1e798c5\") "
Sep 08 16:11:32.344985 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.344855    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd554\" (UniqueName: \"kubernetes.io/projected/d672182b-42d5-4333-9830-c853a1e798c5-kube-api-access-sd554\") pod \"d672182b-42d5-4333-9830-c853a1e798c5\" (UID: \"d672182b-42d5-4333-9830-c853a1e798c5\") "
Sep 08 16:11:32.345198 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.345133    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2482a9f4-045d-48ee-8b16-28f42f4ebc13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2482a9f4-045d-48ee-8b16-28f42f4ebc13" (UID: "2482a9f4-045d-48ee-8b16-28f42f4ebc13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:11:32.345198 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.345164    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccce6db9-8fad-43a8-97fc-8b731216feaf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ccce6db9-8fad-43a8-97fc-8b731216feaf" (UID: "ccce6db9-8fad-43a8-97fc-8b731216feaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:11:32.345313 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.345226    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d672182b-42d5-4333-9830-c853a1e798c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d672182b-42d5-4333-9830-c853a1e798c5" (UID: "d672182b-42d5-4333-9830-c853a1e798c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:11:32.347085 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.347050    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccce6db9-8fad-43a8-97fc-8b731216feaf-kube-api-access-76v5w" (OuterVolumeSpecName: "kube-api-access-76v5w") pod "ccce6db9-8fad-43a8-97fc-8b731216feaf" (UID: "ccce6db9-8fad-43a8-97fc-8b731216feaf"). InnerVolumeSpecName "kube-api-access-76v5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:11:32.347195 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.347137    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2482a9f4-045d-48ee-8b16-28f42f4ebc13" (UID: "2482a9f4-045d-48ee-8b16-28f42f4ebc13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:11:32.347195 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.347175    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2482a9f4-045d-48ee-8b16-28f42f4ebc13-kube-api-access-zl75t" (OuterVolumeSpecName: "kube-api-access-zl75t") pod "2482a9f4-045d-48ee-8b16-28f42f4ebc13" (UID: "2482a9f4-045d-48ee-8b16-28f42f4ebc13"). InnerVolumeSpecName "kube-api-access-zl75t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:11:32.347556 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.347535    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2482a9f4-045d-48ee-8b16-28f42f4ebc13" (UID: "2482a9f4-045d-48ee-8b16-28f42f4ebc13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:11:32.347625 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.347556    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d672182b-42d5-4333-9830-c853a1e798c5-kube-api-access-sd554" (OuterVolumeSpecName: "kube-api-access-sd554") pod "d672182b-42d5-4333-9830-c853a1e798c5" (UID: "d672182b-42d5-4333-9830-c853a1e798c5"). InnerVolumeSpecName "kube-api-access-sd554". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:11:32.395985 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.395956    2581 generic.go:356] "Generic (PLEG): container finished" podID="2482a9f4-045d-48ee-8b16-28f42f4ebc13" containerID="5e39d0066666f030b65d993effcf94c4f3c2bd47cd5a0df4b1f63fccf46067de" exitCode=0
Sep 08 16:11:32.396082 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.396000    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"
Sep 08 16:11:32.396082 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.396012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt" event={"ID":"2482a9f4-045d-48ee-8b16-28f42f4ebc13","Type":"ContainerDied","Data":"5e39d0066666f030b65d993effcf94c4f3c2bd47cd5a0df4b1f63fccf46067de"}
Sep 08 16:11:32.396082 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.396067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt" event={"ID":"2482a9f4-045d-48ee-8b16-28f42f4ebc13","Type":"ContainerDied","Data":"bb84e9f3661026698916c0659c65a793212d52ea99b5689572742d8d6d847264"}
Sep 08 16:11:32.396241 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.396100    2581 scope.go:117] "RemoveContainer" containerID="5e39d0066666f030b65d993effcf94c4f3c2bd47cd5a0df4b1f63fccf46067de"
Sep 08 16:11:32.397484 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.397457    2581 generic.go:356] "Generic (PLEG): container finished" podID="d672182b-42d5-4333-9830-c853a1e798c5" containerID="3aa10240ffda3f3c2b4f099f254dd0e9267da51e97f3f83ca6742e7198429b1e" exitCode=0
Sep 08 16:11:32.397592 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.397485    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l" event={"ID":"d672182b-42d5-4333-9830-c853a1e798c5","Type":"ContainerDied","Data":"3aa10240ffda3f3c2b4f099f254dd0e9267da51e97f3f83ca6742e7198429b1e"}
Sep 08 16:11:32.397592 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.397499    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"
Sep 08 16:11:32.397592 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.397535    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l" event={"ID":"d672182b-42d5-4333-9830-c853a1e798c5","Type":"ContainerDied","Data":"340e9f9617d909002f73ac1ab83d1a6e30d576b1cf6e24e3d70951af5ddd4f3b"}
Sep 08 16:11:32.399080 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.399057    2581 generic.go:356] "Generic (PLEG): container finished" podID="ccce6db9-8fad-43a8-97fc-8b731216feaf" containerID="d1ae863c4dcf209d08b0bbacd60432fb067d5ef38feff6fbe408af732191a03b" exitCode=0
Sep 08 16:11:32.399194 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.399101    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"
Sep 08 16:11:32.399194 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.399133    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs" event={"ID":"ccce6db9-8fad-43a8-97fc-8b731216feaf","Type":"ContainerDied","Data":"d1ae863c4dcf209d08b0bbacd60432fb067d5ef38feff6fbe408af732191a03b"}
Sep 08 16:11:32.399194 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.399179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs" event={"ID":"ccce6db9-8fad-43a8-97fc-8b731216feaf","Type":"ContainerDied","Data":"24fda8c15462dc51f2dcdc3c73ba248470783a898de16b11bc6895bdd32ea163"}
Sep 08 16:11:32.407594 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.407554    2581 scope.go:117] "RemoveContainer" containerID="5e39d0066666f030b65d993effcf94c4f3c2bd47cd5a0df4b1f63fccf46067de"
Sep 08 16:11:32.410792 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:11:32.410756    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e39d0066666f030b65d993effcf94c4f3c2bd47cd5a0df4b1f63fccf46067de\": container with ID starting with 5e39d0066666f030b65d993effcf94c4f3c2bd47cd5a0df4b1f63fccf46067de not found: ID does not exist" containerID="5e39d0066666f030b65d993effcf94c4f3c2bd47cd5a0df4b1f63fccf46067de"
Sep 08 16:11:32.410908 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.410804    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e39d0066666f030b65d993effcf94c4f3c2bd47cd5a0df4b1f63fccf46067de"} err="failed to get container status \"5e39d0066666f030b65d993effcf94c4f3c2bd47cd5a0df4b1f63fccf46067de\": rpc error: code = NotFound desc = could not find container \"5e39d0066666f030b65d993effcf94c4f3c2bd47cd5a0df4b1f63fccf46067de\": container with ID starting with 5e39d0066666f030b65d993effcf94c4f3c2bd47cd5a0df4b1f63fccf46067de not found: ID does not exist"
Sep 08 16:11:32.410908 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.410830    2581 scope.go:117] "RemoveContainer" containerID="3aa10240ffda3f3c2b4f099f254dd0e9267da51e97f3f83ca6742e7198429b1e"
Sep 08 16:11:32.422205 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.422182    2581 scope.go:117] "RemoveContainer" containerID="3aa10240ffda3f3c2b4f099f254dd0e9267da51e97f3f83ca6742e7198429b1e"
Sep 08 16:11:32.422822 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:11:32.422801    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa10240ffda3f3c2b4f099f254dd0e9267da51e97f3f83ca6742e7198429b1e\": container with ID starting with 3aa10240ffda3f3c2b4f099f254dd0e9267da51e97f3f83ca6742e7198429b1e not found: ID does not exist" containerID="3aa10240ffda3f3c2b4f099f254dd0e9267da51e97f3f83ca6742e7198429b1e"
Sep 08 16:11:32.422898 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.422829    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa10240ffda3f3c2b4f099f254dd0e9267da51e97f3f83ca6742e7198429b1e"} err="failed to get container status \"3aa10240ffda3f3c2b4f099f254dd0e9267da51e97f3f83ca6742e7198429b1e\": rpc error: code = NotFound desc = could not find container \"3aa10240ffda3f3c2b4f099f254dd0e9267da51e97f3f83ca6742e7198429b1e\": container with ID starting with 3aa10240ffda3f3c2b4f099f254dd0e9267da51e97f3f83ca6742e7198429b1e not found: ID does not exist"
Sep 08 16:11:32.422898 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.422844    2581 scope.go:117] "RemoveContainer" containerID="d1ae863c4dcf209d08b0bbacd60432fb067d5ef38feff6fbe408af732191a03b"
Sep 08 16:11:32.428634 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.428612    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"]
Sep 08 16:11:32.431179 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.431158    2581 scope.go:117] "RemoveContainer" containerID="d1ae863c4dcf209d08b0bbacd60432fb067d5ef38feff6fbe408af732191a03b"
Sep 08 16:11:32.431507 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:11:32.431486    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ae863c4dcf209d08b0bbacd60432fb067d5ef38feff6fbe408af732191a03b\": container with ID starting with d1ae863c4dcf209d08b0bbacd60432fb067d5ef38feff6fbe408af732191a03b not found: ID does not exist" containerID="d1ae863c4dcf209d08b0bbacd60432fb067d5ef38feff6fbe408af732191a03b"
Sep 08 16:11:32.431582 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.431513    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ae863c4dcf209d08b0bbacd60432fb067d5ef38feff6fbe408af732191a03b"} err="failed to get container status \"d1ae863c4dcf209d08b0bbacd60432fb067d5ef38feff6fbe408af732191a03b\": rpc error: code = NotFound desc = could not find container \"d1ae863c4dcf209d08b0bbacd60432fb067d5ef38feff6fbe408af732191a03b\": container with ID starting with d1ae863c4dcf209d08b0bbacd60432fb067d5ef38feff6fbe408af732191a03b not found: ID does not exist"
Sep 08 16:11:32.431930 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.431913    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a7297-5665cd9b58-zf6tt"]
Sep 08 16:11:32.441399 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.441368    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"]
Sep 08 16:11:32.446571 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.446553    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76v5w\" (UniqueName: \"kubernetes.io/projected/ccce6db9-8fad-43a8-97fc-8b731216feaf-kube-api-access-76v5w\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:32.446627 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.446573    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl75t\" (UniqueName: \"kubernetes.io/projected/2482a9f4-045d-48ee-8b16-28f42f4ebc13-kube-api-access-zl75t\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:32.446627 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.446584    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d672182b-42d5-4333-9830-c853a1e798c5-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:32.446627 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.446592    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd554\" (UniqueName: \"kubernetes.io/projected/d672182b-42d5-4333-9830-c853a1e798c5-kube-api-access-sd554\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:32.446627 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.446601    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-metrics-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:32.446627 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.446609    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccce6db9-8fad-43a8-97fc-8b731216feaf-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:32.446627 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.446617    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2482a9f4-045d-48ee-8b16-28f42f4ebc13-maas-api-tls\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:32.446627 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.446625    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2482a9f4-045d-48ee-8b16-28f42f4ebc13-openshift-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:32.453832 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.453807    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7a7297-6694d99b94-xldhs"]
Sep 08 16:11:32.466942 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.466667    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"]
Sep 08 16:11:32.468868 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:32.468847    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a7297-8667c7c49c-x9v6l"]
Sep 08 16:11:33.902339 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:33.902306    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2482a9f4-045d-48ee-8b16-28f42f4ebc13" path="/var/lib/kubelet/pods/2482a9f4-045d-48ee-8b16-28f42f4ebc13/volumes"
Sep 08 16:11:33.902709 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:33.902691    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccce6db9-8fad-43a8-97fc-8b731216feaf" path="/var/lib/kubelet/pods/ccce6db9-8fad-43a8-97fc-8b731216feaf/volumes"
Sep 08 16:11:33.903005 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:33.902992    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d672182b-42d5-4333-9830-c853a1e798c5" path="/var/lib/kubelet/pods/d672182b-42d5-4333-9830-c853a1e798c5/volumes"
Sep 08 16:11:49.306892 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.306851    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"]
Sep 08 16:11:49.307455 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.307061    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w" podUID="f7c3105e-4fdb-4dd9-898c-1aebdabb6cff" containerName="maas-api" containerID="cri-o://575b3d10a79052fc191b5016963929b8937de53e4e1ec5b14e0e76de60b066f9" gracePeriod=30
Sep 08 16:11:49.357226 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.357192    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"]
Sep 08 16:11:49.357529 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.357471    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx" podUID="8b96330b-5c01-48d5-acc0-422148a59dd3" containerName="payload-processing" containerID="cri-o://92d1f6377ae412ed546f669089e83eee7af64a51fad363a589f5dd398d42a7a8" gracePeriod=30
Sep 08 16:11:49.364211 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.364185    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"]
Sep 08 16:11:49.364435 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.364403    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz" podUID="9d8b1004-ff7e-449a-8351-e5b9d69ee84d" containerName="payload-pre-processing" containerID="cri-o://5e45d77acf6601c26a035077887c0b0f6acdb5b9dbaad12be248cd2cf83b9a0a" gracePeriod=30
Sep 08 16:11:49.487868 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.487832    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b96330b-5c01-48d5-acc0-422148a59dd3" containerID="92d1f6377ae412ed546f669089e83eee7af64a51fad363a589f5dd398d42a7a8" exitCode=0
Sep 08 16:11:49.488020 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.487929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx" event={"ID":"8b96330b-5c01-48d5-acc0-422148a59dd3","Type":"ContainerDied","Data":"92d1f6377ae412ed546f669089e83eee7af64a51fad363a589f5dd398d42a7a8"}
Sep 08 16:11:49.490159 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.490133    2581 generic.go:356] "Generic (PLEG): container finished" podID="9d8b1004-ff7e-449a-8351-e5b9d69ee84d" containerID="5e45d77acf6601c26a035077887c0b0f6acdb5b9dbaad12be248cd2cf83b9a0a" exitCode=0
Sep 08 16:11:49.490286 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.490210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz" event={"ID":"9d8b1004-ff7e-449a-8351-e5b9d69ee84d","Type":"ContainerDied","Data":"5e45d77acf6601c26a035077887c0b0f6acdb5b9dbaad12be248cd2cf83b9a0a"}
Sep 08 16:11:49.492117 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.492095    2581 generic.go:356] "Generic (PLEG): container finished" podID="f7c3105e-4fdb-4dd9-898c-1aebdabb6cff" containerID="575b3d10a79052fc191b5016963929b8937de53e4e1ec5b14e0e76de60b066f9" exitCode=0
Sep 08 16:11:49.492231 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.492162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w" event={"ID":"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff","Type":"ContainerDied","Data":"575b3d10a79052fc191b5016963929b8937de53e4e1ec5b14e0e76de60b066f9"}
Sep 08 16:11:49.629628 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.629602    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"
Sep 08 16:11:49.639108 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.639080    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:11:49.675870 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.675844    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"
Sep 08 16:11:49.689538 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.689512    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-maas-api-tls\") pod \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") "
Sep 08 16:11:49.689635 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.689551    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-metrics-certs\") pod \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") "
Sep 08 16:11:49.689635 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.689590    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgddj\" (UniqueName: \"kubernetes.io/projected/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-kube-api-access-lgddj\") pod \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") "
Sep 08 16:11:49.689739 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.689640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwjnl\" (UniqueName: \"kubernetes.io/projected/8b96330b-5c01-48d5-acc0-422148a59dd3-kube-api-access-jwjnl\") pod \"8b96330b-5c01-48d5-acc0-422148a59dd3\" (UID: \"8b96330b-5c01-48d5-acc0-422148a59dd3\") "
Sep 08 16:11:49.689739 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.689666    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b96330b-5c01-48d5-acc0-422148a59dd3-plugins-config-volume\") pod \"8b96330b-5c01-48d5-acc0-422148a59dd3\" (UID: \"8b96330b-5c01-48d5-acc0-422148a59dd3\") "
Sep 08 16:11:49.689739 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.689687    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-openshift-service-ca\") pod \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\" (UID: \"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff\") "
Sep 08 16:11:49.692432 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.691996    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b96330b-5c01-48d5-acc0-422148a59dd3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b96330b-5c01-48d5-acc0-422148a59dd3" (UID: "8b96330b-5c01-48d5-acc0-422148a59dd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:11:49.692432 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.692164    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f7c3105e-4fdb-4dd9-898c-1aebdabb6cff" (UID: "f7c3105e-4fdb-4dd9-898c-1aebdabb6cff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:11:49.693892 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.693850    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b96330b-5c01-48d5-acc0-422148a59dd3-kube-api-access-jwjnl" (OuterVolumeSpecName: "kube-api-access-jwjnl") pod "8b96330b-5c01-48d5-acc0-422148a59dd3" (UID: "8b96330b-5c01-48d5-acc0-422148a59dd3"). InnerVolumeSpecName "kube-api-access-jwjnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:11:49.694006 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.693902    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f7c3105e-4fdb-4dd9-898c-1aebdabb6cff" (UID: "f7c3105e-4fdb-4dd9-898c-1aebdabb6cff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:11:49.694006 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.693987    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f7c3105e-4fdb-4dd9-898c-1aebdabb6cff" (UID: "f7c3105e-4fdb-4dd9-898c-1aebdabb6cff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:11:49.694116 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.694039    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-kube-api-access-lgddj" (OuterVolumeSpecName: "kube-api-access-lgddj") pod "f7c3105e-4fdb-4dd9-898c-1aebdabb6cff" (UID: "f7c3105e-4fdb-4dd9-898c-1aebdabb6cff"). InnerVolumeSpecName "kube-api-access-lgddj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:11:49.791213 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.791188    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8b1004-ff7e-449a-8351-e5b9d69ee84d-plugins-config-volume\") pod \"9d8b1004-ff7e-449a-8351-e5b9d69ee84d\" (UID: \"9d8b1004-ff7e-449a-8351-e5b9d69ee84d\") "
Sep 08 16:11:49.791334 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.791231    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22gp2\" (UniqueName: \"kubernetes.io/projected/9d8b1004-ff7e-449a-8351-e5b9d69ee84d-kube-api-access-22gp2\") pod \"9d8b1004-ff7e-449a-8351-e5b9d69ee84d\" (UID: \"9d8b1004-ff7e-449a-8351-e5b9d69ee84d\") "
Sep 08 16:11:49.791610 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.791585    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwjnl\" (UniqueName: \"kubernetes.io/projected/8b96330b-5c01-48d5-acc0-422148a59dd3-kube-api-access-jwjnl\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:49.791679 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.791603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8b1004-ff7e-449a-8351-e5b9d69ee84d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d8b1004-ff7e-449a-8351-e5b9d69ee84d" (UID: "9d8b1004-ff7e-449a-8351-e5b9d69ee84d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:11:49.791679 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.791613    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b96330b-5c01-48d5-acc0-422148a59dd3-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:49.791679 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.791628    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-openshift-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:49.791679 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.791640    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-maas-api-tls\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:49.791679 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.791649    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-metrics-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:49.791679 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.791656    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgddj\" (UniqueName: \"kubernetes.io/projected/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff-kube-api-access-lgddj\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:49.793150 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.793125    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8b1004-ff7e-449a-8351-e5b9d69ee84d-kube-api-access-22gp2" (OuterVolumeSpecName: "kube-api-access-22gp2") pod "9d8b1004-ff7e-449a-8351-e5b9d69ee84d" (UID: "9d8b1004-ff7e-449a-8351-e5b9d69ee84d"). InnerVolumeSpecName "kube-api-access-22gp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:11:49.892939 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.892915    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8b1004-ff7e-449a-8351-e5b9d69ee84d-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:49.892939 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:49.892938    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22gp2\" (UniqueName: \"kubernetes.io/projected/9d8b1004-ff7e-449a-8351-e5b9d69ee84d-kube-api-access-22gp2\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:11:50.498042 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.498010    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"
Sep 08 16:11:50.498497 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.498045    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w" event={"ID":"f7c3105e-4fdb-4dd9-898c-1aebdabb6cff","Type":"ContainerDied","Data":"a8ae926f34f6eea1e506c2ec41694b6c2a5601899d7b3d7f55339aa3718e73b1"}
Sep 08 16:11:50.498497 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.498100    2581 scope.go:117] "RemoveContainer" containerID="575b3d10a79052fc191b5016963929b8937de53e4e1ec5b14e0e76de60b066f9"
Sep 08 16:11:50.499542 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.499523    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"
Sep 08 16:11:50.499542 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.499529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx" event={"ID":"8b96330b-5c01-48d5-acc0-422148a59dd3","Type":"ContainerDied","Data":"155f1cca28a3d65ba6a4147fba098ddd2a907dffaa2639344b3d4d8751d1c721"}
Sep 08 16:11:50.501179 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.501149    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz" event={"ID":"9d8b1004-ff7e-449a-8351-e5b9d69ee84d","Type":"ContainerDied","Data":"93c6d83d1b31da666fe3e7112726cdd0c71c22342be1575943b1d376aca42808"}
Sep 08 16:11:50.501273 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.501180    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"
Sep 08 16:11:50.508252 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.508236    2581 scope.go:117] "RemoveContainer" containerID="92d1f6377ae412ed546f669089e83eee7af64a51fad363a589f5dd398d42a7a8"
Sep 08 16:11:50.516138 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.516120    2581 scope.go:117] "RemoveContainer" containerID="5e45d77acf6601c26a035077887c0b0f6acdb5b9dbaad12be248cd2cf83b9a0a"
Sep 08 16:11:50.520282 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.520254    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"]
Sep 08 16:11:50.524245 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.524177    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-108124-69bd494b6f-84z5w"]
Sep 08 16:11:50.533244 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.533220    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"]
Sep 08 16:11:50.535553 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.535506    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-108124-67fd864db6-6bgjz"]
Sep 08 16:11:50.544869 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.544846    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"]
Sep 08 16:11:50.548222 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:50.548203    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-108124-55f5d77799-mjnhx"]
Sep 08 16:11:51.901492 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:51.901459    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b96330b-5c01-48d5-acc0-422148a59dd3" path="/var/lib/kubelet/pods/8b96330b-5c01-48d5-acc0-422148a59dd3/volumes"
Sep 08 16:11:51.902002 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:51.901982    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d8b1004-ff7e-449a-8351-e5b9d69ee84d" path="/var/lib/kubelet/pods/9d8b1004-ff7e-449a-8351-e5b9d69ee84d/volumes"
Sep 08 16:11:51.902387 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:51.902368    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7c3105e-4fdb-4dd9-898c-1aebdabb6cff" path="/var/lib/kubelet/pods/f7c3105e-4fdb-4dd9-898c-1aebdabb6cff/volumes"
Sep 08 16:11:56.140775 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.140740    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"]
Sep 08 16:11:56.141135 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.140950    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerName="istio-proxy" containerID="cri-o://c46cdef1f9ba6ae01ff0640da2b7bb769896a7e4612d682c6ba4fcaedcdca750" gracePeriod=30
Sep 08 16:11:56.445302 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445232    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz"]
Sep 08 16:11:56.445621 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445608    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccce6db9-8fad-43a8-97fc-8b731216feaf" containerName="payload-pre-processing"
Sep 08 16:11:56.445674 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445623    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccce6db9-8fad-43a8-97fc-8b731216feaf" containerName="payload-pre-processing"
Sep 08 16:11:56.445674 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445630    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2482a9f4-045d-48ee-8b16-28f42f4ebc13" containerName="maas-api"
Sep 08 16:11:56.445674 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445635    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2482a9f4-045d-48ee-8b16-28f42f4ebc13" containerName="maas-api"
Sep 08 16:11:56.445674 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445644    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerName="istio-proxy"
Sep 08 16:11:56.445674 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445650    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerName="istio-proxy"
Sep 08 16:11:56.445674 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445657    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b96330b-5c01-48d5-acc0-422148a59dd3" containerName="payload-processing"
Sep 08 16:11:56.445674 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445664    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b96330b-5c01-48d5-acc0-422148a59dd3" containerName="payload-processing"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445677    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8b1004-ff7e-449a-8351-e5b9d69ee84d" containerName="payload-pre-processing"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445682    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8b1004-ff7e-449a-8351-e5b9d69ee84d" containerName="payload-pre-processing"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445690    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c3105e-4fdb-4dd9-898c-1aebdabb6cff" containerName="maas-api"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445696    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c3105e-4fdb-4dd9-898c-1aebdabb6cff" containerName="maas-api"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445704    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerName="istio-proxy"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445709    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerName="istio-proxy"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445722    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d672182b-42d5-4333-9830-c853a1e798c5" containerName="payload-processing"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445727    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d672182b-42d5-4333-9830-c853a1e798c5" containerName="payload-processing"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445733    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4cd44f9-5de2-40a8-9909-c988977619a8" containerName="payload-processing"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445738    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4cd44f9-5de2-40a8-9909-c988977619a8" containerName="payload-processing"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445797    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccce6db9-8fad-43a8-97fc-8b731216feaf" containerName="payload-pre-processing"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445805    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8b1004-ff7e-449a-8351-e5b9d69ee84d" containerName="payload-pre-processing"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445810    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b96330b-5c01-48d5-acc0-422148a59dd3" containerName="payload-processing"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445816    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerName="istio-proxy"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445822    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d40d8bd-69bd-482a-91c6-098992fe5324" containerName="istio-proxy"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445828    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2482a9f4-045d-48ee-8b16-28f42f4ebc13" containerName="maas-api"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445835    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d672182b-42d5-4333-9830-c853a1e798c5" containerName="payload-processing"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445839    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4cd44f9-5de2-40a8-9909-c988977619a8" containerName="payload-processing"
Sep 08 16:11:56.445885 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.445844    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c3105e-4fdb-4dd9-898c-1aebdabb6cff" containerName="maas-api"
Sep 08 16:11:56.450405 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.450369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz"
Sep 08 16:11:56.452694 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.452670    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:11:56.452810 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.452738    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65h2p\""
Sep 08 16:11:56.456785 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.456758    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz"]
Sep 08 16:11:56.542298 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.542269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd63fec1-2c15-42df-b7fe-11233be04433-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz\" (UID: \"dd63fec1-2c15-42df-b7fe-11233be04433\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz"
Sep 08 16:11:56.643870 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.643837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd63fec1-2c15-42df-b7fe-11233be04433-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz\" (UID: \"dd63fec1-2c15-42df-b7fe-11233be04433\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz"
Sep 08 16:11:56.644487 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.644467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd63fec1-2c15-42df-b7fe-11233be04433-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz\" (UID: \"dd63fec1-2c15-42df-b7fe-11233be04433\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz"
Sep 08 16:11:56.762097 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:56.762031    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz"
Sep 08 16:11:57.033262 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:57.033235    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz"]
Sep 08 16:11:57.036497 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:11:57.036458    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd63fec1_2c15_42df_b7fe_11233be04433.slice/crio-c8af38c500190d3e4ddf5cc73f6801471614e9daf2a4954caa53fa8606ed0899 WatchSource:0}: Error finding container c8af38c500190d3e4ddf5cc73f6801471614e9daf2a4954caa53fa8606ed0899: Status 404 returned error can't find the container with id c8af38c500190d3e4ddf5cc73f6801471614e9daf2a4954caa53fa8606ed0899
Sep 08 16:11:57.536780 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:57.536742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz" event={"ID":"dd63fec1-2c15-42df-b7fe-11233be04433","Type":"ContainerStarted","Data":"3fa3c893f0e5ae8e9f36b3abb3b8026fb359eebcb9e68e6d1706ffbd8dffd86f"}
Sep 08 16:11:57.537120 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:57.536791    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz" event={"ID":"dd63fec1-2c15-42df-b7fe-11233be04433","Type":"ContainerStarted","Data":"c8af38c500190d3e4ddf5cc73f6801471614e9daf2a4954caa53fa8606ed0899"}
Sep 08 16:11:58.542882 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:58.542849    2581 generic.go:356] "Generic (PLEG): container finished" podID="dd63fec1-2c15-42df-b7fe-11233be04433" containerID="3fa3c893f0e5ae8e9f36b3abb3b8026fb359eebcb9e68e6d1706ffbd8dffd86f" exitCode=0
Sep 08 16:11:58.543224 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:58.542936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz" event={"ID":"dd63fec1-2c15-42df-b7fe-11233be04433","Type":"ContainerDied","Data":"3fa3c893f0e5ae8e9f36b3abb3b8026fb359eebcb9e68e6d1706ffbd8dffd86f"}
Sep 08 16:11:59.681039 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:59.681013    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz"
Sep 08 16:11:59.770525 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:59.770494    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd63fec1-2c15-42df-b7fe-11233be04433-maas-api-service-ca\") pod \"dd63fec1-2c15-42df-b7fe-11233be04433\" (UID: \"dd63fec1-2c15-42df-b7fe-11233be04433\") "
Sep 08 16:11:59.770827 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:59.770804    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd63fec1-2c15-42df-b7fe-11233be04433-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd63fec1-2c15-42df-b7fe-11233be04433" (UID: "dd63fec1-2c15-42df-b7fe-11233be04433"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:11:59.871676 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:11:59.871616    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd63fec1-2c15-42df-b7fe-11233be04433-maas-api-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:00.554015 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:00.553984    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz"
Sep 08 16:12:00.554185 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:00.554025    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz" event={"ID":"dd63fec1-2c15-42df-b7fe-11233be04433","Type":"ContainerDied","Data":"c8af38c500190d3e4ddf5cc73f6801471614e9daf2a4954caa53fa8606ed0899"}
Sep 08 16:12:00.554185 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:00.554075    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8af38c500190d3e4ddf5cc73f6801471614e9daf2a4954caa53fa8606ed0899"
Sep 08 16:12:01.400868 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.400843    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:12:01.485114 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485053    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-envoy\") pod \"49d4b521-010e-4f98-9e55-efba77ed63c0\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") "
Sep 08 16:12:01.485114 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485093    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gzt9\" (UniqueName: \"kubernetes.io/projected/49d4b521-010e-4f98-9e55-efba77ed63c0-kube-api-access-5gzt9\") pod \"49d4b521-010e-4f98-9e55-efba77ed63c0\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") "
Sep 08 16:12:01.485295 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485136    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-workload-certs\") pod \"49d4b521-010e-4f98-9e55-efba77ed63c0\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") "
Sep 08 16:12:01.485295 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485163    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d4b521-010e-4f98-9e55-efba77ed63c0-istiod-ca-cert\") pod \"49d4b521-010e-4f98-9e55-efba77ed63c0\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") "
Sep 08 16:12:01.485488 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485320    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-workload-socket\") pod \"49d4b521-010e-4f98-9e55-efba77ed63c0\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") "
Sep 08 16:12:01.485488 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485414    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-credential-socket\") pod \"49d4b521-010e-4f98-9e55-efba77ed63c0\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") "
Sep 08 16:12:01.485488 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485460    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-podinfo\") pod \"49d4b521-010e-4f98-9e55-efba77ed63c0\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") "
Sep 08 16:12:01.485488 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485470    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "49d4b521-010e-4f98-9e55-efba77ed63c0" (UID: "49d4b521-010e-4f98-9e55-efba77ed63c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:12:01.485709 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485490    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-data\") pod \"49d4b521-010e-4f98-9e55-efba77ed63c0\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") "
Sep 08 16:12:01.485709 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485544    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-token\") pod \"49d4b521-010e-4f98-9e55-efba77ed63c0\" (UID: \"49d4b521-010e-4f98-9e55-efba77ed63c0\") "
Sep 08 16:12:01.485709 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485598    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "49d4b521-010e-4f98-9e55-efba77ed63c0" (UID: "49d4b521-010e-4f98-9e55-efba77ed63c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:12:01.485709 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485667    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d4b521-010e-4f98-9e55-efba77ed63c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "49d4b521-010e-4f98-9e55-efba77ed63c0" (UID: "49d4b521-010e-4f98-9e55-efba77ed63c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:12:01.485923 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "49d4b521-010e-4f98-9e55-efba77ed63c0" (UID: "49d4b521-010e-4f98-9e55-efba77ed63c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:12:01.485923 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485801    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "49d4b521-010e-4f98-9e55-efba77ed63c0" (UID: "49d4b521-010e-4f98-9e55-efba77ed63c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:12:01.485923 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485857    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-workload-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:01.485923 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485876    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d4b521-010e-4f98-9e55-efba77ed63c0-istiod-ca-cert\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:01.485923 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485890    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-workload-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:01.485923 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.485904    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-credential-socket\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:01.487307 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.487287    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "49d4b521-010e-4f98-9e55-efba77ed63c0" (UID: "49d4b521-010e-4f98-9e55-efba77ed63c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:12:01.487549 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.487532    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "49d4b521-010e-4f98-9e55-efba77ed63c0" (UID: "49d4b521-010e-4f98-9e55-efba77ed63c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 16:12:01.487777 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.487752    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d4b521-010e-4f98-9e55-efba77ed63c0-kube-api-access-5gzt9" (OuterVolumeSpecName: "kube-api-access-5gzt9") pod "49d4b521-010e-4f98-9e55-efba77ed63c0" (UID: "49d4b521-010e-4f98-9e55-efba77ed63c0"). InnerVolumeSpecName "kube-api-access-5gzt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:12:01.487863 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.487810    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "49d4b521-010e-4f98-9e55-efba77ed63c0" (UID: "49d4b521-010e-4f98-9e55-efba77ed63c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:12:01.560883 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.560857    2581 generic.go:356] "Generic (PLEG): container finished" podID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerID="c46cdef1f9ba6ae01ff0640da2b7bb769896a7e4612d682c6ba4fcaedcdca750" exitCode=0
Sep 08 16:12:01.560995 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.560914    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"
Sep 08 16:12:01.560995 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.560949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" event={"ID":"49d4b521-010e-4f98-9e55-efba77ed63c0","Type":"ContainerDied","Data":"c46cdef1f9ba6ae01ff0640da2b7bb769896a7e4612d682c6ba4fcaedcdca750"}
Sep 08 16:12:01.561069 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.561006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb" event={"ID":"49d4b521-010e-4f98-9e55-efba77ed63c0","Type":"ContainerDied","Data":"5102759b43742baaa1f7ff1c7743c9c2c6b98cd29bfeb6190fcdcab759b1dd8e"}
Sep 08 16:12:01.561069 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.561031    2581 scope.go:117] "RemoveContainer" containerID="c46cdef1f9ba6ae01ff0640da2b7bb769896a7e4612d682c6ba4fcaedcdca750"
Sep 08 16:12:01.570595 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.570578    2581 scope.go:117] "RemoveContainer" containerID="8cc99d2d69c27e72d644e8afce08b80d2ffea00857aaa6dca0ee1ad3335b9121"
Sep 08 16:12:01.578158 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.578139    2581 scope.go:117] "RemoveContainer" containerID="c46cdef1f9ba6ae01ff0640da2b7bb769896a7e4612d682c6ba4fcaedcdca750"
Sep 08 16:12:01.578424 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:12:01.578382    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46cdef1f9ba6ae01ff0640da2b7bb769896a7e4612d682c6ba4fcaedcdca750\": container with ID starting with c46cdef1f9ba6ae01ff0640da2b7bb769896a7e4612d682c6ba4fcaedcdca750 not found: ID does not exist" containerID="c46cdef1f9ba6ae01ff0640da2b7bb769896a7e4612d682c6ba4fcaedcdca750"
Sep 08 16:12:01.578501 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.578435    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46cdef1f9ba6ae01ff0640da2b7bb769896a7e4612d682c6ba4fcaedcdca750"} err="failed to get container status \"c46cdef1f9ba6ae01ff0640da2b7bb769896a7e4612d682c6ba4fcaedcdca750\": rpc error: code = NotFound desc = could not find container \"c46cdef1f9ba6ae01ff0640da2b7bb769896a7e4612d682c6ba4fcaedcdca750\": container with ID starting with c46cdef1f9ba6ae01ff0640da2b7bb769896a7e4612d682c6ba4fcaedcdca750 not found: ID does not exist"
Sep 08 16:12:01.578501 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.578459    2581 scope.go:117] "RemoveContainer" containerID="8cc99d2d69c27e72d644e8afce08b80d2ffea00857aaa6dca0ee1ad3335b9121"
Sep 08 16:12:01.578735 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:12:01.578718    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc99d2d69c27e72d644e8afce08b80d2ffea00857aaa6dca0ee1ad3335b9121\": container with ID starting with 8cc99d2d69c27e72d644e8afce08b80d2ffea00857aaa6dca0ee1ad3335b9121 not found: ID does not exist" containerID="8cc99d2d69c27e72d644e8afce08b80d2ffea00857aaa6dca0ee1ad3335b9121"
Sep 08 16:12:01.578778 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.578743    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc99d2d69c27e72d644e8afce08b80d2ffea00857aaa6dca0ee1ad3335b9121"} err="failed to get container status \"8cc99d2d69c27e72d644e8afce08b80d2ffea00857aaa6dca0ee1ad3335b9121\": rpc error: code = NotFound desc = could not find container \"8cc99d2d69c27e72d644e8afce08b80d2ffea00857aaa6dca0ee1ad3335b9121\": container with ID starting with 8cc99d2d69c27e72d644e8afce08b80d2ffea00857aaa6dca0ee1ad3335b9121 not found: ID does not exist"
Sep 08 16:12:01.582376 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.582355    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"]
Sep 08 16:12:01.586525 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.586506    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-108124-openshift-default-7db57979fb-899rb"]
Sep 08 16:12:01.586838 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.586821    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-envoy\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:01.586884 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.586842    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gzt9\" (UniqueName: \"kubernetes.io/projected/49d4b521-010e-4f98-9e55-efba77ed63c0-kube-api-access-5gzt9\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:01.586884 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.586853    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-podinfo\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:01.586884 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.586865    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-data\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:01.586884 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.586874    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d4b521-010e-4f98-9e55-efba77ed63c0-istio-token\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:01.900519 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:01.900494    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" path="/var/lib/kubelet/pods/49d4b521-010e-4f98-9e55-efba77ed63c0/volumes"
Sep 08 16:12:06.917421 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:06.917370    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"]
Sep 08 16:12:06.917847 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:06.917633    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp" podUID="e98fd537-6f64-425e-8ee1-6930ff87564c" containerName="payload-processing" containerID="cri-o://66949c58c31f55b86c4c28d9bf1047dfdfd5dcbc8a8ddfb8e8151d8022dc747d" gracePeriod=30
Sep 08 16:12:06.921336 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:06.921314    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"]
Sep 08 16:12:06.921525 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:06.921498    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv" podUID="9665892f-5c67-4c10-8263-bbde957aa053" containerName="payload-pre-processing" containerID="cri-o://f0803866168f0c89c55213d4db51c61508db94dc808e621efbdcca1f1c0a8f1b" gracePeriod=30
Sep 08 16:12:07.059593 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.059536    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz"]
Sep 08 16:12:07.063224 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.063197    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a09aa5-7z7wz"]
Sep 08 16:12:07.195843 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.195818    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"
Sep 08 16:12:07.198508 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.198492    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"
Sep 08 16:12:07.329797 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.329748    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98fd537-6f64-425e-8ee1-6930ff87564c-plugins-config-volume\") pod \"e98fd537-6f64-425e-8ee1-6930ff87564c\" (UID: \"e98fd537-6f64-425e-8ee1-6930ff87564c\") "
Sep 08 16:12:07.329977 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.329849    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd2nx\" (UniqueName: \"kubernetes.io/projected/9665892f-5c67-4c10-8263-bbde957aa053-kube-api-access-dd2nx\") pod \"9665892f-5c67-4c10-8263-bbde957aa053\" (UID: \"9665892f-5c67-4c10-8263-bbde957aa053\") "
Sep 08 16:12:07.329977 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.329888    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2pph\" (UniqueName: \"kubernetes.io/projected/e98fd537-6f64-425e-8ee1-6930ff87564c-kube-api-access-g2pph\") pod \"e98fd537-6f64-425e-8ee1-6930ff87564c\" (UID: \"e98fd537-6f64-425e-8ee1-6930ff87564c\") "
Sep 08 16:12:07.329977 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.329935    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9665892f-5c67-4c10-8263-bbde957aa053-plugins-config-volume\") pod \"9665892f-5c67-4c10-8263-bbde957aa053\" (UID: \"9665892f-5c67-4c10-8263-bbde957aa053\") "
Sep 08 16:12:07.330265 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.330195    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e98fd537-6f64-425e-8ee1-6930ff87564c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e98fd537-6f64-425e-8ee1-6930ff87564c" (UID: "e98fd537-6f64-425e-8ee1-6930ff87564c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:12:07.330429 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.330374    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9665892f-5c67-4c10-8263-bbde957aa053-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9665892f-5c67-4c10-8263-bbde957aa053" (UID: "9665892f-5c67-4c10-8263-bbde957aa053"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:12:07.331932 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.331908    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9665892f-5c67-4c10-8263-bbde957aa053-kube-api-access-dd2nx" (OuterVolumeSpecName: "kube-api-access-dd2nx") pod "9665892f-5c67-4c10-8263-bbde957aa053" (UID: "9665892f-5c67-4c10-8263-bbde957aa053"). InnerVolumeSpecName "kube-api-access-dd2nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:12:07.332022 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.331944    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98fd537-6f64-425e-8ee1-6930ff87564c-kube-api-access-g2pph" (OuterVolumeSpecName: "kube-api-access-g2pph") pod "e98fd537-6f64-425e-8ee1-6930ff87564c" (UID: "e98fd537-6f64-425e-8ee1-6930ff87564c"). InnerVolumeSpecName "kube-api-access-g2pph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:12:07.430615 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.430558    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98fd537-6f64-425e-8ee1-6930ff87564c-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:07.430615 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.430579    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd2nx\" (UniqueName: \"kubernetes.io/projected/9665892f-5c67-4c10-8263-bbde957aa053-kube-api-access-dd2nx\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:07.430615 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.430591    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2pph\" (UniqueName: \"kubernetes.io/projected/e98fd537-6f64-425e-8ee1-6930ff87564c-kube-api-access-g2pph\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:07.430615 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.430599    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9665892f-5c67-4c10-8263-bbde957aa053-plugins-config-volume\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:12:07.591094 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.591061    2581 generic.go:356] "Generic (PLEG): container finished" podID="e98fd537-6f64-425e-8ee1-6930ff87564c" containerID="66949c58c31f55b86c4c28d9bf1047dfdfd5dcbc8a8ddfb8e8151d8022dc747d" exitCode=0
Sep 08 16:12:07.591224 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.591103    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"
Sep 08 16:12:07.591224 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.591146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp" event={"ID":"e98fd537-6f64-425e-8ee1-6930ff87564c","Type":"ContainerDied","Data":"66949c58c31f55b86c4c28d9bf1047dfdfd5dcbc8a8ddfb8e8151d8022dc747d"}
Sep 08 16:12:07.591224 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.591194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp" event={"ID":"e98fd537-6f64-425e-8ee1-6930ff87564c","Type":"ContainerDied","Data":"69337d3855cfa7a019948ad4080cc05672d32cb446cfeb05fbeb197a8d26370d"}
Sep 08 16:12:07.591224 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.591223    2581 scope.go:117] "RemoveContainer" containerID="66949c58c31f55b86c4c28d9bf1047dfdfd5dcbc8a8ddfb8e8151d8022dc747d"
Sep 08 16:12:07.592570 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.592549    2581 generic.go:356] "Generic (PLEG): container finished" podID="9665892f-5c67-4c10-8263-bbde957aa053" containerID="f0803866168f0c89c55213d4db51c61508db94dc808e621efbdcca1f1c0a8f1b" exitCode=0
Sep 08 16:12:07.592694 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.592579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv" event={"ID":"9665892f-5c67-4c10-8263-bbde957aa053","Type":"ContainerDied","Data":"f0803866168f0c89c55213d4db51c61508db94dc808e621efbdcca1f1c0a8f1b"}
Sep 08 16:12:07.592694 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.592625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv" event={"ID":"9665892f-5c67-4c10-8263-bbde957aa053","Type":"ContainerDied","Data":"a837a9e888426a265339356f7f2f80a98477c01e72a5f0e06128ec323928d74c"}
Sep 08 16:12:07.592694 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.592591    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"
Sep 08 16:12:07.601421 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.601385    2581 scope.go:117] "RemoveContainer" containerID="66949c58c31f55b86c4c28d9bf1047dfdfd5dcbc8a8ddfb8e8151d8022dc747d"
Sep 08 16:12:07.601698 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:12:07.601679    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66949c58c31f55b86c4c28d9bf1047dfdfd5dcbc8a8ddfb8e8151d8022dc747d\": container with ID starting with 66949c58c31f55b86c4c28d9bf1047dfdfd5dcbc8a8ddfb8e8151d8022dc747d not found: ID does not exist" containerID="66949c58c31f55b86c4c28d9bf1047dfdfd5dcbc8a8ddfb8e8151d8022dc747d"
Sep 08 16:12:07.601763 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.601705    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66949c58c31f55b86c4c28d9bf1047dfdfd5dcbc8a8ddfb8e8151d8022dc747d"} err="failed to get container status \"66949c58c31f55b86c4c28d9bf1047dfdfd5dcbc8a8ddfb8e8151d8022dc747d\": rpc error: code = NotFound desc = could not find container \"66949c58c31f55b86c4c28d9bf1047dfdfd5dcbc8a8ddfb8e8151d8022dc747d\": container with ID starting with 66949c58c31f55b86c4c28d9bf1047dfdfd5dcbc8a8ddfb8e8151d8022dc747d not found: ID does not exist"
Sep 08 16:12:07.601763 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.601722    2581 scope.go:117] "RemoveContainer" containerID="f0803866168f0c89c55213d4db51c61508db94dc808e621efbdcca1f1c0a8f1b"
Sep 08 16:12:07.609337 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.609317    2581 scope.go:117] "RemoveContainer" containerID="f0803866168f0c89c55213d4db51c61508db94dc808e621efbdcca1f1c0a8f1b"
Sep 08 16:12:07.609588 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:12:07.609568    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0803866168f0c89c55213d4db51c61508db94dc808e621efbdcca1f1c0a8f1b\": container with ID starting with f0803866168f0c89c55213d4db51c61508db94dc808e621efbdcca1f1c0a8f1b not found: ID does not exist" containerID="f0803866168f0c89c55213d4db51c61508db94dc808e621efbdcca1f1c0a8f1b"
Sep 08 16:12:07.609640 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.609594    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0803866168f0c89c55213d4db51c61508db94dc808e621efbdcca1f1c0a8f1b"} err="failed to get container status \"f0803866168f0c89c55213d4db51c61508db94dc808e621efbdcca1f1c0a8f1b\": rpc error: code = NotFound desc = could not find container \"f0803866168f0c89c55213d4db51c61508db94dc808e621efbdcca1f1c0a8f1b\": container with ID starting with f0803866168f0c89c55213d4db51c61508db94dc808e621efbdcca1f1c0a8f1b not found: ID does not exist"
Sep 08 16:12:07.614601 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.614579    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"]
Sep 08 16:12:07.618499 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.618481    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a09aa5-7799c657d6-hltwv"]
Sep 08 16:12:07.626839 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.626820    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"]
Sep 08 16:12:07.632447 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.632428    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a09aa5-6f8bd94d75-9wplp"]
Sep 08 16:12:07.901773 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.901743    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9665892f-5c67-4c10-8263-bbde957aa053" path="/var/lib/kubelet/pods/9665892f-5c67-4c10-8263-bbde957aa053/volumes"
Sep 08 16:12:07.902079 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.902066    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd63fec1-2c15-42df-b7fe-11233be04433" path="/var/lib/kubelet/pods/dd63fec1-2c15-42df-b7fe-11233be04433/volumes"
Sep 08 16:12:07.902382 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:12:07.902370    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e98fd537-6f64-425e-8ee1-6930ff87564c" path="/var/lib/kubelet/pods/e98fd537-6f64-425e-8ee1-6930ff87564c/volumes"
Sep 08 16:14:13.968482 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:14:13.968453    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 16:14:13.970653 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:14:13.970625    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 16:15:00.129627 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.129593    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814735-4s85z"]
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.129926    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e98fd537-6f64-425e-8ee1-6930ff87564c" containerName="payload-processing"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.129937    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e98fd537-6f64-425e-8ee1-6930ff87564c" containerName="payload-processing"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.129957    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerName="istio-proxy"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.129963    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerName="istio-proxy"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.129969    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd63fec1-2c15-42df-b7fe-11233be04433" containerName="revoke-keys"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.129975    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd63fec1-2c15-42df-b7fe-11233be04433" containerName="revoke-keys"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.129982    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9665892f-5c67-4c10-8263-bbde957aa053" containerName="payload-pre-processing"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.129988    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9665892f-5c67-4c10-8263-bbde957aa053" containerName="payload-pre-processing"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.129995    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerName="istio-proxy"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.130001    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerName="istio-proxy"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.130051    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e98fd537-6f64-425e-8ee1-6930ff87564c" containerName="payload-processing"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.130058    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerName="istio-proxy"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.130064    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="49d4b521-010e-4f98-9e55-efba77ed63c0" containerName="istio-proxy"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.130070    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9665892f-5c67-4c10-8263-bbde957aa053" containerName="payload-pre-processing"
Sep 08 16:15:00.131925 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.130076    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd63fec1-2c15-42df-b7fe-11233be04433" containerName="revoke-keys"
Sep 08 16:15:00.132950 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.132934    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814735-4s85z"
Sep 08 16:15:00.135196 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.135178    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-65h2p\""
Sep 08 16:15:00.141543 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.141515    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814735-4s85z"]
Sep 08 16:15:00.143896 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.143878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814735-4s85z"
Sep 08 16:15:00.413592 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:00.413560    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814735-4s85z"]
Sep 08 16:15:00.417577 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:15:00.417546    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bb4892_e87e_4287_8781_77d5325d831f.slice/crio-678ee250b8c5924b08065d56cad65b81d5422fa09f13d94bcaa374466ea2d898 WatchSource:0}: Error finding container 678ee250b8c5924b08065d56cad65b81d5422fa09f13d94bcaa374466ea2d898: Status 404 returned error can't find the container with id 678ee250b8c5924b08065d56cad65b81d5422fa09f13d94bcaa374466ea2d898
Sep 08 16:15:01.387556 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:01.387514    2581 generic.go:356] "Generic (PLEG): container finished" podID="98bb4892-e87e-4287-8781-77d5325d831f" containerID="8fddb4357ff843d9022e88bf108f13e9e4ce9571309ab324ba6feaf5966e5e1f" exitCode=0
Sep 08 16:15:01.387930 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:01.387605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814735-4s85z" event={"ID":"98bb4892-e87e-4287-8781-77d5325d831f","Type":"ContainerDied","Data":"8fddb4357ff843d9022e88bf108f13e9e4ce9571309ab324ba6feaf5966e5e1f"}
Sep 08 16:15:01.387930 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:01.387705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814735-4s85z" event={"ID":"98bb4892-e87e-4287-8781-77d5325d831f","Type":"ContainerStarted","Data":"678ee250b8c5924b08065d56cad65b81d5422fa09f13d94bcaa374466ea2d898"}
Sep 08 16:15:02.525921 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:02.525898    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814735-4s85z"
Sep 08 16:15:03.399039 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:03.399010    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814735-4s85z"
Sep 08 16:15:03.399039 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:03.399028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814735-4s85z" event={"ID":"98bb4892-e87e-4287-8781-77d5325d831f","Type":"ContainerDied","Data":"678ee250b8c5924b08065d56cad65b81d5422fa09f13d94bcaa374466ea2d898"}
Sep 08 16:15:03.399303 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:15:03.399071    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="678ee250b8c5924b08065d56cad65b81d5422fa09f13d94bcaa374466ea2d898"
Sep 08 16:16:25.758630 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.758584    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"]
Sep 08 16:16:25.759217 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.759066    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98bb4892-e87e-4287-8781-77d5325d831f" containerName="cleanup"
Sep 08 16:16:25.759217 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.759086    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="98bb4892-e87e-4287-8781-77d5325d831f" containerName="cleanup"
Sep 08 16:16:25.759217 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.759207    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="98bb4892-e87e-4287-8781-77d5325d831f" containerName="cleanup"
Sep 08 16:16:25.762137 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.762116    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.764252 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.764222    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-483460b5-kserve-self-signed-certs\""
Sep 08 16:16:25.770229 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.770205    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"]
Sep 08 16:16:25.871664 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.871636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-home\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.871795 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.871680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-tmp-dir\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.871795 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.871698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-dshm\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.871795 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.871768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf6dx\" (UniqueName: \"kubernetes.io/projected/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-kube-api-access-sf6dx\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.871921 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.871819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-model-cache\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.871921 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.871833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-tls-certs\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.973130 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.973104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-model-cache\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.973130 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.973131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-tls-certs\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.973317 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.973151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-home\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.973317 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.973178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-tmp-dir\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.973317 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.973195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-dshm\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.973317 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.973237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf6dx\" (UniqueName: \"kubernetes.io/projected/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-kube-api-access-sf6dx\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.973575 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.973551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-model-cache\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.973638 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.973601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-home\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.973638 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.973623    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-tmp-dir\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.975375 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.975353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-dshm\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.975724 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.975709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-tls-certs\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:25.980668 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:25.980650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf6dx\" (UniqueName: \"kubernetes.io/projected/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-kube-api-access-sf6dx\") pod \"e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") " pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:26.072133 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:26.072079    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:26.341996 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:26.341920    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"]
Sep 08 16:16:26.344774 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:16:26.344748    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37fa125e_ab38_43af_9ea4_a2ab443ed4e9.slice/crio-9a5cde0f7532eeb4742f83d538a7af1f4ddf47120d7a77012895f6913b793797 WatchSource:0}: Error finding container 9a5cde0f7532eeb4742f83d538a7af1f4ddf47120d7a77012895f6913b793797: Status 404 returned error can't find the container with id 9a5cde0f7532eeb4742f83d538a7af1f4ddf47120d7a77012895f6913b793797
Sep 08 16:16:26.778484 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:26.778438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx" event={"ID":"37fa125e-ab38-43af-9ea4-a2ab443ed4e9","Type":"ContainerStarted","Data":"f3f693036d894a2d98d05785da17f357807b2f8f4f23cdde964637173a69b6cd"}
Sep 08 16:16:26.778484 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:26.778493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx" event={"ID":"37fa125e-ab38-43af-9ea4-a2ab443ed4e9","Type":"ContainerStarted","Data":"9a5cde0f7532eeb4742f83d538a7af1f4ddf47120d7a77012895f6913b793797"}
Sep 08 16:16:26.796567 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:26.796519    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx" podStartSLOduration=1.796501797 podStartE2EDuration="1.796501797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:16:26.794922745 +0000 UTC m=+1933.418615069" watchObservedRunningTime="2026-09-08 16:16:26.796501797 +0000 UTC m=+1933.420194117"
Sep 08 16:16:36.072195 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.072160    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:36.072640 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.072221    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:36.084713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.084690    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:36.270299 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.270266    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"]
Sep 08 16:16:36.273714 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.273692    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.275866 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.275846    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-483460b5-kserve-self-signed-certs\""
Sep 08 16:16:36.284651 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.284625    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"]
Sep 08 16:16:36.359078 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.359023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-tls-certs\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.359078 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.359060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2mdx\" (UniqueName: \"kubernetes.io/projected/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-kube-api-access-h2mdx\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.359210 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.359139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-model-cache\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.359210 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.359204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-tmp-dir\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.359282 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.359225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-dshm\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.359282 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.359242    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-home\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.460015 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.459993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-tmp-dir\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.460126 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.460021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-dshm\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.460126 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.460036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-home\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.460126 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.460073    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-tls-certs\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.460316 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.460186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2mdx\" (UniqueName: \"kubernetes.io/projected/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-kube-api-access-h2mdx\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.460316 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.460282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-model-cache\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.460485 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.460464    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-tmp-dir\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.460545 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.460475    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-home\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.460632 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.460615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-model-cache\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.462340 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.462319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-dshm\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.462524 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.462508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-tls-certs\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.468900 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.468878    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2mdx\" (UniqueName: \"kubernetes.io/projected/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-kube-api-access-h2mdx\") pod \"e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") " pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.584491 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.584466    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:36.839975 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.839949    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:16:36.858635 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:36.858611    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"]
Sep 08 16:16:36.860170 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:16:36.860145    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e79361b_7f8f_4263_97e6_c2a97ad0c69d.slice/crio-6a636c37670917bd07d3ec6548b0d26cb509081ff55e7d76ff6de313b1fab44e WatchSource:0}: Error finding container 6a636c37670917bd07d3ec6548b0d26cb509081ff55e7d76ff6de313b1fab44e: Status 404 returned error can't find the container with id 6a636c37670917bd07d3ec6548b0d26cb509081ff55e7d76ff6de313b1fab44e
Sep 08 16:16:37.840142 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:37.840102    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466" event={"ID":"1e79361b-7f8f-4263-97e6-c2a97ad0c69d","Type":"ContainerStarted","Data":"1ccba8dd3443a1a1d675bfd9bf7f2189c27216d5d3d885ee6b44c1a333f7af62"}
Sep 08 16:16:37.840575 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:37.840156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466" event={"ID":"1e79361b-7f8f-4263-97e6-c2a97ad0c69d","Type":"ContainerStarted","Data":"6a636c37670917bd07d3ec6548b0d26cb509081ff55e7d76ff6de313b1fab44e"}
Sep 08 16:16:37.854874 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:37.854822    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466" podStartSLOduration=1.8548055159999999 podStartE2EDuration="1.854805516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:16:37.854328871 +0000 UTC m=+1944.478021196" watchObservedRunningTime="2026-09-08 16:16:37.854805516 +0000 UTC m=+1944.478497841"
Sep 08 16:16:46.584981 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:46.584940    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:46.584981 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:46.584991    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:46.597495 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:46.597469    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:16:46.893668 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:16:46.893640    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:17:01.832273 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:01.832236    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"]
Sep 08 16:17:03.257809 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:03.257776    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"]
Sep 08 16:17:16.852110 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:16.852058    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx" podUID="37fa125e-ab38-43af-9ea4-a2ab443ed4e9" containerName="main" containerID="cri-o://f3f693036d894a2d98d05785da17f357807b2f8f4f23cdde964637173a69b6cd" gracePeriod=45
Sep 08 16:17:17.037995 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.037965    2581 generic.go:356] "Generic (PLEG): container finished" podID="37fa125e-ab38-43af-9ea4-a2ab443ed4e9" containerID="f3f693036d894a2d98d05785da17f357807b2f8f4f23cdde964637173a69b6cd" exitCode=0
Sep 08 16:17:17.038137 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.038039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx" event={"ID":"37fa125e-ab38-43af-9ea4-a2ab443ed4e9","Type":"ContainerDied","Data":"f3f693036d894a2d98d05785da17f357807b2f8f4f23cdde964637173a69b6cd"}
Sep 08 16:17:17.101217 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.101198    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:17:17.200558 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.200536    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf6dx\" (UniqueName: \"kubernetes.io/projected/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-kube-api-access-sf6dx\") pod \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") "
Sep 08 16:17:17.200713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.200571    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-model-cache\") pod \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") "
Sep 08 16:17:17.200713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.200596    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-tmp-dir\") pod \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") "
Sep 08 16:17:17.200713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.200623    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-tls-certs\") pod \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") "
Sep 08 16:17:17.200713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.200695    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-dshm\") pod \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") "
Sep 08 16:17:17.200713 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.200714    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-home\") pod \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\" (UID: \"37fa125e-ab38-43af-9ea4-a2ab443ed4e9\") "
Sep 08 16:17:17.200983 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.200862    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "37fa125e-ab38-43af-9ea4-a2ab443ed4e9" (UID: "37fa125e-ab38-43af-9ea4-a2ab443ed4e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:17:17.200983 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.200874    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37fa125e-ab38-43af-9ea4-a2ab443ed4e9" (UID: "37fa125e-ab38-43af-9ea4-a2ab443ed4e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:17:17.201097 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.201012    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-model-cache\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:17.201097 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.201039    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-tmp-dir\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:17.201097 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.201006    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-home" (OuterVolumeSpecName: "home") pod "37fa125e-ab38-43af-9ea4-a2ab443ed4e9" (UID: "37fa125e-ab38-43af-9ea4-a2ab443ed4e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:17:17.202860 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.202835    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-kube-api-access-sf6dx" (OuterVolumeSpecName: "kube-api-access-sf6dx") pod "37fa125e-ab38-43af-9ea4-a2ab443ed4e9" (UID: "37fa125e-ab38-43af-9ea4-a2ab443ed4e9"). InnerVolumeSpecName "kube-api-access-sf6dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:17:17.202927 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.202840    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37fa125e-ab38-43af-9ea4-a2ab443ed4e9" (UID: "37fa125e-ab38-43af-9ea4-a2ab443ed4e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:17:17.202927 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.202873    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-dshm" (OuterVolumeSpecName: "dshm") pod "37fa125e-ab38-43af-9ea4-a2ab443ed4e9" (UID: "37fa125e-ab38-43af-9ea4-a2ab443ed4e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:17:17.302272 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.302236    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf6dx\" (UniqueName: \"kubernetes.io/projected/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-kube-api-access-sf6dx\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:17.302272 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.302268    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-tls-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:17.302272 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.302278    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-dshm\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:17.302550 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:17.302286    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37fa125e-ab38-43af-9ea4-a2ab443ed4e9-home\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:18.044022 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.043995    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"
Sep 08 16:17:18.044459 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.044012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx" event={"ID":"37fa125e-ab38-43af-9ea4-a2ab443ed4e9","Type":"ContainerDied","Data":"9a5cde0f7532eeb4742f83d538a7af1f4ddf47120d7a77012895f6913b793797"}
Sep 08 16:17:18.044459 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.044068    2581 scope.go:117] "RemoveContainer" containerID="f3f693036d894a2d98d05785da17f357807b2f8f4f23cdde964637173a69b6cd"
Sep 08 16:17:18.059119 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.059095    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"]
Sep 08 16:17:18.065401 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.065370    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-483460b5-kserve-74dd768dc-9v4qx"]
Sep 08 16:17:18.277416 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.277308    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466" podUID="1e79361b-7f8f-4263-97e6-c2a97ad0c69d" containerName="main" containerID="cri-o://1ccba8dd3443a1a1d675bfd9bf7f2189c27216d5d3d885ee6b44c1a333f7af62" gracePeriod=45
Sep 08 16:17:18.525892 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.525865    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:17:18.612227 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.612162    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-dshm\") pod \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") "
Sep 08 16:17:18.612227 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.612205    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-tmp-dir\") pod \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") "
Sep 08 16:17:18.612227 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.612227    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-home\") pod \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") "
Sep 08 16:17:18.612522 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.612263    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-tls-certs\") pod \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") "
Sep 08 16:17:18.612522 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.612340    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2mdx\" (UniqueName: \"kubernetes.io/projected/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-kube-api-access-h2mdx\") pod \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") "
Sep 08 16:17:18.612522 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.612361    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-model-cache\") pod \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\" (UID: \"1e79361b-7f8f-4263-97e6-c2a97ad0c69d\") "
Sep 08 16:17:18.612522 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.612487    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e79361b-7f8f-4263-97e6-c2a97ad0c69d" (UID: "1e79361b-7f8f-4263-97e6-c2a97ad0c69d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:17:18.612707 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.612518    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-home" (OuterVolumeSpecName: "home") pod "1e79361b-7f8f-4263-97e6-c2a97ad0c69d" (UID: "1e79361b-7f8f-4263-97e6-c2a97ad0c69d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:17:18.612707 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.612684    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-tmp-dir\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:18.612707 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.612703    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-home\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:18.612866 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.612716    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e79361b-7f8f-4263-97e6-c2a97ad0c69d" (UID: "1e79361b-7f8f-4263-97e6-c2a97ad0c69d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:17:18.614361 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.614341    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-kube-api-access-h2mdx" (OuterVolumeSpecName: "kube-api-access-h2mdx") pod "1e79361b-7f8f-4263-97e6-c2a97ad0c69d" (UID: "1e79361b-7f8f-4263-97e6-c2a97ad0c69d"). InnerVolumeSpecName "kube-api-access-h2mdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:17:18.614454 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.614364    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-dshm" (OuterVolumeSpecName: "dshm") pod "1e79361b-7f8f-4263-97e6-c2a97ad0c69d" (UID: "1e79361b-7f8f-4263-97e6-c2a97ad0c69d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:17:18.614497 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.614456    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e79361b-7f8f-4263-97e6-c2a97ad0c69d" (UID: "1e79361b-7f8f-4263-97e6-c2a97ad0c69d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:17:18.713902 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.713874    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2mdx\" (UniqueName: \"kubernetes.io/projected/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-kube-api-access-h2mdx\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:18.713902 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.713898    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-model-cache\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:18.714046 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.713912    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-dshm\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:18.714046 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:18.713924    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e79361b-7f8f-4263-97e6-c2a97ad0c69d-tls-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:19.049977 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:19.049944    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e79361b-7f8f-4263-97e6-c2a97ad0c69d" containerID="1ccba8dd3443a1a1d675bfd9bf7f2189c27216d5d3d885ee6b44c1a333f7af62" exitCode=0
Sep 08 16:17:19.050539 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:19.049987    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"
Sep 08 16:17:19.050539 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:19.050029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466" event={"ID":"1e79361b-7f8f-4263-97e6-c2a97ad0c69d","Type":"ContainerDied","Data":"1ccba8dd3443a1a1d675bfd9bf7f2189c27216d5d3d885ee6b44c1a333f7af62"}
Sep 08 16:17:19.050539 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:19.050077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466" event={"ID":"1e79361b-7f8f-4263-97e6-c2a97ad0c69d","Type":"ContainerDied","Data":"6a636c37670917bd07d3ec6548b0d26cb509081ff55e7d76ff6de313b1fab44e"}
Sep 08 16:17:19.050539 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:19.050103    2581 scope.go:117] "RemoveContainer" containerID="1ccba8dd3443a1a1d675bfd9bf7f2189c27216d5d3d885ee6b44c1a333f7af62"
Sep 08 16:17:19.059455 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:19.059429    2581 scope.go:117] "RemoveContainer" containerID="1ccba8dd3443a1a1d675bfd9bf7f2189c27216d5d3d885ee6b44c1a333f7af62"
Sep 08 16:17:19.059761 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:17:19.059736    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccba8dd3443a1a1d675bfd9bf7f2189c27216d5d3d885ee6b44c1a333f7af62\": container with ID starting with 1ccba8dd3443a1a1d675bfd9bf7f2189c27216d5d3d885ee6b44c1a333f7af62 not found: ID does not exist" containerID="1ccba8dd3443a1a1d675bfd9bf7f2189c27216d5d3d885ee6b44c1a333f7af62"
Sep 08 16:17:19.059844 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:19.059767    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccba8dd3443a1a1d675bfd9bf7f2189c27216d5d3d885ee6b44c1a333f7af62"} err="failed to get container status \"1ccba8dd3443a1a1d675bfd9bf7f2189c27216d5d3d885ee6b44c1a333f7af62\": rpc error: code = NotFound desc = could not find container \"1ccba8dd3443a1a1d675bfd9bf7f2189c27216d5d3d885ee6b44c1a333f7af62\": container with ID starting with 1ccba8dd3443a1a1d675bfd9bf7f2189c27216d5d3d885ee6b44c1a333f7af62 not found: ID does not exist"
Sep 08 16:17:19.070519 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:19.070497    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"]
Sep 08 16:17:19.076339 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:19.076319    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-483460b5-kserve-dcb8bcc9f-7w466"]
Sep 08 16:17:19.901624 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:19.901592    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e79361b-7f8f-4263-97e6-c2a97ad0c69d" path="/var/lib/kubelet/pods/1e79361b-7f8f-4263-97e6-c2a97ad0c69d/volumes"
Sep 08 16:17:19.901982 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:19.901968    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37fa125e-ab38-43af-9ea4-a2ab443ed4e9" path="/var/lib/kubelet/pods/37fa125e-ab38-43af-9ea4-a2ab443ed4e9/volumes"
Sep 08 16:17:26.772807 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.772768    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"]
Sep 08 16:17:26.773179 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.773112    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37fa125e-ab38-43af-9ea4-a2ab443ed4e9" containerName="main"
Sep 08 16:17:26.773179 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.773124    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="37fa125e-ab38-43af-9ea4-a2ab443ed4e9" containerName="main"
Sep 08 16:17:26.773179 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.773145    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e79361b-7f8f-4263-97e6-c2a97ad0c69d" containerName="main"
Sep 08 16:17:26.773179 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.773151    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e79361b-7f8f-4263-97e6-c2a97ad0c69d" containerName="main"
Sep 08 16:17:26.773315 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.773201    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e79361b-7f8f-4263-97e6-c2a97ad0c69d" containerName="main"
Sep 08 16:17:26.773315 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.773213    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="37fa125e-ab38-43af-9ea4-a2ab443ed4e9" containerName="main"
Sep 08 16:17:26.777555 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.777529    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:26.779967 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.779941    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 16:17:26.781104 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.781083    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmx7t\""
Sep 08 16:17:26.781210 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.781089    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:17:26.781210 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.781141    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 16:17:26.781314 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.781228    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 16:17:26.786326 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.786301    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"]
Sep 08 16:17:26.788744 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.788719    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"]
Sep 08 16:17:26.792431 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.792412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:26.803770 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.803743    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"]
Sep 08 16:17:26.878387 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.878365    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7130fe-10be-463e-8b9c-f881f8d503ef-openshift-service-ca\") pod \"maas-api-5ff8b9db6-dmn5w\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") " pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:26.878529 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.878420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f33b95-95f0-472d-b895-6258b326993f-openshift-service-ca\") pod \"maas-api-5b5cf869c4-rzcnh\" (UID: \"33f33b95-95f0-472d-b895-6258b326993f\") " pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:26.878529 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.878505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvsxc\" (UniqueName: \"kubernetes.io/projected/33f33b95-95f0-472d-b895-6258b326993f-kube-api-access-fvsxc\") pod \"maas-api-5b5cf869c4-rzcnh\" (UID: \"33f33b95-95f0-472d-b895-6258b326993f\") " pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:26.878650 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.878573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7130fe-10be-463e-8b9c-f881f8d503ef-maas-api-tls\") pod \"maas-api-5ff8b9db6-dmn5w\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") " pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:26.878650 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.878614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f33b95-95f0-472d-b895-6258b326993f-metrics-certs\") pod \"maas-api-5b5cf869c4-rzcnh\" (UID: \"33f33b95-95f0-472d-b895-6258b326993f\") " pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:26.878736 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.878675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f33b95-95f0-472d-b895-6258b326993f-maas-api-tls\") pod \"maas-api-5b5cf869c4-rzcnh\" (UID: \"33f33b95-95f0-472d-b895-6258b326993f\") " pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:26.878736 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.878699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7130fe-10be-463e-8b9c-f881f8d503ef-metrics-certs\") pod \"maas-api-5ff8b9db6-dmn5w\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") " pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:26.878736 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.878728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfwqq\" (UniqueName: \"kubernetes.io/projected/ce7130fe-10be-463e-8b9c-f881f8d503ef-kube-api-access-qfwqq\") pod \"maas-api-5ff8b9db6-dmn5w\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") " pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:26.979179 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.979156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f33b95-95f0-472d-b895-6258b326993f-maas-api-tls\") pod \"maas-api-5b5cf869c4-rzcnh\" (UID: \"33f33b95-95f0-472d-b895-6258b326993f\") " pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:26.979276 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.979186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7130fe-10be-463e-8b9c-f881f8d503ef-metrics-certs\") pod \"maas-api-5ff8b9db6-dmn5w\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") " pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:26.979276 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.979209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfwqq\" (UniqueName: \"kubernetes.io/projected/ce7130fe-10be-463e-8b9c-f881f8d503ef-kube-api-access-qfwqq\") pod \"maas-api-5ff8b9db6-dmn5w\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") " pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:26.979433 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.979342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7130fe-10be-463e-8b9c-f881f8d503ef-openshift-service-ca\") pod \"maas-api-5ff8b9db6-dmn5w\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") " pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:26.979433 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.979366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f33b95-95f0-472d-b895-6258b326993f-openshift-service-ca\") pod \"maas-api-5b5cf869c4-rzcnh\" (UID: \"33f33b95-95f0-472d-b895-6258b326993f\") " pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:26.979561 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.979434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvsxc\" (UniqueName: \"kubernetes.io/projected/33f33b95-95f0-472d-b895-6258b326993f-kube-api-access-fvsxc\") pod \"maas-api-5b5cf869c4-rzcnh\" (UID: \"33f33b95-95f0-472d-b895-6258b326993f\") " pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:26.979561 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.979481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7130fe-10be-463e-8b9c-f881f8d503ef-maas-api-tls\") pod \"maas-api-5ff8b9db6-dmn5w\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") " pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:26.979561 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.979520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f33b95-95f0-472d-b895-6258b326993f-metrics-certs\") pod \"maas-api-5b5cf869c4-rzcnh\" (UID: \"33f33b95-95f0-472d-b895-6258b326993f\") " pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:26.980128 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.980103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7130fe-10be-463e-8b9c-f881f8d503ef-openshift-service-ca\") pod \"maas-api-5ff8b9db6-dmn5w\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") " pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:26.980251 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.980226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f33b95-95f0-472d-b895-6258b326993f-openshift-service-ca\") pod \"maas-api-5b5cf869c4-rzcnh\" (UID: \"33f33b95-95f0-472d-b895-6258b326993f\") " pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:26.981846 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.981816    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7130fe-10be-463e-8b9c-f881f8d503ef-metrics-certs\") pod \"maas-api-5ff8b9db6-dmn5w\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") " pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:26.981928 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.981906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7130fe-10be-463e-8b9c-f881f8d503ef-maas-api-tls\") pod \"maas-api-5ff8b9db6-dmn5w\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") " pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:26.982114 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.982098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f33b95-95f0-472d-b895-6258b326993f-maas-api-tls\") pod \"maas-api-5b5cf869c4-rzcnh\" (UID: \"33f33b95-95f0-472d-b895-6258b326993f\") " pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:26.982152 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.982120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f33b95-95f0-472d-b895-6258b326993f-metrics-certs\") pod \"maas-api-5b5cf869c4-rzcnh\" (UID: \"33f33b95-95f0-472d-b895-6258b326993f\") " pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:26.986456 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.986432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvsxc\" (UniqueName: \"kubernetes.io/projected/33f33b95-95f0-472d-b895-6258b326993f-kube-api-access-fvsxc\") pod \"maas-api-5b5cf869c4-rzcnh\" (UID: \"33f33b95-95f0-472d-b895-6258b326993f\") " pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:26.986696 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:26.986677    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfwqq\" (UniqueName: \"kubernetes.io/projected/ce7130fe-10be-463e-8b9c-f881f8d503ef-kube-api-access-qfwqq\") pod \"maas-api-5ff8b9db6-dmn5w\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") " pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:27.089275 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:27.089210    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:27.103561 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:27.103539    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:27.362520 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:27.362496    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"]
Sep 08 16:17:27.364595 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:17:27.364567    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7130fe_10be_463e_8b9c_f881f8d503ef.slice/crio-3c03ce6942fa68a6d3d73c14aa45be2ea9aa09c2c714759e4f95c911523c5aa6 WatchSource:0}: Error finding container 3c03ce6942fa68a6d3d73c14aa45be2ea9aa09c2c714759e4f95c911523c5aa6: Status 404 returned error can't find the container with id 3c03ce6942fa68a6d3d73c14aa45be2ea9aa09c2c714759e4f95c911523c5aa6
Sep 08 16:17:27.366041 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:27.366023    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:17:27.382375 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:27.382346    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"]
Sep 08 16:17:27.384757 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:17:27.384738    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f33b95_95f0_472d_b895_6258b326993f.slice/crio-1e068fb75e366416a304b21152354b39c6d5d0bb41b73c01fa0ed59f691ebf8a WatchSource:0}: Error finding container 1e068fb75e366416a304b21152354b39c6d5d0bb41b73c01fa0ed59f691ebf8a: Status 404 returned error can't find the container with id 1e068fb75e366416a304b21152354b39c6d5d0bb41b73c01fa0ed59f691ebf8a
Sep 08 16:17:28.098016 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:28.097931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh" event={"ID":"33f33b95-95f0-472d-b895-6258b326993f","Type":"ContainerStarted","Data":"fbc01b30ef232c701dc1f493fbd871bd2c161f9adf4afc68e47c147516b6af97"}
Sep 08 16:17:28.098459 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:28.098047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh" event={"ID":"33f33b95-95f0-472d-b895-6258b326993f","Type":"ContainerStarted","Data":"1e068fb75e366416a304b21152354b39c6d5d0bb41b73c01fa0ed59f691ebf8a"}
Sep 08 16:17:28.098459 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:28.098172    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:28.099243 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:28.099222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w" event={"ID":"ce7130fe-10be-463e-8b9c-f881f8d503ef","Type":"ContainerStarted","Data":"3c03ce6942fa68a6d3d73c14aa45be2ea9aa09c2c714759e4f95c911523c5aa6"}
Sep 08 16:17:28.114372 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:28.114323    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh" podStartSLOduration=1.603601484 podStartE2EDuration="2.114309725s" totalImagesPullingTime="510.708241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:28.11266362 +0000 UTC m=+1994.736355944" watchObservedRunningTime="2026-09-08 16:17:28.114309725 +0000 UTC m=+1994.738002060"
Sep 08 16:17:29.106042 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:29.106006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w" event={"ID":"ce7130fe-10be-463e-8b9c-f881f8d503ef","Type":"ContainerStarted","Data":"88f711e2243a24cd5e2dd8c83fa6c0e3cb15c706febb6a9776439c1f7050e0e3"}
Sep 08 16:17:29.106455 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:29.106105    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:29.123226 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:29.123176    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w" podStartSLOduration=2.327080916 podStartE2EDuration="3.123163313s" totalImagesPullingTime="796.082397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:29.121388243 +0000 UTC m=+1995.745080566" watchObservedRunningTime="2026-09-08 16:17:29.123163313 +0000 UTC m=+1995.746855639"
Sep 08 16:17:34.112309 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.112269    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b5cf869c4-rzcnh"
Sep 08 16:17:34.156741 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.156706    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"]
Sep 08 16:17:34.157000 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.156957    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w" podUID="ce7130fe-10be-463e-8b9c-f881f8d503ef" containerName="maas-api" containerID="cri-o://88f711e2243a24cd5e2dd8c83fa6c0e3cb15c706febb6a9776439c1f7050e0e3" gracePeriod=30
Sep 08 16:17:34.162951 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.162928    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:34.404190 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.404168    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:34.543276 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.543239    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7130fe-10be-463e-8b9c-f881f8d503ef-openshift-service-ca\") pod \"ce7130fe-10be-463e-8b9c-f881f8d503ef\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") "
Sep 08 16:17:34.543466 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.543325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7130fe-10be-463e-8b9c-f881f8d503ef-maas-api-tls\") pod \"ce7130fe-10be-463e-8b9c-f881f8d503ef\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") "
Sep 08 16:17:34.543466 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.543414    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfwqq\" (UniqueName: \"kubernetes.io/projected/ce7130fe-10be-463e-8b9c-f881f8d503ef-kube-api-access-qfwqq\") pod \"ce7130fe-10be-463e-8b9c-f881f8d503ef\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") "
Sep 08 16:17:34.543550 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.543498    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7130fe-10be-463e-8b9c-f881f8d503ef-metrics-certs\") pod \"ce7130fe-10be-463e-8b9c-f881f8d503ef\" (UID: \"ce7130fe-10be-463e-8b9c-f881f8d503ef\") "
Sep 08 16:17:34.543697 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.543673    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7130fe-10be-463e-8b9c-f881f8d503ef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ce7130fe-10be-463e-8b9c-f881f8d503ef" (UID: "ce7130fe-10be-463e-8b9c-f881f8d503ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:17:34.543800 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.543787    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7130fe-10be-463e-8b9c-f881f8d503ef-openshift-service-ca\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:34.545552 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.545527    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7130fe-10be-463e-8b9c-f881f8d503ef-kube-api-access-qfwqq" (OuterVolumeSpecName: "kube-api-access-qfwqq") pod "ce7130fe-10be-463e-8b9c-f881f8d503ef" (UID: "ce7130fe-10be-463e-8b9c-f881f8d503ef"). InnerVolumeSpecName "kube-api-access-qfwqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:17:34.545669 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.545559    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce7130fe-10be-463e-8b9c-f881f8d503ef-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ce7130fe-10be-463e-8b9c-f881f8d503ef" (UID: "ce7130fe-10be-463e-8b9c-f881f8d503ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:17:34.545721 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.545673    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce7130fe-10be-463e-8b9c-f881f8d503ef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ce7130fe-10be-463e-8b9c-f881f8d503ef" (UID: "ce7130fe-10be-463e-8b9c-f881f8d503ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:17:34.644224 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.644196    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7130fe-10be-463e-8b9c-f881f8d503ef-maas-api-tls\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:34.644224 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.644222    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfwqq\" (UniqueName: \"kubernetes.io/projected/ce7130fe-10be-463e-8b9c-f881f8d503ef-kube-api-access-qfwqq\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:34.644483 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:34.644233    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce7130fe-10be-463e-8b9c-f881f8d503ef-metrics-certs\") on node \"ip-10-0-134-107.ec2.internal\" DevicePath \"\""
Sep 08 16:17:35.136611 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:35.136581    2581 generic.go:356] "Generic (PLEG): container finished" podID="ce7130fe-10be-463e-8b9c-f881f8d503ef" containerID="88f711e2243a24cd5e2dd8c83fa6c0e3cb15c706febb6a9776439c1f7050e0e3" exitCode=0
Sep 08 16:17:35.137036 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:35.136622    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"
Sep 08 16:17:35.137036 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:35.136669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w" event={"ID":"ce7130fe-10be-463e-8b9c-f881f8d503ef","Type":"ContainerDied","Data":"88f711e2243a24cd5e2dd8c83fa6c0e3cb15c706febb6a9776439c1f7050e0e3"}
Sep 08 16:17:35.137036 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:35.136732    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w" event={"ID":"ce7130fe-10be-463e-8b9c-f881f8d503ef","Type":"ContainerDied","Data":"3c03ce6942fa68a6d3d73c14aa45be2ea9aa09c2c714759e4f95c911523c5aa6"}
Sep 08 16:17:35.137036 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:35.136764    2581 scope.go:117] "RemoveContainer" containerID="88f711e2243a24cd5e2dd8c83fa6c0e3cb15c706febb6a9776439c1f7050e0e3"
Sep 08 16:17:35.146046 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:35.146030    2581 scope.go:117] "RemoveContainer" containerID="88f711e2243a24cd5e2dd8c83fa6c0e3cb15c706febb6a9776439c1f7050e0e3"
Sep 08 16:17:35.146287 ip-10-0-134-107 kubenswrapper[2581]: E0908 16:17:35.146266    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f711e2243a24cd5e2dd8c83fa6c0e3cb15c706febb6a9776439c1f7050e0e3\": container with ID starting with 88f711e2243a24cd5e2dd8c83fa6c0e3cb15c706febb6a9776439c1f7050e0e3 not found: ID does not exist" containerID="88f711e2243a24cd5e2dd8c83fa6c0e3cb15c706febb6a9776439c1f7050e0e3"
Sep 08 16:17:35.146353 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:35.146299    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f711e2243a24cd5e2dd8c83fa6c0e3cb15c706febb6a9776439c1f7050e0e3"} err="failed to get container status \"88f711e2243a24cd5e2dd8c83fa6c0e3cb15c706febb6a9776439c1f7050e0e3\": rpc error: code = NotFound desc = could not find container \"88f711e2243a24cd5e2dd8c83fa6c0e3cb15c706febb6a9776439c1f7050e0e3\": container with ID starting with 88f711e2243a24cd5e2dd8c83fa6c0e3cb15c706febb6a9776439c1f7050e0e3 not found: ID does not exist"
Sep 08 16:17:35.157632 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:35.157608    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"]
Sep 08 16:17:35.161608 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:35.161590    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ff8b9db6-dmn5w"]
Sep 08 16:17:35.900458 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:35.900425    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce7130fe-10be-463e-8b9c-f881f8d503ef" path="/var/lib/kubelet/pods/ce7130fe-10be-463e-8b9c-f881f8d503ef/volumes"
Sep 08 16:17:56.650798 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:17:56.650755    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b89456cc-lj4m5_9293816e-2555-4268-9e94-d87394144cd3/authorino/0.log"
Sep 08 16:18:02.469145 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:02.469119    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d4b5866d6-b5dpm_2b2fc001-999d-4c56-8fde-828543a20324/copy-manifests/0.log"
Sep 08 16:18:02.474716 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:02.474686    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d4b5866d6-b5dpm_2b2fc001-999d-4c56-8fde-828543a20324/manager/0.log"
Sep 08 16:18:03.052895 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:03.052870    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5b5cf869c4-rzcnh_33f33b95-95f0-472d-b895-6258b326993f/maas-api/0.log"
Sep 08 16:18:03.376106 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:03.376027    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814735-4s85z_98bb4892-e87e-4287-8781-77d5325d831f/cleanup/0.log"
Sep 08 16:18:04.257437 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:04.257409    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b89456cc-lj4m5_9293816e-2555-4268-9e94-d87394144cd3/authorino/0.log"
Sep 08 16:18:04.369953 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:04.369928    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-m279r_451adf51-dc13-41b2-b0cd-bbce32e8a2d2/manager/0.log"
Sep 08 16:18:05.889277 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:05.889242    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6qm57_3ac0e430-ceb6-4fd8-aac3-db1b5911ba65/payload-pre-processing/0.log"
Sep 08 16:18:06.124072 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:06.124047    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f6c69889d-c7hpw_73240d9c-1436-48bd-90b2-a665d22d7006/router/0.log"
Sep 08 16:18:06.696214 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:06.696187    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-h5z9h_2c052f9f-6efe-4788-88e5-9ed2318a3cbb/main/0.log"
Sep 08 16:18:07.048312 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:07.048239    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-xmv6x_e599441e-6561-4eb4-a54b-6f49fccd80c2/main/0.log"
Sep 08 16:18:12.368774 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.368745    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jllsk/must-gather-8v8mk"]
Sep 08 16:18:12.369136 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.369075    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce7130fe-10be-463e-8b9c-f881f8d503ef" containerName="maas-api"
Sep 08 16:18:12.369136 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.369086    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce7130fe-10be-463e-8b9c-f881f8d503ef" containerName="maas-api"
Sep 08 16:18:12.369209 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.369157    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce7130fe-10be-463e-8b9c-f881f8d503ef" containerName="maas-api"
Sep 08 16:18:12.372641 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.372621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jllsk/must-gather-8v8mk"
Sep 08 16:18:12.375172 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.375146    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jllsk\"/\"kube-root-ca.crt\""
Sep 08 16:18:12.376019 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.376000    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jllsk\"/\"openshift-service-ca.crt\""
Sep 08 16:18:12.376114 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.376022    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jllsk\"/\"default-dockercfg-m9wgw\""
Sep 08 16:18:12.381538 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.381518    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jllsk/must-gather-8v8mk"]
Sep 08 16:18:12.453373 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.453350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn892\" (UniqueName: \"kubernetes.io/projected/308d561f-03c5-425c-a23c-b61c3ab63010-kube-api-access-tn892\") pod \"must-gather-8v8mk\" (UID: \"308d561f-03c5-425c-a23c-b61c3ab63010\") " pod="openshift-must-gather-jllsk/must-gather-8v8mk"
Sep 08 16:18:12.453486 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.453462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/308d561f-03c5-425c-a23c-b61c3ab63010-must-gather-output\") pod \"must-gather-8v8mk\" (UID: \"308d561f-03c5-425c-a23c-b61c3ab63010\") " pod="openshift-must-gather-jllsk/must-gather-8v8mk"
Sep 08 16:18:12.554180 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.554157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn892\" (UniqueName: \"kubernetes.io/projected/308d561f-03c5-425c-a23c-b61c3ab63010-kube-api-access-tn892\") pod \"must-gather-8v8mk\" (UID: \"308d561f-03c5-425c-a23c-b61c3ab63010\") " pod="openshift-must-gather-jllsk/must-gather-8v8mk"
Sep 08 16:18:12.554258 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.554221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/308d561f-03c5-425c-a23c-b61c3ab63010-must-gather-output\") pod \"must-gather-8v8mk\" (UID: \"308d561f-03c5-425c-a23c-b61c3ab63010\") " pod="openshift-must-gather-jllsk/must-gather-8v8mk"
Sep 08 16:18:12.554529 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.554513    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/308d561f-03c5-425c-a23c-b61c3ab63010-must-gather-output\") pod \"must-gather-8v8mk\" (UID: \"308d561f-03c5-425c-a23c-b61c3ab63010\") " pod="openshift-must-gather-jllsk/must-gather-8v8mk"
Sep 08 16:18:12.563455 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.563422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn892\" (UniqueName: \"kubernetes.io/projected/308d561f-03c5-425c-a23c-b61c3ab63010-kube-api-access-tn892\") pod \"must-gather-8v8mk\" (UID: \"308d561f-03c5-425c-a23c-b61c3ab63010\") " pod="openshift-must-gather-jllsk/must-gather-8v8mk"
Sep 08 16:18:12.682497 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.682475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jllsk/must-gather-8v8mk"
Sep 08 16:18:12.955781 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:12.955756    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jllsk/must-gather-8v8mk"]
Sep 08 16:18:12.958479 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:18:12.958447    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod308d561f_03c5_425c_a23c_b61c3ab63010.slice/crio-888fe9333bce5fd9dbf551a9b5c254eea2492e829e7ceb5241e8cbfe38771c08 WatchSource:0}: Error finding container 888fe9333bce5fd9dbf551a9b5c254eea2492e829e7ceb5241e8cbfe38771c08: Status 404 returned error can't find the container with id 888fe9333bce5fd9dbf551a9b5c254eea2492e829e7ceb5241e8cbfe38771c08
Sep 08 16:18:13.320376 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:13.320300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jllsk/must-gather-8v8mk" event={"ID":"308d561f-03c5-425c-a23c-b61c3ab63010","Type":"ContainerStarted","Data":"888fe9333bce5fd9dbf551a9b5c254eea2492e829e7ceb5241e8cbfe38771c08"}
Sep 08 16:18:14.312111 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:14.312072    2581 scope.go:117] "RemoveContainer" containerID="3fa3c893f0e5ae8e9f36b3abb3b8026fb359eebcb9e68e6d1706ffbd8dffd86f"
Sep 08 16:18:15.333996 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:15.333952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jllsk/must-gather-8v8mk" event={"ID":"308d561f-03c5-425c-a23c-b61c3ab63010","Type":"ContainerStarted","Data":"ac4416c69fdf72b1439db0659913e6989fa8bf6d7a0f7a7298089ffd1cc9ebad"}
Sep 08 16:18:15.334414 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:15.334005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jllsk/must-gather-8v8mk" event={"ID":"308d561f-03c5-425c-a23c-b61c3ab63010","Type":"ContainerStarted","Data":"ee2f8fe041a59fd40514da199d25f1e78ca6b0afbcbde49630bbcdcd448d7f24"}
Sep 08 16:18:15.350687 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:15.350641    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jllsk/must-gather-8v8mk" podStartSLOduration=1.8838685960000001 podStartE2EDuration="3.350626839s" totalImagesPullingTime="1.466758243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:18:15.349506819 +0000 UTC m=+2041.973199144" watchObservedRunningTime="2026-09-08 16:18:15.350626839 +0000 UTC m=+2041.974319162"
Sep 08 16:18:16.029987 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:16.029959    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xbnhj_e92972bf-2b13-43de-bc6c-ccf50050fa24/global-pull-secret-syncer/0.log"
Sep 08 16:18:16.100657 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:16.100619    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-thsrk_4ed950cd-31e9-4a85-89ac-83818614ad83/konnectivity-agent/0.log"
Sep 08 16:18:16.157541 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:16.157508    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-107.ec2.internal_5032d201ce1921952e1262cfc0d74711/haproxy/0.log"
Sep 08 16:18:20.834127 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:20.834065    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b89456cc-lj4m5_9293816e-2555-4268-9e94-d87394144cd3/authorino/0.log"
Sep 08 16:18:20.861592 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:20.861521    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-m279r_451adf51-dc13-41b2-b0cd-bbce32e8a2d2/manager/0.log"
Sep 08 16:18:22.820212 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:22.820172    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-n5fbl_280e0b1c-6fa9-4d0a-8769-8589179684a3/cluster-monitoring-operator/0.log"
Sep 08 16:18:23.100360 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:23.100272    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r529b_9573799b-c0da-416e-a64a-6bbcf1344ee4/node-exporter/0.log"
Sep 08 16:18:23.125123 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:23.125088    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r529b_9573799b-c0da-416e-a64a-6bbcf1344ee4/kube-rbac-proxy/0.log"
Sep 08 16:18:23.155102 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:23.155064    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r529b_9573799b-c0da-416e-a64a-6bbcf1344ee4/init-textfile/0.log"
Sep 08 16:18:25.058373 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:25.058342    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-jjj7p_7ff830b7-be75-496c-8e4e-bf04e35912ee/networking-console-plugin/0.log"
Sep 08 16:18:25.683281 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:25.683254    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/1.log"
Sep 08 16:18:25.690418 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:25.690373    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fzq5z_a27cac25-7d24-41e2-bb79-a2412491978b/console-operator/2.log"
Sep 08 16:18:26.182180 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.182151    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66f7cf5496-w8l6k_026b9f7a-72c6-4eb7-b8fb-e50d2d72788e/console/0.log"
Sep 08 16:18:26.212577 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.212540    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-95ztd_624ad5f8-22ca-4629-aba5-214f01af0f7d/download-server/0.log"
Sep 08 16:18:26.608378 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.608272    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"]
Sep 08 16:18:26.616307 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.616266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.622649 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.622618    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"]
Sep 08 16:18:26.716578 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.716539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50974b4a-ad7d-4fed-a20f-5761874611ca-proc\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.716895 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.716873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50974b4a-ad7d-4fed-a20f-5761874611ca-sys\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.717055 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.717037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50974b4a-ad7d-4fed-a20f-5761874611ca-podres\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.717202 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.717187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9n44\" (UniqueName: \"kubernetes.io/projected/50974b4a-ad7d-4fed-a20f-5761874611ca-kube-api-access-c9n44\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.717326 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.717305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50974b4a-ad7d-4fed-a20f-5761874611ca-lib-modules\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.817953 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.817915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50974b4a-ad7d-4fed-a20f-5761874611ca-podres\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.818216 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.818185    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50974b4a-ad7d-4fed-a20f-5761874611ca-podres\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.818381 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.818241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9n44\" (UniqueName: \"kubernetes.io/projected/50974b4a-ad7d-4fed-a20f-5761874611ca-kube-api-access-c9n44\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.818545 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.818528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50974b4a-ad7d-4fed-a20f-5761874611ca-lib-modules\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.818686 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.818672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50974b4a-ad7d-4fed-a20f-5761874611ca-proc\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.818828 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.818815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50974b4a-ad7d-4fed-a20f-5761874611ca-sys\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.818989 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.818734    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50974b4a-ad7d-4fed-a20f-5761874611ca-proc\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.819129 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.818693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50974b4a-ad7d-4fed-a20f-5761874611ca-lib-modules\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.819228 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.818882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50974b4a-ad7d-4fed-a20f-5761874611ca-sys\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.829155 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.829130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9n44\" (UniqueName: \"kubernetes.io/projected/50974b4a-ad7d-4fed-a20f-5761874611ca-kube-api-access-c9n44\") pod \"perf-node-gather-daemonset-llknq\" (UID: \"50974b4a-ad7d-4fed-a20f-5761874611ca\") " pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:26.909526 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.909495    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-z67qz_c30ad658-852c-412f-af05-83940b0551b2/volume-data-source-validator/0.log"
Sep 08 16:18:26.929269 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:26.929234    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:27.231117 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:27.231025    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"]
Sep 08 16:18:27.234496 ip-10-0-134-107 kubenswrapper[2581]: W0908 16:18:27.234469    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod50974b4a_ad7d_4fed_a20f_5761874611ca.slice/crio-2b187b16b194b890fc6d09bc9c31b1d5efc0d6c1396d7d0143f275d5d8c7d112 WatchSource:0}: Error finding container 2b187b16b194b890fc6d09bc9c31b1d5efc0d6c1396d7d0143f275d5d8c7d112: Status 404 returned error can't find the container with id 2b187b16b194b890fc6d09bc9c31b1d5efc0d6c1396d7d0143f275d5d8c7d112
Sep 08 16:18:27.423070 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:27.423035    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq" event={"ID":"50974b4a-ad7d-4fed-a20f-5761874611ca","Type":"ContainerStarted","Data":"0b082f01143adfe89a326398fe3702654353c82a3aef82ce6905315d371aacc2"}
Sep 08 16:18:27.423363 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:27.423335    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:27.423509 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:27.423376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq" event={"ID":"50974b4a-ad7d-4fed-a20f-5761874611ca","Type":"ContainerStarted","Data":"2b187b16b194b890fc6d09bc9c31b1d5efc0d6c1396d7d0143f275d5d8c7d112"}
Sep 08 16:18:27.438336 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:27.438289    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq" podStartSLOduration=1.438275293 podStartE2EDuration="1.438275293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:18:27.437594721 +0000 UTC m=+2054.061287045" watchObservedRunningTime="2026-09-08 16:18:27.438275293 +0000 UTC m=+2054.061967617"
Sep 08 16:18:27.970023 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:27.969994    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6vlx8_b43463d7-de74-46e8-af40-2db7ab47f8ff/dns/0.log"
Sep 08 16:18:27.987556 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:27.987523    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6vlx8_b43463d7-de74-46e8-af40-2db7ab47f8ff/kube-rbac-proxy/0.log"
Sep 08 16:18:28.125174 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:28.125150    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w4nwh_20a23223-f446-4941-9b20-6badf3aee2e3/dns-node-resolver/0.log"
Sep 08 16:18:28.638516 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:28.638486    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l6lrc_2201c616-9356-494d-915c-edee0306a6bf/node-ca/0.log"
Sep 08 16:18:29.888800 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:29.888770    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6qm57_3ac0e430-ceb6-4fd8-aac3-db1b5911ba65/payload-pre-processing/0.log"
Sep 08 16:18:29.957062 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:29.957033    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f6c69889d-c7hpw_73240d9c-1436-48bd-90b2-a665d22d7006/router/0.log"
Sep 08 16:18:30.784873 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:30.784841    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-62rfg_b3c368f6-fe5b-4cd8-8d03-0c75a4478a19/serve-healthcheck-canary/0.log"
Sep 08 16:18:31.268175 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:31.268144    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-nxt9p_b6a25233-dc78-40bb-9a74-25f3aae52b3c/insights-operator/1.log"
Sep 08 16:18:31.269513 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:31.269484    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-nxt9p_b6a25233-dc78-40bb-9a74-25f3aae52b3c/insights-operator/0.log"
Sep 08 16:18:31.349341 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:31.349315    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qvc9_c82d6a57-745e-47b9-afd5-b5c466bdf5d6/kube-rbac-proxy/0.log"
Sep 08 16:18:31.368241 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:31.368211    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qvc9_c82d6a57-745e-47b9-afd5-b5c466bdf5d6/exporter/0.log"
Sep 08 16:18:31.389588 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:31.389556    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qvc9_c82d6a57-745e-47b9-afd5-b5c466bdf5d6/extractor/0.log"
Sep 08 16:18:33.441247 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:33.441219    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jllsk/perf-node-gather-daemonset-llknq"
Sep 08 16:18:33.818114 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:33.818028    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d4b5866d6-b5dpm_2b2fc001-999d-4c56-8fde-828543a20324/manager/0.log"
Sep 08 16:18:33.839141 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:33.839111    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d4b5866d6-b5dpm_2b2fc001-999d-4c56-8fde-828543a20324/copy-manifests/0.log"
Sep 08 16:18:35.170456 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:35.170423    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-578d6cf975-h7ppl_d6e2519b-a925-4dbb-9383-2ab9454866bb/manager/0.log"
Sep 08 16:18:36.211005 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:36.210977    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6qm57_3ac0e430-ceb6-4fd8-aac3-db1b5911ba65/payload-pre-processing/0.log"
Sep 08 16:18:36.268777 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:36.268713    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f6c69889d-c7hpw_73240d9c-1436-48bd-90b2-a665d22d7006/router/0.log"
Sep 08 16:18:38.127915 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:38.127831    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6qm57_3ac0e430-ceb6-4fd8-aac3-db1b5911ba65/payload-pre-processing/0.log"
Sep 08 16:18:38.184371 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:38.184340    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f6c69889d-c7hpw_73240d9c-1436-48bd-90b2-a665d22d7006/router/0.log"
Sep 08 16:18:40.280851 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:40.280824    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-q5cth_a588199a-f0f0-476f-8806-b158e9539913/migrator/0.log"
Sep 08 16:18:40.299191 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:40.299167    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-q5cth_a588199a-f0f0-476f-8806-b158e9539913/graceful-termination/0.log"
Sep 08 16:18:40.622789 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:40.622695    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-vdxpx_593f6dab-17b1-48a1-a4bb-a9164f694d90/kube-storage-version-migrator-operator/1.log"
Sep 08 16:18:40.623592 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:40.623568    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-vdxpx_593f6dab-17b1-48a1-a4bb-a9164f694d90/kube-storage-version-migrator-operator/0.log"
Sep 08 16:18:41.786999 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:41.786972    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ph9tx_65d13726-37f1-465a-bbd5-1ebf50e5ead8/kube-multus-additional-cni-plugins/0.log"
Sep 08 16:18:41.805332 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:41.805310    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ph9tx_65d13726-37f1-465a-bbd5-1ebf50e5ead8/egress-router-binary-copy/0.log"
Sep 08 16:18:41.822303 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:41.822283    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ph9tx_65d13726-37f1-465a-bbd5-1ebf50e5ead8/cni-plugins/0.log"
Sep 08 16:18:41.839093 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:41.839075    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ph9tx_65d13726-37f1-465a-bbd5-1ebf50e5ead8/bond-cni-plugin/0.log"
Sep 08 16:18:41.856620 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:41.856604    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ph9tx_65d13726-37f1-465a-bbd5-1ebf50e5ead8/routeoverride-cni/0.log"
Sep 08 16:18:41.876983 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:41.876965    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ph9tx_65d13726-37f1-465a-bbd5-1ebf50e5ead8/whereabouts-cni-bincopy/0.log"
Sep 08 16:18:41.893584 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:41.893565    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ph9tx_65d13726-37f1-465a-bbd5-1ebf50e5ead8/whereabouts-cni/0.log"
Sep 08 16:18:41.939550 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:41.939526    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jbqc7_d6b5446c-990e-467e-a319-150edf10bcc1/kube-multus/0.log"
Sep 08 16:18:42.079717 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:42.079622    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nkn7w_996123f7-ff5c-4ffd-8a4d-ebfb862d664e/network-metrics-daemon/0.log"
Sep 08 16:18:42.092936 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:42.092898    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nkn7w_996123f7-ff5c-4ffd-8a4d-ebfb862d664e/kube-rbac-proxy/0.log"
Sep 08 16:18:43.330889 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:43.330858    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2wc2_44035804-234e-4b1f-9f69-938469b3446f/ovn-controller/0.log"
Sep 08 16:18:43.344648 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:43.344627    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2wc2_44035804-234e-4b1f-9f69-938469b3446f/ovn-acl-logging/0.log"
Sep 08 16:18:43.363876 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:43.363819    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2wc2_44035804-234e-4b1f-9f69-938469b3446f/kube-rbac-proxy-node/0.log"
Sep 08 16:18:43.383150 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:43.383121    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2wc2_44035804-234e-4b1f-9f69-938469b3446f/kube-rbac-proxy-ovn-metrics/0.log"
Sep 08 16:18:43.399886 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:43.399857    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2wc2_44035804-234e-4b1f-9f69-938469b3446f/northd/0.log"
Sep 08 16:18:43.417945 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:43.417917    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2wc2_44035804-234e-4b1f-9f69-938469b3446f/nbdb/0.log"
Sep 08 16:18:43.436272 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:43.436246    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2wc2_44035804-234e-4b1f-9f69-938469b3446f/sbdb/0.log"
Sep 08 16:18:43.516407 ip-10-0-134-107 kubenswrapper[2581]: I0908 16:18:43.516358    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2wc2_44035804-234e-4b1f-9f69-938469b3446f/ovnkube-controller/0.log"