Sep 10 15:15:36.131997 ip-10-0-134-115 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 15:15:36.558199 ip-10-0-134-115 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:15:36.558199 ip-10-0-134-115 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 15:15:36.558199 ip-10-0-134-115 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:15:36.558199 ip-10-0-134-115 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 15:15:36.558199 ip-10-0-134-115 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:15:36.558859 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.558195    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 15:15:36.562657 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562626    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:15:36.562657 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562647    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:15:36.562657 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562651    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:15:36.562657 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562654    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:15:36.562657 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562659    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:15:36.562657 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562663    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:15:36.562657 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562666    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562669    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562674    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562678    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562681    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562685    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562688    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562691    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562693    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562696    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562700    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562718    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562722    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562725    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562728    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562731    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562733    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562736    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562738    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:15:36.562926 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562741    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562747    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562750    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562753    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562755    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562758    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562761    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562763    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562766    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562768    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562773    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562776    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562779    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562782    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562785    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562788    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562791    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562793    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562796    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:15:36.563393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562799    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562801    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562804    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562807    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562809    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562812    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562815    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562817    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562819    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562822    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562825    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562827    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562830    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562833    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562836    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562838    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562841    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562843    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562846    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:15:36.563916 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562850    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562852    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562855    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562857    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562860    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562863    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562865    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562868    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562871    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562874    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562876    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562879    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562881    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562884    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562886    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562889    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562891    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562894    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562896    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562899    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:15:36.564382 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562903    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562906    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562908    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562910    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562913    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562915    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562918    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562921    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562924    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562927    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562930    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562932    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562935    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562938    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562941    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:15:36.564873 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.562943    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:15:36.565338 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565325    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:15:36.565338 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565337    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565340    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565343    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565346    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565349    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565351    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565354    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565356    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565359    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565361    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565364    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565366    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565369    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565371    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565374    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565376    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565383    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565385    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565388    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565390    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:15:36.565392 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565393    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565395    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565398    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565401    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565404    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565407    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565409    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565412    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565414    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565417    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565419    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565422    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565425    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565427    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565430    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565432    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565435    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565438    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565440    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:15:36.565902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565444    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565448    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565451    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565453    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565456    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565458    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565461    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565463    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565466    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565468    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565471    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565474    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565477    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565480    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565482    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565486    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565490    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565493    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565495    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:15:36.566399 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565498    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565500    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565503    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565506    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565509    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565511    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565514    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565516    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565519    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565521    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565524    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565527    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565529    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565532    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565535    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565537    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565539    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565542    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565544    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565548    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:15:36.566878 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565550    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565553    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565555    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565558    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565561    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565565    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565569    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565572    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565575    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565578    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565580    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565583    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565585    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565588    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565590    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565593    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565595    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565599    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565603    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:15:36.567376 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.565605    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566009    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566015    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566018    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566025    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566028    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566032    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566035    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566038    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566040    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566043    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566045    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566048    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566051    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566053    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566056    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566058    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566066    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566069    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566071    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:15:36.567855 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566074    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566077    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566080    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566083    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566085    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566088    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566090    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566093    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566095    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566098    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566100    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566103    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566105    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566108    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566110    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566114    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566116    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566119    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566121    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566124    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:15:36.568335 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566126    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566129    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566131    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566133    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566136    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566139    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566142    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566144    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566147    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566149    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566157    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566160    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566162    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566165    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566168    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566170    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566173    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566175    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566178    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:15:36.568931 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566180    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566183    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566185    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566188    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566190    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566193    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566195    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566198    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566200    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566203    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566206    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566210    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566212    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566215    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566219    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566222    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566225    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566228    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566231    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:15:36.569404 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566234    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566237    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566239    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566243    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566246    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566258    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566262    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566265    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566268    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566271    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566273    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566276    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566278    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566280    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566283    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566286    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566288    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566291    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566293    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566296    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:15:36.569882 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566298    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.566301    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566373    2573 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566406    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566414    2573 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566418    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566422    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566425    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566429    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566433    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566436    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566439    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566442    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566445    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566448    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566451    2573 flags.go:64] FLAG: --cgroup-root=""
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566454    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566457    2573 flags.go:64] FLAG: --client-ca-file=""
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566460    2573 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566463    2573 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566467    2573 flags.go:64] FLAG: --cluster-domain=""
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566469    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566472    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566475    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 15:15:36.570367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566479    2573 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566482    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566485    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566488    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566491    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566493    2573 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566496    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566499    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566502    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566505    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566509    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566512    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566515    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566518    2573 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566520    2573 flags.go:64] FLAG: --enable-server="true"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566523    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566528    2573 flags.go:64] FLAG: --event-burst="100"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566531    2573 flags.go:64] FLAG: --event-qps="50"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566533    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566536    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566539    2573 flags.go:64] FLAG: --eviction-hard=""
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566543    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566546    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566549    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566552    2573 flags.go:64] FLAG: --eviction-soft=""
Sep 10 15:15:36.570957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566555    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566558    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566561    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566564    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566567    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566570    2573 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566572    2573 flags.go:64] FLAG: --feature-gates=""
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566576    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566579    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566582    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566584    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566587    2573 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566591    2573 flags.go:64] FLAG: --help="false"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566593    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-134-115.ec2.internal"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566596    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566599    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566602    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566605    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566609    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566611    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566614    2573 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566619    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566622    2573 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 15:15:36.571569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566625    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566628    2573 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566630    2573 flags.go:64] FLAG: --kube-reserved=""
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566633    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566636    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566639    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566641    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566644    2573 flags.go:64] FLAG: --lock-file=""
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566647    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566650    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566654    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566659    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566662    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566665    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566668    2573 flags.go:64] FLAG: --logging-format="text"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566671    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566674    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566676    2573 flags.go:64] FLAG: --manifest-url=""
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566679    2573 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566686    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566689    2573 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566693    2573 flags.go:64] FLAG: --max-pods="110"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566696    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566699    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566702    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 15:15:36.572212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566718    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566721    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566724    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566727    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566734    2573 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566739    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566742    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566745    2573 flags.go:64] FLAG: --pod-cidr=""
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566748    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566753    2573 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566755    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566758    2573 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566761    2573 flags.go:64] FLAG: --port="10250"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566764    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566767    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ebc9bd3972bc40de"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566770    2573 flags.go:64] FLAG: --qos-reserved=""
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566773    2573 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566778    2573 flags.go:64] FLAG: --register-node="true"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566781    2573 flags.go:64] FLAG: --register-with-taints=""
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566784    2573 flags.go:64] FLAG: --registry-burst="10"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566788    2573 flags.go:64] FLAG: --registry-qps="5"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566790    2573 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566793    2573 flags.go:64] FLAG: --reserved-memory=""
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566797    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566800    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 15:15:36.572836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566803    2573 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566806    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566808    2573 flags.go:64] FLAG: --runonce="false"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566811    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566814    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566817    2573 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566820    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566823    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566825    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566828    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566831    2573 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566834    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566837    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566840    2573 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566843    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566846    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566849    2573 flags.go:64] FLAG: --system-cgroups=""
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566852    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566857    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566860    2573 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566862    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566866    2573 flags.go:64] FLAG: --tls-min-version=""
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566869    2573 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566872    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566876    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 15:15:36.573443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566879    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566882    2573 flags.go:64] FLAG: --v="2"
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566886    2573 flags.go:64] FLAG: --version="false"
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566891    2573 flags.go:64] FLAG: --vmodule=""
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566895    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.566898    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567014    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567026    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567030    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567033    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567036    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567039    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567042    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567045    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567048    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567050    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567053    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567056    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567059    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567062    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567066    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:15:36.574065 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567068    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567071    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567074    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567077    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567079    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567082    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567085    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567087    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567089    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567092    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567095    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567098    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567100    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567103    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567106    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567109    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567112    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567114    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567117    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567119    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:15:36.574572 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567122    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567126    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567130    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567133    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567135    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567138    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567141    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567143    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567145    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567148    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567150    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567154    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567156    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567159    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567161    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567164    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567166    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567169    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:15:36.575079 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567172    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567176    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567179    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567181    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567186    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567190    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567194    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567196    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567199    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567202    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567206    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567209    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567212    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567215    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567217    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567220    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567223    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567226    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567228    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567231    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:15:36.575548 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567233    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567235    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567238    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567240    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567243    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567247    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567249    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567252    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567254    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567257    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567259    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567262    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567265    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567267    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567270    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567273    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567276    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567279    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567281    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567284    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:15:36.576049 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567286    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:15:36.576530 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567289    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:15:36.576530 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567292    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:15:36.576530 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567296    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:15:36.576530 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567298    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:15:36.576530 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.567301    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:15:36.576530 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.567971    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:15:36.584899 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.584876    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 15:15:36.584899 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.584899    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 15:15:36.585767 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.585757    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 15:15:36.585805 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.585768    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 15:15:36.585845 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585836    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:15:36.585845 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585845    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585848    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585851    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585855    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585857    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585860    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585863    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585865    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585868    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585871    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585874    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585876    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585880    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585884    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585887    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585890    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585893    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585896    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585899    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:15:36.585905 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585902    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585905    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585908    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585911    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585913    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585916    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585920    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585922    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585925    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585934    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585936    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585939    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585941    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585943    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585946    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585948    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585951    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585953    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585955    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585958    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:15:36.586374 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585960    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585962    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585965    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585967    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585970    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585972    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585975    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585978    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585980    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585984    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585987    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585990    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585993    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585996    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.585999    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586002    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586005    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586007    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586010    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:15:36.586868 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586012    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586014    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586017    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586025    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586027    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586031    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586033    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586037    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586040    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586042    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586045    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586047    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586050    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586053    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586055    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586058    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586060    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586063    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586065    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:15:36.587337 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586068    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586070    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586073    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586075    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586078    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586080    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586083    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586085    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586088    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586091    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586094    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586096    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586099    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586101    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586104    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586106    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586109    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586117    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586119    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586122    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:15:36.587853 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586124    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.586130    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586271    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586276    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586279    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586281    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586284    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586286    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586289    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586291    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586294    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586296    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586299    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586301    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586304    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:15:36.588344 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586306    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586308    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586311    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586313    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586317    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586321    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586324    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586327    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586330    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586333    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586335    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586338    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586340    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586343    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586345    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586353    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586357    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586359    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586361    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586364    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:15:36.588917 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586366    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586369    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586371    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586374    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586376    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586379    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586382    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586384    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586387    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586389    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586392    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586394    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586397    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586399    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586401    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586404    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586406    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586409    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586412    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586414    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:15:36.589421 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586416    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586419    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586421    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586424    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586426    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586429    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586431    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586434    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586442    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586445    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586447    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586450    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586453    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586455    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586457    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586460    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586462    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586465    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586467    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:15:36.589901 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586469    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586472    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586474    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586477    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586479    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586482    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586485    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586487    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586489    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586492    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586495    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586498    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586500    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586504    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586507    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586510    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586512    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586514    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586517    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:15:36.590363 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586519    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:15:36.590843 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586521    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:15:36.590843 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586524    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:15:36.590843 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586533    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:15:36.590843 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586537    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:15:36.590843 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586539    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:15:36.590843 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586542    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:15:36.590843 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:36.586545    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:15:36.590843 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.586550    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:15:36.590843 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.586677    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 15:15:36.590843 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.589067    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 15:15:36.590843 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.590104    2573 server.go:1019] "Starting client certificate rotation"
Sep 10 15:15:36.590843 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.590202    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 15:15:36.590843 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.590245    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 15:15:36.595401 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.595383    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 15:15:36.596771 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.596741    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 15:15:36.614434 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.614406    2573 log.go:25] "Validated CRI v1 runtime API"
Sep 10 15:15:36.617588 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.617568    2573 log.go:25] "Validated CRI v1 image API"
Sep 10 15:15:36.618429 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.618403    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 15:15:36.621875 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.621855    2573 fs.go:135] Filesystem UUIDs: map[520c8b83-be82-4d1b-8315-161c7ecf95dd:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7d45ea38-b05d-4859-a34d-352a3fadb8a4:/dev/nvme0n1p4]
Sep 10 15:15:36.621951 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.621875    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 15:15:36.623763 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.623744    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 15:15:36.628468 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.628349    2573 manager.go:216] Machine: {Timestamp:2026-09-10 15:15:36.627119339 +0000 UTC m=+0.380998292 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097202 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24c45d72bb13c99e898b9eae399f54 SystemUUID:ec24c45d-72bb-13c9-9e89-8b9eae399f54 BootID:67e38867-a350-425a-8b65-377c7b2a0d22 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:39:b9:c7:62:4b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:39:b9:c7:62:4b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:06:6d:54:b3:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 15:15:36.628468 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.628463    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 15:15:36.628591 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.628553    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 15:15:36.632964 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.632926    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 15:15:36.633120 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.632967    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-115.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 15:15:36.633833 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.633823    2573 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 15:15:36.633872 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.633835    2573 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 15:15:36.633872 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.633844    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 15:15:36.634514 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.634503    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 15:15:36.636172 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.636161    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 15:15:36.640572 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.640559    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 15:15:36.640725 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.640700    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 15:15:36.640763 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.640755    2573 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 15:15:36.640794 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.640769    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 15:15:36.640794 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.640788    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 15:15:36.640844 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.640798    2573 kubelet.go:398] "Adding apiserver pod source"
Sep 10 15:15:36.640844 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.640807    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 15:15:36.641196 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.641182    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 15:15:36.641240 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.641203    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 15:15:36.643588 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.643567    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 15:15:36.646741 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.646701    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 15:15:36.646838 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.646762    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 15:15:36.646957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.646945    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 15:15:36.647013 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.646964    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 15:15:36.647013 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.646973    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 15:15:36.647013 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.646982    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 15:15:36.647013 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.646994    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 15:15:36.647013 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.647004    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 15:15:36.647652 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.647640    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 15:15:36.647726 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.647654    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 15:15:36.647726 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.647665    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 15:15:36.647726 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.647674    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 15:15:36.647726 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.647694    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 15:15:36.647726 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.647726    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 15:15:36.649482 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.649470    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 15:15:36.649532 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.649487    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 15:15:36.651738 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.651717    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-115.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 15:15:36.652606 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.652572    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 15:15:36.652606 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.652578    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 15:15:36.654303 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.654286    2573 server.go:1268] "Started kubelet"
Sep 10 15:15:36.654425 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.654372    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 15:15:36.654527 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.654381    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 15:15:36.654627 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.654544    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 15:15:36.655219 ip-10-0-134-115 systemd[1]: Started Kubernetes Kubelet.
Sep 10 15:15:36.656322 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.656214    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 15:15:36.657143 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.657128    2573 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 15:15:36.658914 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.658894    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 15:15:36.661135 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.660029    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-115.ec2.internal.18d3fe87e780e1d7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-115.ec2.internal,UID:ip-10-0-134-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-115.ec2.internal,},FirstTimestamp:2026-09-10 15:15:36.654242263 +0000 UTC m=+0.408121217,LastTimestamp:2026-09-10 15:15:36.654242263 +0000 UTC m=+0.408121217,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-115.ec2.internal,}"
Sep 10 15:15:36.662095 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.662081    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 15:15:36.662232 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.662207    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:36.662303 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.662237    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 15:15:36.662871 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.662848    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 15:15:36.662871 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.662851    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 15:15:36.662999 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.662883    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 15:15:36.662999 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.662984    2573 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 15:15:36.662999 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.662994    2573 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 15:15:36.663137 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.663035    2573 factory.go:55] Registering systemd factory
Sep 10 15:15:36.663137 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.663053    2573 factory.go:223] Registration of the systemd container factory successfully
Sep 10 15:15:36.663450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.663431    2573 factory.go:153] Registering CRI-O factory
Sep 10 15:15:36.663450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.663449    2573 factory.go:223] Registration of the crio container factory successfully
Sep 10 15:15:36.663586 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.663533    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 15:15:36.663586 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.663550    2573 factory.go:103] Registering Raw factory
Sep 10 15:15:36.663586 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.663564    2573 manager.go:1196] Started watching for new ooms in manager
Sep 10 15:15:36.664114 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.664025    2573 manager.go:349] Starting recovery of all containers
Sep 10 15:15:36.669865 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.669828    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 15:15:36.669965 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.669947    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 15:15:36.673210 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.673185    2573 manager.go:354] Recovery completed
Sep 10 15:15:36.676614 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.676500    2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 10 15:15:36.679910 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.679895    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:15:36.681879 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.681856    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:15:36.681950 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.681897    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:15:36.681950 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.681908    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:15:36.682514 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.682503    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 15:15:36.682550 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.682514    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 15:15:36.682550 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.682523    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:15:36.682620 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.682530    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 15:15:36.683227 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.683210    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:15:36.683279 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.683238    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:15:36.683279 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.683250    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:15:36.683881 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.683807    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-115.ec2.internal.18d3fe87e926967a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-115.ec2.internal,UID:ip-10-0-134-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-115.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-115.ec2.internal,},FirstTimestamp:2026-09-10 15:15:36.681879162 +0000 UTC m=+0.435758120,LastTimestamp:2026-09-10 15:15:36.681879162 +0000 UTC m=+0.435758120,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-115.ec2.internal,}"
Sep 10 15:15:36.684879 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.684867    2573 policy_none.go:49] "None policy: Start"
Sep 10 15:15:36.684927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.684883    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 15:15:36.684927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.684893    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 15:15:36.686782 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.686771    2573 policy_none.go:47] "Start"
Sep 10 15:15:36.687692 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.687676    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2wx5d"
Sep 10 15:15:36.694110 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.694041    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-115.ec2.internal.18d3fe87e926f363  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-115.ec2.internal,UID:ip-10-0-134-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-115.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-115.ec2.internal,},FirstTimestamp:2026-09-10 15:15:36.681902947 +0000 UTC m=+0.435781901,LastTimestamp:2026-09-10 15:15:36.681902947 +0000 UTC m=+0.435781901,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-115.ec2.internal,}"
Sep 10 15:15:36.695442 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.695424    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2wx5d"
Sep 10 15:15:36.729398 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.729380    2573 manager.go:337] "Starting Device Plugin manager"
Sep 10 15:15:36.751210 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.729431    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 15:15:36.751210 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.729443    2573 server.go:87] "Starting device plugin registration server"
Sep 10 15:15:36.751210 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.729677    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 15:15:36.751210 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.729688    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 15:15:36.751210 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.729854    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 15:15:36.751210 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.730001    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 15:15:36.751210 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.730008    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 15:15:36.751210 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.730356    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 15:15:36.751210 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.730404    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:36.779599 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.779561    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 15:15:36.780973 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.780957    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 15:15:36.781076 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.781021    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 15:15:36.781076 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.781051    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 15:15:36.781161 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.781100    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 15:15:36.783551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.783527    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:15:36.830899 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.830811    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:15:36.831782 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.831765    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:15:36.831897 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.831800    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:15:36.831897 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.831814    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:15:36.831897 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.831844    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-115.ec2.internal"
Sep 10 15:15:36.841237 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.841217    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-115.ec2.internal"
Sep 10 15:15:36.841344 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.841247    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-115.ec2.internal\": node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:36.867701 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.867670    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:36.881906 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.881882    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-115.ec2.internal"]
Sep 10 15:15:36.881995 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.881950    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:15:36.883889 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.883873    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:15:36.883981 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.883905    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:15:36.883981 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.883919    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:15:36.886250 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.886235    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:15:36.886388 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.886365    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:36.886446 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.886395    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:15:36.887321 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.887302    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:15:36.887420 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.887337    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:15:36.887420 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.887348    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:15:36.887420 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.887302    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:15:36.887518 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.887419    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:15:36.887518 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.887435    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:15:36.889557 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.889541    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:36.889627 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.889565    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:15:36.890286 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.890268    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:15:36.890355 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.890298    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:15:36.890355 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.890309    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:15:36.922352 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.922326    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-115.ec2.internal\" not found" node="ip-10-0-134-115.ec2.internal"
Sep 10 15:15:36.925772 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.925755    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-115.ec2.internal\" not found" node="ip-10-0-134-115.ec2.internal"
Sep 10 15:15:36.965053 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.965023    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17327a0c112603d3f5deb1d673c13162-config\") pod \"kube-apiserver-proxy-ip-10-0-134-115.ec2.internal\" (UID: \"17327a0c112603d3f5deb1d673c13162\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:36.965053 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.965050    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71617ef53acb8f894161ea9a17f46531-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal\" (UID: \"71617ef53acb8f894161ea9a17f46531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:36.965226 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:36.965068    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71617ef53acb8f894161ea9a17f46531-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal\" (UID: \"71617ef53acb8f894161ea9a17f46531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:36.968136 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:36.968120    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:37.066152 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.066063    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71617ef53acb8f894161ea9a17f46531-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal\" (UID: \"71617ef53acb8f894161ea9a17f46531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:37.066152 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.066150    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71617ef53acb8f894161ea9a17f46531-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal\" (UID: \"71617ef53acb8f894161ea9a17f46531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:37.066323 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.066179    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17327a0c112603d3f5deb1d673c13162-config\") pod \"kube-apiserver-proxy-ip-10-0-134-115.ec2.internal\" (UID: \"17327a0c112603d3f5deb1d673c13162\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:37.066323 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.066197    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71617ef53acb8f894161ea9a17f46531-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal\" (UID: \"71617ef53acb8f894161ea9a17f46531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:37.066323 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.066233    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71617ef53acb8f894161ea9a17f46531-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal\" (UID: \"71617ef53acb8f894161ea9a17f46531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:37.066323 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.066273    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17327a0c112603d3f5deb1d673c13162-config\") pod \"kube-apiserver-proxy-ip-10-0-134-115.ec2.internal\" (UID: \"17327a0c112603d3f5deb1d673c13162\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:37.069167 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:37.069153    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:37.170064 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:37.169984    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:37.225170 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.225141    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:37.228787 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.228766    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:37.271087 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:37.270904    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:37.371430 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:37.371380    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:37.472018 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:37.471939    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:37.572448 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:37.572422    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:37.589627 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.589599    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 15:15:37.589789 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.589747    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 15:15:37.673515 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:37.673485    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:37.698264 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.698228    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 15:10:38 +0000 UTC" deadline="2028-06-26 09:46:44.191971706 +0000 UTC"
Sep 10 15:15:37.698264 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.698259    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15714h31m6.493716059s"
Sep 10 15:15:37.705387 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.705369    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:15:37.707442 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.707425    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:15:37.707550 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.707456    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:15:37.707550 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.707466    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:15:37.773884 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:37.773825    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:37.874171 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:37.874135    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-115.ec2.internal\" not found"
Sep 10 15:15:37.889072 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.889046    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:15:37.913033 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.912391    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:15:37.962593 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.962566    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:37.972209 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:37.972182    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71617ef53acb8f894161ea9a17f46531.slice/crio-44c7719df6cfa875cf20865d4a1eff1e2119724c3c043549a761c98e3016cb23 WatchSource:0}: Error finding container 44c7719df6cfa875cf20865d4a1eff1e2119724c3c043549a761c98e3016cb23: Status 404 returned error can't find the container with id 44c7719df6cfa875cf20865d4a1eff1e2119724c3c043549a761c98e3016cb23
Sep 10 15:15:37.976575 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.976543    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 15:15:37.976738 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.976722    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:15:37.977522 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.977511    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-115.ec2.internal"
Sep 10 15:15:37.986971 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:37.986954    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 15:15:38.072344 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:38.072258    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:15:39.980869 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:39.980842    2573 apiserver.go:52] "Watching apiserver"
Sep 10 15:15:39.988886 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:39.988855    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 15:15:39.990717 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:39.990691    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-krgl6","openshift-ovn-kubernetes/ovnkube-node-mcn9l","kube-system/konnectivity-agent-zc2lh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj","openshift-cluster-node-tuning-operator/tuned-9x6dh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal","openshift-multus/multus-additional-cni-plugins-x7wkz","openshift-multus/multus-ql8lm","openshift-multus/network-metrics-daemon-5kk8k","openshift-network-diagnostics/network-check-target-dqbgv","kube-system/global-pull-secret-syncer-wkpgh","kube-system/kube-apiserver-proxy-ip-10-0-134-115.ec2.internal","openshift-dns/node-resolver-kqvj8","openshift-image-registry/node-ca-xsmjp"]
Sep 10 15:15:39.993409 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:39.993385    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:39.995623 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:39.995582    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kqvj8"
Sep 10 15:15:39.997933 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:39.997909    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zc2lh"
Sep 10 15:15:40.000090 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.000072    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.001294 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.001274    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 15:15:40.002093 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.002075    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 15:15:40.002447 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.002431    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 15:15:40.002680 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.002665    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-brv7s\""
Sep 10 15:15:40.003358 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.002898    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.003358 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.003221    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 15:15:40.003488 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.003361    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 15:15:40.003594 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.003577    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 15:15:40.003677 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.003579    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 15:15:40.003886 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.003869    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hf496\""
Sep 10 15:15:40.005091 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.004193    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 15:15:40.005091 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.004212    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 15:15:40.005091 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.004298    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 15:15:40.005091 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.004488    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 15:15:40.005091 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.004494    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f4mcd\""
Sep 10 15:15:40.005091 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.004530    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 15:15:40.005091 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.004720    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z9l6k\""
Sep 10 15:15:40.005091 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.004752    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xsmjp"
Sep 10 15:15:40.005091 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.004772    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 15:15:40.006501 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.006481    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:15:40.006708 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.006688    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m2w42\""
Sep 10 15:15:40.007981 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.007962    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:40.008068 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.008042    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:15:40.008444 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.008418    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fq8tg\""
Sep 10 15:15:40.008626 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.008613    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 15:15:40.008803 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.008773    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 15:15:40.008986 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.008970    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 15:15:40.011638 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.011614    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 15:15:40.013696 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.012767    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.014044 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.014030    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-krgl6"
Sep 10 15:15:40.014498 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.014480    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-sys\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.014573 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.014517    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7392e59-f889-4a0e-bcf0-95a7ba7da181-cnibin\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.014573 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.014551    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-device-dir\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.014666 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.014574    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.014666 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.014604    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-socket-dir\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.014666 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.014632    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-registration-dir\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.014755 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.014661    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-modprobe-d\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.014755 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.014738    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-sysctl-d\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.014906 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.014868    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:40.014975 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.014937    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:15:40.015341 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.015316    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-sysctl-conf\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.015415 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.015361    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-systemd\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.015415 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.015409    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7392e59-f889-4a0e-bcf0-95a7ba7da181-system-cni-dir\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.016210 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016188    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e0b284-688f-4bd7-a282-f9052bb11bf4-tmp-dir\") pod \"node-resolver-kqvj8\" (UID: \"04e0b284-688f-4bd7-a282-f9052bb11bf4\") " pod="openshift-dns/node-resolver-kqvj8"
Sep 10 15:15:40.016351 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016275    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-kubernetes\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.016351 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016303    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-var-lib-kubelet\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.016433 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016367    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/216864e7-bd81-4a5a-a0d7-f243edea4857-agent-certs\") pod \"konnectivity-agent-zc2lh\" (UID: \"216864e7-bd81-4a5a-a0d7-f243edea4857\") " pod="kube-system/konnectivity-agent-zc2lh"
Sep 10 15:15:40.016433 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016399    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-sysconfig\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.016433 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016427    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-run\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.016595 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016572    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-lib-modules\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.016646 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016600    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 15:15:40.016646 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-822sp\" (UniqueName: \"kubernetes.io/projected/c9032d89-2a88-412b-ae35-5088f9eaad77-kube-api-access-822sp\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.016737 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016657    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7392e59-f889-4a0e-bcf0-95a7ba7da181-cni-binary-copy\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.016737 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016602    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 15:15:40.016737 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016690    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c7392e59-f889-4a0e-bcf0-95a7ba7da181-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.016737 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016661    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 15:15:40.016998 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.016917    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 15:15:40.017066 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.017041    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 15:15:40.017266 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.017249    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pqmdk\""
Sep 10 15:15:40.017357 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.017328    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:15:40.017551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.017531    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 15:15:40.017551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.017540    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9pct9\""
Sep 10 15:15:40.017706 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.017692    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 15:15:40.017830 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.017778    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 15:15:40.017830 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.017813    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:40.018221 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.018200    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:15:40.020249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.018401    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfr6p\" (UniqueName: \"kubernetes.io/projected/c7392e59-f889-4a0e-bcf0-95a7ba7da181-kube-api-access-gfr6p\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.020249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.018441    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/216864e7-bd81-4a5a-a0d7-f243edea4857-konnectivity-ca\") pod \"konnectivity-agent-zc2lh\" (UID: \"216864e7-bd81-4a5a-a0d7-f243edea4857\") " pod="kube-system/konnectivity-agent-zc2lh"
Sep 10 15:15:40.020249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.018466    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-sys-fs\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.020249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.018489    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmmvp\" (UniqueName: \"kubernetes.io/projected/92ed881e-9100-40d2-b868-10d616b69e49-kube-api-access-cmmvp\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.020249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.018526    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-host\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.020249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.018551    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-tuned\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.020249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.018573    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7392e59-f889-4a0e-bcf0-95a7ba7da181-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.020249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.018596    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz2bq\" (UniqueName: \"kubernetes.io/projected/04e0b284-688f-4bd7-a282-f9052bb11bf4-kube-api-access-sz2bq\") pod \"node-resolver-kqvj8\" (UID: \"04e0b284-688f-4bd7-a282-f9052bb11bf4\") " pod="openshift-dns/node-resolver-kqvj8"
Sep 10 15:15:40.020249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.018637    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-etc-selinux\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.020249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.018682    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9032d89-2a88-412b-ae35-5088f9eaad77-tmp\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.020249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.018716    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c7392e59-f889-4a0e-bcf0-95a7ba7da181-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.020249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.018740    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04e0b284-688f-4bd7-a282-f9052bb11bf4-hosts-file\") pod \"node-resolver-kqvj8\" (UID: \"04e0b284-688f-4bd7-a282-f9052bb11bf4\") " pod="openshift-dns/node-resolver-kqvj8"
Sep 10 15:15:40.020988 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.020971    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 15:15:40.023330 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.023031    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.025228 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.025207    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 15:15:40.025691 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.025670    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k98s5\""
Sep 10 15:15:40.037838 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.037818    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2dpbs"
Sep 10 15:15:40.045928 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.045901    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2dpbs"
Sep 10 15:15:40.102599 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.102575    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 15:15:40.119177 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119141    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c7392e59-f889-4a0e-bcf0-95a7ba7da181-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.119317 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfr6p\" (UniqueName: \"kubernetes.io/projected/c7392e59-f889-4a0e-bcf0-95a7ba7da181-kube-api-access-gfr6p\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.119317 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119212    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-hostroot\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.119317 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119237    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-multus-daemon-config\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.119317 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d-serviceca\") pod \"node-ca-xsmjp\" (UID: \"b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d\") " pod="openshift-image-registry/node-ca-xsmjp"
Sep 10 15:15:40.119317 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119284    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-node-log\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.119317 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759rr\" (UniqueName: \"kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr\") pod \"network-check-target-dqbgv\" (UID: \"42483446-4d7f-4a0d-96c7-7771f0b5a14c\") " pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:40.119542 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119330    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:40.119542 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119355    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh98q\" (UniqueName: \"kubernetes.io/projected/c38e7952-153e-4a04-8e6c-15470498aea9-kube-api-access-qh98q\") pod \"iptables-alerter-krgl6\" (UID: \"c38e7952-153e-4a04-8e6c-15470498aea9\") " pod="openshift-network-operator/iptables-alerter-krgl6"
Sep 10 15:15:40.119542 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119371    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-host\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.119542 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119391    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-tuned\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.119542 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119415    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81669089-b97a-4d9f-907d-4d62da491cac-ovnkube-script-lib\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.119542 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119477    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-host\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.119542 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119512    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9032d89-2a88-412b-ae35-5088f9eaad77-tmp\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.119542 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119536    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c7392e59-f889-4a0e-bcf0-95a7ba7da181-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.119826 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04e0b284-688f-4bd7-a282-f9052bb11bf4-hosts-file\") pod \"node-resolver-kqvj8\" (UID: \"04e0b284-688f-4bd7-a282-f9052bb11bf4\") " pod="openshift-dns/node-resolver-kqvj8"
Sep 10 15:15:40.119826 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119639    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-multus-conf-dir\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.119826 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119674    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.119826 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119693    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 15:15:40.119826 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119783    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81669089-b97a-4d9f-907d-4d62da491cac-ovnkube-config\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.120052 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119943    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04e0b284-688f-4bd7-a282-f9052bb11bf4-hosts-file\") pod \"node-resolver-kqvj8\" (UID: \"04e0b284-688f-4bd7-a282-f9052bb11bf4\") " pod="openshift-dns/node-resolver-kqvj8"
Sep 10 15:15:40.120052 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.119978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81669089-b97a-4d9f-907d-4d62da491cac-env-overrides\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.120052 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120002    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d-host\") pod \"node-ca-xsmjp\" (UID: \"b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d\") " pod="openshift-image-registry/node-ca-xsmjp"
Sep 10 15:15:40.120052 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120036    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-device-dir\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.120258 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120061    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-system-cni-dir\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.120258 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120115    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.120258 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120142    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-registration-dir\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.120258 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120217    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrf78\" (UniqueName: \"kubernetes.io/projected/190adfd7-a800-4b59-b03c-fd0508ada6de-kube-api-access-rrf78\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:40.120258 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120244    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-systemd-units\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.120442 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120269    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-sysctl-d\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.120442 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120294    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7392e59-f889-4a0e-bcf0-95a7ba7da181-system-cni-dir\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.120442 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e0b284-688f-4bd7-a282-f9052bb11bf4-tmp-dir\") pod \"node-resolver-kqvj8\" (UID: \"04e0b284-688f-4bd7-a282-f9052bb11bf4\") " pod="openshift-dns/node-resolver-kqvj8"
Sep 10 15:15:40.120442 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120342    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-cni-bin\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.120442 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120362    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-kubernetes\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.120442 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120378    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-var-lib-kubelet\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.120442 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120433    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/216864e7-bd81-4a5a-a0d7-f243edea4857-agent-certs\") pod \"konnectivity-agent-zc2lh\" (UID: \"216864e7-bd81-4a5a-a0d7-f243edea4857\") " pod="kube-system/konnectivity-agent-zc2lh"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120449    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-run-multus-certs\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-run-ovn\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120490    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81669089-b97a-4d9f-907d-4d62da491cac-ovn-node-metrics-cert\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120505    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-run\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120540    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-lib-modules\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120555    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7392e59-f889-4a0e-bcf0-95a7ba7da181-cni-binary-copy\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120571    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/216864e7-bd81-4a5a-a0d7-f243edea4857-konnectivity-ca\") pod \"konnectivity-agent-zc2lh\" (UID: \"216864e7-bd81-4a5a-a0d7-f243edea4857\") " pod="kube-system/konnectivity-agent-zc2lh"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-sys-fs\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120624    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmmvp\" (UniqueName: \"kubernetes.io/projected/92ed881e-9100-40d2-b868-10d616b69e49-kube-api-access-cmmvp\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120643    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-run-netns\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120657    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-var-lib-kubelet\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120671    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-cni-binary-copy\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120685    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-etc-kubernetes\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120700    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-slash\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120707    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c7392e59-f889-4a0e-bcf0-95a7ba7da181-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.120776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120714    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-run-netns\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-var-lib-openvswitch\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120826    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-lib-modules\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120827    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-etc-openvswitch\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120859    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-cni-netd\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120877    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c38e7952-153e-4a04-8e6c-15470498aea9-iptables-alerter-script\") pod \"iptables-alerter-krgl6\" (UID: \"c38e7952-153e-4a04-8e6c-15470498aea9\") " pod="openshift-network-operator/iptables-alerter-krgl6"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120898    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7392e59-f889-4a0e-bcf0-95a7ba7da181-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120915    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz2bq\" (UniqueName: \"kubernetes.io/projected/04e0b284-688f-4bd7-a282-f9052bb11bf4-kube-api-access-sz2bq\") pod \"node-resolver-kqvj8\" (UID: \"04e0b284-688f-4bd7-a282-f9052bb11bf4\") " pod="openshift-dns/node-resolver-kqvj8"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120914    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-registration-dir\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120944    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-etc-selinux\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120971    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f858dfb-38ba-44b7-8b7d-b38cfb655428-dbus\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120999    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-multus-socket-dir-parent\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121027    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121053    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c38e7952-153e-4a04-8e6c-15470498aea9-host-slash\") pod \"iptables-alerter-krgl6\" (UID: \"c38e7952-153e-4a04-8e6c-15470498aea9\") " pod="openshift-network-operator/iptables-alerter-krgl6"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121078    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-sys\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121119    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7392e59-f889-4a0e-bcf0-95a7ba7da181-cnibin\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121144    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-cnibin\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.121679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121172    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-kubelet\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121202    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-run-systemd\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121226    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfhg7\" (UniqueName: \"kubernetes.io/projected/81669089-b97a-4d9f-907d-4d62da491cac-kube-api-access-nfhg7\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121253    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkngs\" (UniqueName: \"kubernetes.io/projected/b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d-kube-api-access-qkngs\") pod \"node-ca-xsmjp\" (UID: \"b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d\") " pod="openshift-image-registry/node-ca-xsmjp"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121314    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-device-dir\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121356    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121397    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7392e59-f889-4a0e-bcf0-95a7ba7da181-cni-binary-copy\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.120876    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-var-lib-kubelet\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121439    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-sysctl-d\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121474    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7392e59-f889-4a0e-bcf0-95a7ba7da181-system-cni-dir\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121530    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7392e59-f889-4a0e-bcf0-95a7ba7da181-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121589    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-sys-fs\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121708    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e0b284-688f-4bd7-a282-f9052bb11bf4-tmp-dir\") pod \"node-resolver-kqvj8\" (UID: \"04e0b284-688f-4bd7-a282-f9052bb11bf4\") " pod="openshift-dns/node-resolver-kqvj8"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121733    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-socket-dir\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121711    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/216864e7-bd81-4a5a-a0d7-f243edea4857-konnectivity-ca\") pod \"konnectivity-agent-zc2lh\" (UID: \"216864e7-bd81-4a5a-a0d7-f243edea4857\") " pod="kube-system/konnectivity-agent-zc2lh"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121765    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-run-k8s-cni-cncf-io\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121811    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-var-lib-cni-multus\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.122483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121817    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-kubernetes\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121842    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj9dm\" (UniqueName: \"kubernetes.io/projected/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-kube-api-access-fj9dm\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121877    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f858dfb-38ba-44b7-8b7d-b38cfb655428-kubelet-config\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121892    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-socket-dir\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121905    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-modprobe-d\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121908    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-run\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121932    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-sysctl-conf\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.121227    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c7392e59-f889-4a0e-bcf0-95a7ba7da181-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122031    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-sysctl-conf\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122046    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92ed881e-9100-40d2-b868-10d616b69e49-etc-selinux\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122069    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7392e59-f889-4a0e-bcf0-95a7ba7da181-cnibin\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122092    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-sys\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122128    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-modprobe-d\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122160    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-systemd\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122183    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-multus-cni-dir\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122207    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-var-lib-cni-bin\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122247    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-log-socket\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.123287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122290    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-systemd\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.124040 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122299    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-run-openvswitch\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.124040 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122357    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.124040 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122407    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-sysconfig\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.124040 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122439    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-822sp\" (UniqueName: \"kubernetes.io/projected/c9032d89-2a88-412b-ae35-5088f9eaad77-kube-api-access-822sp\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.124040 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.122681    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-sysconfig\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.124655 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.124629    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9032d89-2a88-412b-ae35-5088f9eaad77-tmp\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.125437 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.125414    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/216864e7-bd81-4a5a-a0d7-f243edea4857-agent-certs\") pod \"konnectivity-agent-zc2lh\" (UID: \"216864e7-bd81-4a5a-a0d7-f243edea4857\") " pod="kube-system/konnectivity-agent-zc2lh"
Sep 10 15:15:40.126958 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.126936    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9032d89-2a88-412b-ae35-5088f9eaad77-etc-tuned\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.127307 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.127268    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-115.ec2.internal" event={"ID":"17327a0c112603d3f5deb1d673c13162","Type":"ContainerStarted","Data":"197d4125a6acba381e66afbd8deededd2b9605bc6b2a10f1c99e98627013bae7"}
Sep 10 15:15:40.130464 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.130445    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz2bq\" (UniqueName: \"kubernetes.io/projected/04e0b284-688f-4bd7-a282-f9052bb11bf4-kube-api-access-sz2bq\") pod \"node-resolver-kqvj8\" (UID: \"04e0b284-688f-4bd7-a282-f9052bb11bf4\") " pod="openshift-dns/node-resolver-kqvj8"
Sep 10 15:15:40.131254 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.131178    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal" event={"ID":"71617ef53acb8f894161ea9a17f46531","Type":"ContainerStarted","Data":"44c7719df6cfa875cf20865d4a1eff1e2119724c3c043549a761c98e3016cb23"}
Sep 10 15:15:40.132448 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.132124    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfr6p\" (UniqueName: \"kubernetes.io/projected/c7392e59-f889-4a0e-bcf0-95a7ba7da181-kube-api-access-gfr6p\") pod \"multus-additional-cni-plugins-x7wkz\" (UID: \"c7392e59-f889-4a0e-bcf0-95a7ba7da181\") " pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.132448 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.132419    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-822sp\" (UniqueName: \"kubernetes.io/projected/c9032d89-2a88-412b-ae35-5088f9eaad77-kube-api-access-822sp\") pod \"tuned-9x6dh\" (UID: \"c9032d89-2a88-412b-ae35-5088f9eaad77\") " pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.132741 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.132725    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmmvp\" (UniqueName: \"kubernetes.io/projected/92ed881e-9100-40d2-b868-10d616b69e49-kube-api-access-cmmvp\") pod \"aws-ebs-csi-driver-node-796zj\" (UID: \"92ed881e-9100-40d2-b868-10d616b69e49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.223642 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223595    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-cni-bin\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.223642 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223638    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-run-multus-certs\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.223931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223656    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-run-ovn\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.223931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81669089-b97a-4d9f-907d-4d62da491cac-ovn-node-metrics-cert\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.223931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223702    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-run-netns\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.223931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223723    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-var-lib-kubelet\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.223931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223747    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-cni-binary-copy\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.223931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-etc-kubernetes\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.223931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223808    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-slash\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.223931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223844    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-run-netns\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.223931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-var-lib-openvswitch\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.223931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223889    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-etc-openvswitch\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.223931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223911    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-cni-netd\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.223931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c38e7952-153e-4a04-8e6c-15470498aea9-iptables-alerter-script\") pod \"iptables-alerter-krgl6\" (UID: \"c38e7952-153e-4a04-8e6c-15470498aea9\") " pod="openshift-network-operator/iptables-alerter-krgl6"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223965    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f858dfb-38ba-44b7-8b7d-b38cfb655428-dbus\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.223987    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-multus-socket-dir-parent\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224010    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224029    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c38e7952-153e-4a04-8e6c-15470498aea9-host-slash\") pod \"iptables-alerter-krgl6\" (UID: \"c38e7952-153e-4a04-8e6c-15470498aea9\") " pod="openshift-network-operator/iptables-alerter-krgl6"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224046    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-cnibin\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-kubelet\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224072    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-run-systemd\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224087    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfhg7\" (UniqueName: \"kubernetes.io/projected/81669089-b97a-4d9f-907d-4d62da491cac-kube-api-access-nfhg7\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkngs\" (UniqueName: \"kubernetes.io/projected/b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d-kube-api-access-qkngs\") pod \"node-ca-xsmjp\" (UID: \"b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d\") " pod="openshift-image-registry/node-ca-xsmjp"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-run-k8s-cni-cncf-io\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224149    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-var-lib-cni-multus\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224164    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9dm\" (UniqueName: \"kubernetes.io/projected/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-kube-api-access-fj9dm\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224181    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f858dfb-38ba-44b7-8b7d-b38cfb655428-kubelet-config\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224206    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-multus-cni-dir\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224223    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-var-lib-cni-bin\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224238    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-log-socket\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.224389 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224253    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-run-openvswitch\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224269    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224288    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-hostroot\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224303    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-multus-daemon-config\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d-serviceca\") pod \"node-ca-xsmjp\" (UID: \"b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d\") " pod="openshift-image-registry/node-ca-xsmjp"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224333    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-node-log\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224348    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759rr\" (UniqueName: \"kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr\") pod \"network-check-target-dqbgv\" (UID: \"42483446-4d7f-4a0d-96c7-7771f0b5a14c\") " pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224363    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224377    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh98q\" (UniqueName: \"kubernetes.io/projected/c38e7952-153e-4a04-8e6c-15470498aea9-kube-api-access-qh98q\") pod \"iptables-alerter-krgl6\" (UID: \"c38e7952-153e-4a04-8e6c-15470498aea9\") " pod="openshift-network-operator/iptables-alerter-krgl6"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81669089-b97a-4d9f-907d-4d62da491cac-ovnkube-script-lib\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-multus-conf-dir\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224428    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224442    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81669089-b97a-4d9f-907d-4d62da491cac-ovnkube-config\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81669089-b97a-4d9f-907d-4d62da491cac-env-overrides\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224471    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d-host\") pod \"node-ca-xsmjp\" (UID: \"b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d\") " pod="openshift-image-registry/node-ca-xsmjp"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224488    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-system-cni-dir\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224514    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrf78\" (UniqueName: \"kubernetes.io/projected/190adfd7-a800-4b59-b03c-fd0508ada6de-kube-api-access-rrf78\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:40.224911 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224530    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-systemd-units\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.225575 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224612    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-systemd-units\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.225575 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224644    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-cni-bin\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.225575 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224664    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-run-multus-certs\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.225575 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.224686    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-run-ovn\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.225575 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225210    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f858dfb-38ba-44b7-8b7d-b38cfb655428-dbus\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:40.225575 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225274    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-run-netns\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.225575 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225315    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-var-lib-kubelet\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.225575 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225335    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-multus-socket-dir-parent\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.225575 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225487    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-multus-cni-dir\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.225575 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225541    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-var-lib-cni-bin\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.225593    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225635    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-run-openvswitch\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.225664    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs podName:190adfd7-a800-4b59-b03c-fd0508ada6de nodeName:}" failed. No retries permitted until 2026-09-10 15:15:40.725642488 +0000 UTC m=+3.141438303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs") pod "network-metrics-daemon-5kk8k" (UID: "190adfd7-a800-4b59-b03c-fd0508ada6de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225686    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225707    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c38e7952-153e-4a04-8e6c-15470498aea9-host-slash\") pod \"iptables-alerter-krgl6\" (UID: \"c38e7952-153e-4a04-8e6c-15470498aea9\") " pod="openshift-network-operator/iptables-alerter-krgl6"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225736    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-hostroot\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225765    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-cnibin\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225420    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f858dfb-38ba-44b7-8b7d-b38cfb655428-kubelet-config\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225594    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-log-socket\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225855    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-cni-binary-copy\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225914    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-etc-kubernetes\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225951    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-slash\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.225995    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-run-netns\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226023    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-etc-openvswitch\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226039    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-cni-netd\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.226070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226069    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-var-lib-openvswitch\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226109    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-kubelet\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226143    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-run-systemd\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226335    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d-serviceca\") pod \"node-ca-xsmjp\" (UID: \"b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d\") " pod="openshift-image-registry/node-ca-xsmjp"
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226400    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-node-log\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226434    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-var-lib-cni-multus\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226436    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-multus-daemon-config\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226496    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-host-run-k8s-cni-cncf-io\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226509    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d-host\") pod \"node-ca-xsmjp\" (UID: \"b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d\") " pod="openshift-image-registry/node-ca-xsmjp"
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.226551    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226554    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-multus-conf-dir\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.226595    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret podName:3f858dfb-38ba-44b7-8b7d-b38cfb655428 nodeName:}" failed. No retries permitted until 2026-09-10 15:15:40.726580319 +0000 UTC m=+3.142376135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret") pod "global-pull-secret-syncer-wkpgh" (UID: "3f858dfb-38ba-44b7-8b7d-b38cfb655428") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226625    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-system-cni-dir\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226741    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81669089-b97a-4d9f-907d-4d62da491cac-host-run-ovn-kubernetes\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.226823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.226548    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c38e7952-153e-4a04-8e6c-15470498aea9-iptables-alerter-script\") pod \"iptables-alerter-krgl6\" (UID: \"c38e7952-153e-4a04-8e6c-15470498aea9\") " pod="openshift-network-operator/iptables-alerter-krgl6"
Sep 10 15:15:40.227380 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.227072    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81669089-b97a-4d9f-907d-4d62da491cac-ovnkube-config\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.227431 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.227409    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81669089-b97a-4d9f-907d-4d62da491cac-ovnkube-script-lib\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.227687 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.227665    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81669089-b97a-4d9f-907d-4d62da491cac-env-overrides\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.229626 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.229549    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81669089-b97a-4d9f-907d-4d62da491cac-ovn-node-metrics-cert\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.240634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.239753    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkngs\" (UniqueName: \"kubernetes.io/projected/b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d-kube-api-access-qkngs\") pod \"node-ca-xsmjp\" (UID: \"b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d\") " pod="openshift-image-registry/node-ca-xsmjp"
Sep 10 15:15:40.240634 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.240050    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:15:40.240634 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.240082    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:15:40.240634 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.240093    2573 projected.go:196] Error preparing data for projected volume kube-api-access-759rr for pod openshift-network-diagnostics/network-check-target-dqbgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:15:40.240634 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.240139    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr podName:42483446-4d7f-4a0d-96c7-7771f0b5a14c nodeName:}" failed. No retries permitted until 2026-09-10 15:15:40.740121397 +0000 UTC m=+3.155917194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-759rr" (UniqueName: "kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr") pod "network-check-target-dqbgv" (UID: "42483446-4d7f-4a0d-96c7-7771f0b5a14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:15:40.241657 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.241631    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh98q\" (UniqueName: \"kubernetes.io/projected/c38e7952-153e-4a04-8e6c-15470498aea9-kube-api-access-qh98q\") pod \"iptables-alerter-krgl6\" (UID: \"c38e7952-153e-4a04-8e6c-15470498aea9\") " pod="openshift-network-operator/iptables-alerter-krgl6"
Sep 10 15:15:40.243568 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.243541    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9dm\" (UniqueName: \"kubernetes.io/projected/b9d07e39-b7e2-4f78-8648-4a0c188dc0f4-kube-api-access-fj9dm\") pod \"multus-ql8lm\" (UID: \"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4\") " pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.244035 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.243957    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfhg7\" (UniqueName: \"kubernetes.io/projected/81669089-b97a-4d9f-907d-4d62da491cac-kube-api-access-nfhg7\") pod \"ovnkube-node-mcn9l\" (UID: \"81669089-b97a-4d9f-907d-4d62da491cac\") " pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.246611 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.246590    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrf78\" (UniqueName: \"kubernetes.io/projected/190adfd7-a800-4b59-b03c-fd0508ada6de-kube-api-access-rrf78\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:40.306241 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.306205    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x7wkz"
Sep 10 15:15:40.315146 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.315118    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kqvj8"
Sep 10 15:15:40.316203 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:40.316128    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7392e59_f889_4a0e_bcf0_95a7ba7da181.slice/crio-47a5f7c61344f76be200b632dbd2a34ff75a5373ecddfe85e792dbd2239f6703 WatchSource:0}: Error finding container 47a5f7c61344f76be200b632dbd2a34ff75a5373ecddfe85e792dbd2239f6703: Status 404 returned error can't find the container with id 47a5f7c61344f76be200b632dbd2a34ff75a5373ecddfe85e792dbd2239f6703
Sep 10 15:15:40.325012 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:40.324988    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e0b284_688f_4bd7_a282_f9052bb11bf4.slice/crio-e478e962bd3f91142034f422e6a54e11978f66c424ab504f5ed7a6fd04cdbb08 WatchSource:0}: Error finding container e478e962bd3f91142034f422e6a54e11978f66c424ab504f5ed7a6fd04cdbb08: Status 404 returned error can't find the container with id e478e962bd3f91142034f422e6a54e11978f66c424ab504f5ed7a6fd04cdbb08
Sep 10 15:15:40.328818 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.328762    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zc2lh"
Sep 10 15:15:40.333073 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.333054    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj"
Sep 10 15:15:40.335417 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.335293    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:15:40.339942 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:40.339919    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216864e7_bd81_4a5a_a0d7_f243edea4857.slice/crio-9fb7ea5bdb55abd629d5a55e92b6894d38c1e4bfc8a827d5c8c0646b74d9e2bd WatchSource:0}: Error finding container 9fb7ea5bdb55abd629d5a55e92b6894d38c1e4bfc8a827d5c8c0646b74d9e2bd: Status 404 returned error can't find the container with id 9fb7ea5bdb55abd629d5a55e92b6894d38c1e4bfc8a827d5c8c0646b74d9e2bd
Sep 10 15:15:40.340900 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.340746    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9x6dh"
Sep 10 15:15:40.343883 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:40.343861    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ed881e_9100_40d2_b868_10d616b69e49.slice/crio-3979d86c9b61cf242b628d1191ea90f117eff62cea7bab44e1b942d4a210d5de WatchSource:0}: Error finding container 3979d86c9b61cf242b628d1191ea90f117eff62cea7bab44e1b942d4a210d5de: Status 404 returned error can't find the container with id 3979d86c9b61cf242b628d1191ea90f117eff62cea7bab44e1b942d4a210d5de
Sep 10 15:15:40.350626 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.350606    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xsmjp"
Sep 10 15:15:40.350980 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:40.350921    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9032d89_2a88_412b_ae35_5088f9eaad77.slice/crio-b9dca4d3772fdc9cb1fe5b368c8f504c1c3afb40ce0edbaa9f27520a0a5ba442 WatchSource:0}: Error finding container b9dca4d3772fdc9cb1fe5b368c8f504c1c3afb40ce0edbaa9f27520a0a5ba442: Status 404 returned error can't find the container with id b9dca4d3772fdc9cb1fe5b368c8f504c1c3afb40ce0edbaa9f27520a0a5ba442
Sep 10 15:15:40.358758 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.358741    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:15:40.361369 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:40.361333    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ab4301_eaa0_4db1_9a8d_a2ae36a8808d.slice/crio-1796af945d92cf12affdd4e9ae60693bc9062309ba8937739cb9daad2cbee5fd WatchSource:0}: Error finding container 1796af945d92cf12affdd4e9ae60693bc9062309ba8937739cb9daad2cbee5fd: Status 404 returned error can't find the container with id 1796af945d92cf12affdd4e9ae60693bc9062309ba8937739cb9daad2cbee5fd
Sep 10 15:15:40.368021 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.368000    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-krgl6"
Sep 10 15:15:40.370628 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:40.370593    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81669089_b97a_4d9f_907d_4d62da491cac.slice/crio-77280e41e6dedce9539e5b7736727953c9c4d88ac11ba08b21e6cdc5a405841d WatchSource:0}: Error finding container 77280e41e6dedce9539e5b7736727953c9c4d88ac11ba08b21e6cdc5a405841d: Status 404 returned error can't find the container with id 77280e41e6dedce9539e5b7736727953c9c4d88ac11ba08b21e6cdc5a405841d
Sep 10 15:15:40.375489 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.375333    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ql8lm"
Sep 10 15:15:40.381910 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:40.381881    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38e7952_153e_4a04_8e6c_15470498aea9.slice/crio-f7e6a5b971e4f2a5c6d85f098b2247ada422d3823bcb14b14cc25c820f2680b2 WatchSource:0}: Error finding container f7e6a5b971e4f2a5c6d85f098b2247ada422d3823bcb14b14cc25c820f2680b2: Status 404 returned error can't find the container with id f7e6a5b971e4f2a5c6d85f098b2247ada422d3823bcb14b14cc25c820f2680b2
Sep 10 15:15:40.389927 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:15:40.389901    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d07e39_b7e2_4f78_8648_4a0c188dc0f4.slice/crio-cf9544767109b1e3a1a44b4afcbe3f567e2393935ed3d99bfe60a43ceb0b90f4 WatchSource:0}: Error finding container cf9544767109b1e3a1a44b4afcbe3f567e2393935ed3d99bfe60a43ceb0b90f4: Status 404 returned error can't find the container with id cf9544767109b1e3a1a44b4afcbe3f567e2393935ed3d99bfe60a43ceb0b90f4
Sep 10 15:15:40.727690 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.727604    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:40.727690 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.727669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:40.727921 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.727769    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:15:40.727921 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.727852    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret podName:3f858dfb-38ba-44b7-8b7d-b38cfb655428 nodeName:}" failed. No retries permitted until 2026-09-10 15:15:41.727837645 +0000 UTC m=+4.143633442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret") pod "global-pull-secret-syncer-wkpgh" (UID: "3f858dfb-38ba-44b7-8b7d-b38cfb655428") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:15:40.727921 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.727769    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:15:40.728055 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.727955    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs podName:190adfd7-a800-4b59-b03c-fd0508ada6de nodeName:}" failed. No retries permitted until 2026-09-10 15:15:41.727930918 +0000 UTC m=+4.143726733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs") pod "network-metrics-daemon-5kk8k" (UID: "190adfd7-a800-4b59-b03c-fd0508ada6de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:15:40.829063 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:40.829031    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759rr\" (UniqueName: \"kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr\") pod \"network-check-target-dqbgv\" (UID: \"42483446-4d7f-4a0d-96c7-7771f0b5a14c\") " pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:40.829241 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.829206    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:15:40.829241 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.829229    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:15:40.829241 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.829238    2573 projected.go:196] Error preparing data for projected volume kube-api-access-759rr for pod openshift-network-diagnostics/network-check-target-dqbgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:15:40.829361 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:40.829292    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr podName:42483446-4d7f-4a0d-96c7-7771f0b5a14c nodeName:}" failed. No retries permitted until 2026-09-10 15:15:41.829278522 +0000 UTC m=+4.245074319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-759rr" (UniqueName: "kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr") pod "network-check-target-dqbgv" (UID: "42483446-4d7f-4a0d-96c7-7771f0b5a14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:15:41.049419 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.049345    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:10:40 +0000 UTC" deadline="2028-05-17 01:23:03.318225092 +0000 UTC"
Sep 10 15:15:41.049419 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.049375    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14746h7m22.268852955s"
Sep 10 15:15:41.136034 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.135998    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ql8lm" event={"ID":"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4","Type":"ContainerStarted","Data":"cf9544767109b1e3a1a44b4afcbe3f567e2393935ed3d99bfe60a43ceb0b90f4"}
Sep 10 15:15:41.137724 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.137663    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xsmjp" event={"ID":"b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d","Type":"ContainerStarted","Data":"1796af945d92cf12affdd4e9ae60693bc9062309ba8937739cb9daad2cbee5fd"}
Sep 10 15:15:41.140172 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.140120    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9x6dh" event={"ID":"c9032d89-2a88-412b-ae35-5088f9eaad77","Type":"ContainerStarted","Data":"b9dca4d3772fdc9cb1fe5b368c8f504c1c3afb40ce0edbaa9f27520a0a5ba442"}
Sep 10 15:15:41.144418 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.144390    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj" event={"ID":"92ed881e-9100-40d2-b868-10d616b69e49","Type":"ContainerStarted","Data":"3979d86c9b61cf242b628d1191ea90f117eff62cea7bab44e1b942d4a210d5de"}
Sep 10 15:15:41.147244 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.147216    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kqvj8" event={"ID":"04e0b284-688f-4bd7-a282-f9052bb11bf4","Type":"ContainerStarted","Data":"e478e962bd3f91142034f422e6a54e11978f66c424ab504f5ed7a6fd04cdbb08"}
Sep 10 15:15:41.148645 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.148617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7wkz" event={"ID":"c7392e59-f889-4a0e-bcf0-95a7ba7da181","Type":"ContainerStarted","Data":"47a5f7c61344f76be200b632dbd2a34ff75a5373ecddfe85e792dbd2239f6703"}
Sep 10 15:15:41.150262 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.149912    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-krgl6" event={"ID":"c38e7952-153e-4a04-8e6c-15470498aea9","Type":"ContainerStarted","Data":"f7e6a5b971e4f2a5c6d85f098b2247ada422d3823bcb14b14cc25c820f2680b2"}
Sep 10 15:15:41.151650 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.151628    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l" event={"ID":"81669089-b97a-4d9f-907d-4d62da491cac","Type":"ContainerStarted","Data":"77280e41e6dedce9539e5b7736727953c9c4d88ac11ba08b21e6cdc5a405841d"}
Sep 10 15:15:41.153010 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.152990    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zc2lh" event={"ID":"216864e7-bd81-4a5a-a0d7-f243edea4857","Type":"ContainerStarted","Data":"9fb7ea5bdb55abd629d5a55e92b6894d38c1e4bfc8a827d5c8c0646b74d9e2bd"}
Sep 10 15:15:41.737919 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.737880    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:41.738082 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.737956    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:41.738082 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:41.738079    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:15:41.738184 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:41.738139    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret podName:3f858dfb-38ba-44b7-8b7d-b38cfb655428 nodeName:}" failed. No retries permitted until 2026-09-10 15:15:43.738122293 +0000 UTC m=+6.153918102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret") pod "global-pull-secret-syncer-wkpgh" (UID: "3f858dfb-38ba-44b7-8b7d-b38cfb655428") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:15:41.738261 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:41.738206    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:15:41.738261 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:41.738239    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs podName:190adfd7-a800-4b59-b03c-fd0508ada6de nodeName:}" failed. No retries permitted until 2026-09-10 15:15:43.738229034 +0000 UTC m=+6.154024846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs") pod "network-metrics-daemon-5kk8k" (UID: "190adfd7-a800-4b59-b03c-fd0508ada6de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:15:41.839017 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:41.838983    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759rr\" (UniqueName: \"kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr\") pod \"network-check-target-dqbgv\" (UID: \"42483446-4d7f-4a0d-96c7-7771f0b5a14c\") " pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:41.839166 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:41.839144    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:15:41.839220 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:41.839173    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:15:41.839220 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:41.839187    2573 projected.go:196] Error preparing data for projected volume kube-api-access-759rr for pod openshift-network-diagnostics/network-check-target-dqbgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:15:41.839309 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:41.839239    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr podName:42483446-4d7f-4a0d-96c7-7771f0b5a14c nodeName:}" failed. No retries permitted until 2026-09-10 15:15:43.839221584 +0000 UTC m=+6.255017403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-759rr" (UniqueName: "kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr") pod "network-check-target-dqbgv" (UID: "42483446-4d7f-4a0d-96c7-7771f0b5a14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:15:42.049656 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:42.049547    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:10:40 +0000 UTC" deadline="2028-05-03 16:20:21.355447838 +0000 UTC"
Sep 10 15:15:42.049656 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:42.049609    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14425h4m39.305842549s"
Sep 10 15:15:42.120155 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:42.119924    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:42.120333 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:42.120275    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:15:42.120716 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:42.120696    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:42.120835 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:42.120813    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:15:42.121197 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:42.121178    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:42.121303 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:42.121285    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:15:42.159529 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:42.159492    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-115.ec2.internal" event={"ID":"17327a0c112603d3f5deb1d673c13162","Type":"ContainerStarted","Data":"aa1ccf6c9b74296fbfb43d0a7d3ef9f42605113182ba4c9fe9717eb4bcc5e259"}
Sep 10 15:15:42.181137 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:42.180381    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-115.ec2.internal" podStartSLOduration=3.180360591 podStartE2EDuration="3.180360591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:42.179880098 +0000 UTC m=+4.595675936" watchObservedRunningTime="2026-09-10 15:15:42.180360591 +0000 UTC m=+4.596156420"
Sep 10 15:15:43.167450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:43.167372    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal" event={"ID":"71617ef53acb8f894161ea9a17f46531","Type":"ContainerStarted","Data":"93b20e4a90fb0846aac6264670264e3deab6dba04322abfdcb5ebc0515a12682"}
Sep 10 15:15:43.755343 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:43.755304    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:43.755554 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:43.755387    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:43.755554 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:43.755500    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:15:43.755554 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:43.755555    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret podName:3f858dfb-38ba-44b7-8b7d-b38cfb655428 nodeName:}" failed. No retries permitted until 2026-09-10 15:15:47.755538071 +0000 UTC m=+10.171333874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret") pod "global-pull-secret-syncer-wkpgh" (UID: "3f858dfb-38ba-44b7-8b7d-b38cfb655428") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:15:43.755933 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:43.755912    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:15:43.756036 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:43.755965    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs podName:190adfd7-a800-4b59-b03c-fd0508ada6de nodeName:}" failed. No retries permitted until 2026-09-10 15:15:47.755951682 +0000 UTC m=+10.171747483 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs") pod "network-metrics-daemon-5kk8k" (UID: "190adfd7-a800-4b59-b03c-fd0508ada6de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:15:43.857191 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:43.856507    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759rr\" (UniqueName: \"kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr\") pod \"network-check-target-dqbgv\" (UID: \"42483446-4d7f-4a0d-96c7-7771f0b5a14c\") " pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:43.857191 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:43.856719    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:15:43.857191 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:43.856735    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:15:43.857191 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:43.856746    2573 projected.go:196] Error preparing data for projected volume kube-api-access-759rr for pod openshift-network-diagnostics/network-check-target-dqbgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:15:43.857191 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:43.856817    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr podName:42483446-4d7f-4a0d-96c7-7771f0b5a14c nodeName:}" failed. No retries permitted until 2026-09-10 15:15:47.856783496 +0000 UTC m=+10.272579304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-759rr" (UniqueName: "kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr") pod "network-check-target-dqbgv" (UID: "42483446-4d7f-4a0d-96c7-7771f0b5a14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:15:44.122239 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:44.122162    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:44.122392 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:44.122306    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:15:44.122686 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:44.122667    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:44.122782 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:44.122764    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:15:44.122867 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:44.122836    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:44.122924 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:44.122906    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:15:45.175865 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:45.174964    2573 generic.go:356] "Generic (PLEG): container finished" podID="71617ef53acb8f894161ea9a17f46531" containerID="93b20e4a90fb0846aac6264670264e3deab6dba04322abfdcb5ebc0515a12682" exitCode=0
Sep 10 15:15:45.175865 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:45.175017    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal" event={"ID":"71617ef53acb8f894161ea9a17f46531","Type":"ContainerDied","Data":"93b20e4a90fb0846aac6264670264e3deab6dba04322abfdcb5ebc0515a12682"}
Sep 10 15:15:46.122806 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:46.121956    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:46.122806 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:46.122092    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:15:46.122806 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:46.122483    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:46.122806 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:46.122572    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:15:46.122806 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:46.122659    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:46.122806 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:46.122737    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:15:47.789534 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:47.789478    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:47.789986 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:47.789580    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:47.789986 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:47.789728    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:15:47.789986 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:47.789825    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs podName:190adfd7-a800-4b59-b03c-fd0508ada6de nodeName:}" failed. No retries permitted until 2026-09-10 15:15:55.789779354 +0000 UTC m=+18.205575172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs") pod "network-metrics-daemon-5kk8k" (UID: "190adfd7-a800-4b59-b03c-fd0508ada6de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:15:47.790285 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:47.790261    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:15:47.790370 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:47.790330    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret podName:3f858dfb-38ba-44b7-8b7d-b38cfb655428 nodeName:}" failed. No retries permitted until 2026-09-10 15:15:55.790314764 +0000 UTC m=+18.206110578 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret") pod "global-pull-secret-syncer-wkpgh" (UID: "3f858dfb-38ba-44b7-8b7d-b38cfb655428") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:15:47.890340 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:47.890213    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759rr\" (UniqueName: \"kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr\") pod \"network-check-target-dqbgv\" (UID: \"42483446-4d7f-4a0d-96c7-7771f0b5a14c\") " pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:47.890523 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:47.890400    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:15:47.890523 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:47.890432    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:15:47.890523 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:47.890446    2573 projected.go:196] Error preparing data for projected volume kube-api-access-759rr for pod openshift-network-diagnostics/network-check-target-dqbgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:15:47.890523 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:47.890507    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr podName:42483446-4d7f-4a0d-96c7-7771f0b5a14c nodeName:}" failed. No retries permitted until 2026-09-10 15:15:55.890488289 +0000 UTC m=+18.306284096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-759rr" (UniqueName: "kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr") pod "network-check-target-dqbgv" (UID: "42483446-4d7f-4a0d-96c7-7771f0b5a14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:15:48.120995 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:48.120892    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:48.121794 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:48.121763    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:15:48.122153 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:48.122116    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:48.122244 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:48.122196    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:15:48.122244 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:48.122233    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:48.122345 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:48.122304    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:15:50.119682 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:50.119649    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:50.120163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:50.119649    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:50.120163 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:50.119783    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:15:50.120163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:50.119662    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:50.120163 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:50.119988    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:15:50.120163 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:50.120028    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:15:52.120145 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:52.120113    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:52.120145 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:52.120148    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:52.120657 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:52.120114    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:52.120657 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:52.120260    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:15:52.120657 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:52.120369    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:15:52.120657 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:52.120484    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:15:54.119612 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:54.119571    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:54.120050 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:54.119571    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:54.120050 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:54.119700    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:15:54.120050 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:54.119811    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:15:54.120050 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:54.119571    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:54.120050 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:54.119931    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:15:55.853147 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:55.853109    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:55.853651 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:55.853198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:55.853651 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:55.853240    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:15:55.853651 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:55.853303    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:15:55.853651 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:55.853320    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs podName:190adfd7-a800-4b59-b03c-fd0508ada6de nodeName:}" failed. No retries permitted until 2026-09-10 15:16:11.853297932 +0000 UTC m=+34.269093733 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs") pod "network-metrics-daemon-5kk8k" (UID: "190adfd7-a800-4b59-b03c-fd0508ada6de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:15:55.853651 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:55.853357    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret podName:3f858dfb-38ba-44b7-8b7d-b38cfb655428 nodeName:}" failed. No retries permitted until 2026-09-10 15:16:11.853340548 +0000 UTC m=+34.269136377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret") pod "global-pull-secret-syncer-wkpgh" (UID: "3f858dfb-38ba-44b7-8b7d-b38cfb655428") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:15:55.954207 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:55.954163    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759rr\" (UniqueName: \"kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr\") pod \"network-check-target-dqbgv\" (UID: \"42483446-4d7f-4a0d-96c7-7771f0b5a14c\") " pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:55.954383 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:55.954305    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:15:55.954383 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:55.954335    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:15:55.954383 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:55.954345    2573 projected.go:196] Error preparing data for projected volume kube-api-access-759rr for pod openshift-network-diagnostics/network-check-target-dqbgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:15:55.954516 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:55.954406    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr podName:42483446-4d7f-4a0d-96c7-7771f0b5a14c nodeName:}" failed. No retries permitted until 2026-09-10 15:16:11.954387362 +0000 UTC m=+34.370183225 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-759rr" (UniqueName: "kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr") pod "network-check-target-dqbgv" (UID: "42483446-4d7f-4a0d-96c7-7771f0b5a14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:15:56.120442 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:56.120358    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:56.120594 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:56.120367    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:56.120594 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:56.120488    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:15:56.120594 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:56.120506    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:56.120761 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:56.120610    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:15:56.120761 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:56.120696    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:15:58.120915 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:58.120878    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:15:58.121370 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:58.120987    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:15:58.121370 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:58.121041    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:15:58.121370 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:58.121117    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:15:58.121370 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:58.121217    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:15:58.121370 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:15:58.121269    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:15:59.203923 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.203830    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zc2lh" event={"ID":"216864e7-bd81-4a5a-a0d7-f243edea4857","Type":"ContainerStarted","Data":"6df810c968ba629cef2310674802e8cd2f64b5ae4d4dea0bbe5d44d39c18315b"}
Sep 10 15:15:59.205947 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.205916    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal" event={"ID":"71617ef53acb8f894161ea9a17f46531","Type":"ContainerStarted","Data":"3ae9050991f3d5e8fc6391d6f8bc9a19504b175c81bdc869e93fd7e4db0548e2"}
Sep 10 15:15:59.207393 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.207365    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ql8lm" event={"ID":"b9d07e39-b7e2-4f78-8648-4a0c188dc0f4","Type":"ContainerStarted","Data":"c96e72b43c103485bfc4f4c5b072dc636758f0aa660e4f84499b584576224cde"}
Sep 10 15:15:59.208939 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.208917    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xsmjp" event={"ID":"b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d","Type":"ContainerStarted","Data":"8f5838e972681c069cd10a52327603c44f3cd82d03aaabbab9b1ba00f05eaa45"}
Sep 10 15:15:59.210452 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.210430    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9x6dh" event={"ID":"c9032d89-2a88-412b-ae35-5088f9eaad77","Type":"ContainerStarted","Data":"79ee7c68280b55a308682f1187966f04d2dda4205f271068254667ea371333c6"}
Sep 10 15:15:59.214432 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.214402    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj" event={"ID":"92ed881e-9100-40d2-b868-10d616b69e49","Type":"ContainerStarted","Data":"efa63f3b358953da66de731508efb725d6eb4acf1cf003f4fc611257dae5e61f"}
Sep 10 15:15:59.216206 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.216187    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kqvj8" event={"ID":"04e0b284-688f-4bd7-a282-f9052bb11bf4","Type":"ContainerStarted","Data":"ec7cc659003accc00084d9f0d3f6da17bbc6c8f1d36d6d4259abf6041f31d98d"}
Sep 10 15:15:59.217897 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.217872    2573 generic.go:356] "Generic (PLEG): container finished" podID="c7392e59-f889-4a0e-bcf0-95a7ba7da181" containerID="1a18b2807837489131ff0b168f6bab7de1cd5f2b622698266b48770704b7f612" exitCode=0
Sep 10 15:15:59.217976 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.217942    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7wkz" event={"ID":"c7392e59-f889-4a0e-bcf0-95a7ba7da181","Type":"ContainerDied","Data":"1a18b2807837489131ff0b168f6bab7de1cd5f2b622698266b48770704b7f612"}
Sep 10 15:15:59.220087 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.220067    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l" event={"ID":"81669089-b97a-4d9f-907d-4d62da491cac","Type":"ContainerStarted","Data":"c90858f8170ca09f4ea5e7eb7a6b96c14c0bdb1386813b7cb9d95abe7d1486ba"}
Sep 10 15:15:59.220165 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.220142    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l" event={"ID":"81669089-b97a-4d9f-907d-4d62da491cac","Type":"ContainerStarted","Data":"8b2078f4cc92b758875c71a33c53e83dff4dd6749ba4168163f43cdeb484eadb"}
Sep 10 15:15:59.221862 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.221816    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zc2lh" podStartSLOduration=10.450169725 podStartE2EDuration="21.221768197s" totalImagesPullingTime="10.771598472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:59.220769432 +0000 UTC m=+21.636565250" watchObservedRunningTime="2026-09-10 15:15:59.221768197 +0000 UTC m=+21.637564018"
Sep 10 15:15:59.237983 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.237803    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-115.ec2.internal" podStartSLOduration=20.237773272 podStartE2EDuration="20.237773272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:59.237748467 +0000 UTC m=+21.653544287" watchObservedRunningTime="2026-09-10 15:15:59.237773272 +0000 UTC m=+21.653569092"
Sep 10 15:15:59.270770 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.270711    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ql8lm" podStartSLOduration=2.756315207 podStartE2EDuration="21.27069154s" totalImagesPullingTime="18.514376333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:59.270447779 +0000 UTC m=+21.686243617" watchObservedRunningTime="2026-09-10 15:15:59.27069154 +0000 UTC m=+21.686487360"
Sep 10 15:15:59.271062 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.271026    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xsmjp" podStartSLOduration=2.829400542 podStartE2EDuration="21.271017179s" totalImagesPullingTime="18.441616637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:59.252830192 +0000 UTC m=+21.668626012" watchObservedRunningTime="2026-09-10 15:15:59.271017179 +0000 UTC m=+21.686813031"
Sep 10 15:15:59.310219 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.310175    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kqvj8" podStartSLOduration=2.8057455620000002 podStartE2EDuration="21.310158452s" totalImagesPullingTime="18.50441289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:59.309726169 +0000 UTC m=+21.725521991" watchObservedRunningTime="2026-09-10 15:15:59.310158452 +0000 UTC m=+21.725954272"
Sep 10 15:15:59.327809 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.327751    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9x6dh" podStartSLOduration=2.847611847 podStartE2EDuration="21.327736727s" totalImagesPullingTime="18.48012488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:59.327409506 +0000 UTC m=+21.743205341" watchObservedRunningTime="2026-09-10 15:15:59.327736727 +0000 UTC m=+21.743532546"
Sep 10 15:15:59.501836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.501809    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zc2lh"
Sep 10 15:15:59.502317 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:15:59.502301    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zc2lh"
Sep 10 15:16:00.119759 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:00.119730    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:00.120066 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:00.119816    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:16:00.120066 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:00.119825    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:16:00.120066 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:00.119932    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:16:00.120066 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:00.120023    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:16:00.120285 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:00.120123    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:16:00.226334 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:00.226300    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l" event={"ID":"81669089-b97a-4d9f-907d-4d62da491cac","Type":"ContainerStarted","Data":"6d20f9261080a1f978353e2f10a71a3e9d696edac69d728c2cce451f83e45cfb"}
Sep 10 15:16:00.226706 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:00.226383    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l" event={"ID":"81669089-b97a-4d9f-907d-4d62da491cac","Type":"ContainerStarted","Data":"a90b850c759a486ad90d73dc2a2cb7ba370f41a0a02f06d129cd00c2978cf7bd"}
Sep 10 15:16:00.226706 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:00.226418    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l" event={"ID":"81669089-b97a-4d9f-907d-4d62da491cac","Type":"ContainerStarted","Data":"58bff44605057ff3ac01b512124c9c196f6f4e0b8d285366f72f0a7987d573b5"}
Sep 10 15:16:00.226706 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:00.226456    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l" event={"ID":"81669089-b97a-4d9f-907d-4d62da491cac","Type":"ContainerStarted","Data":"3e62b277418d7a726bd8a356e310386e0b8d57a4bfa66f63e9b81b070b9bd042"}
Sep 10 15:16:00.226706 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:00.226497    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zc2lh"
Sep 10 15:16:00.227381 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:00.227359    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zc2lh"
Sep 10 15:16:00.292244 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:00.292222    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 15:16:01.087962 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:01.087852    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T15:16:00.292240333Z","UUID":"6a890ea8-92e8-4845-9cdf-55aa3be31610","Handler":null,"Name":"","Endpoint":""}
Sep 10 15:16:01.089373 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:01.089348    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 15:16:01.089373 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:01.089378    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 15:16:01.230304 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:01.230270    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj" event={"ID":"92ed881e-9100-40d2-b868-10d616b69e49","Type":"ContainerStarted","Data":"59f51c8db2b2f85b61a78657a4783df5705f710ce7af27d0f91263379757325b"}
Sep 10 15:16:01.231994 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:01.231970    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-krgl6" event={"ID":"c38e7952-153e-4a04-8e6c-15470498aea9","Type":"ContainerStarted","Data":"1e292f10feab03c39de3bd23aef7602b808c1ed7a6d8e35dbd58dddaf80218ff"}
Sep 10 15:16:01.249098 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:01.249040    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-krgl6" podStartSLOduration=4.802294896 podStartE2EDuration="23.249024945s" totalImagesPullingTime="18.446730049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:16:01.248561603 +0000 UTC m=+23.664357413" watchObservedRunningTime="2026-09-10 15:16:01.249024945 +0000 UTC m=+23.664820763"
Sep 10 15:16:02.119469 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:02.119429    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:16:02.119469 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:02.119452    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:16:02.119469 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:02.119472    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:02.119758 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:02.119570    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:16:02.119758 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:02.119690    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:16:02.119758 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:02.119755    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:16:02.237109 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:02.236862    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj" event={"ID":"92ed881e-9100-40d2-b868-10d616b69e49","Type":"ContainerStarted","Data":"1a3b9a21e0735b5f8ad089b86d5fceaeb6487ca95919183bfcbf5b25683688d2"}
Sep 10 15:16:02.246494 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:02.246461    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l" event={"ID":"81669089-b97a-4d9f-907d-4d62da491cac","Type":"ContainerStarted","Data":"8b0e471cd819e3b6a4fae51601c858aed0aef540daa62ccd249e0211b4851d53"}
Sep 10 15:16:02.262257 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:02.262213    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-796zj" podStartSLOduration=11.152961985 podStartE2EDuration="24.262199176s" totalImagesPullingTime="13.109237191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:15:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:16:02.261680042 +0000 UTC m=+24.677475862" watchObservedRunningTime="2026-09-10 15:16:02.262199176 +0000 UTC m=+24.677994997"
Sep 10 15:16:04.122198 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:04.122001    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:16:04.122591 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:04.122281    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:16:04.122697 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:04.122681    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:16:04.122817 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:04.122780    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:16:04.122869 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:04.122850    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:04.122928 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:04.122912    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:16:05.255439 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:05.255404    2573 generic.go:356] "Generic (PLEG): container finished" podID="c7392e59-f889-4a0e-bcf0-95a7ba7da181" containerID="a9801b5062818440f5a325652d8563850ff823cb5bef3c6f55648ff4b956cc63" exitCode=0
Sep 10 15:16:05.256220 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:05.255499    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7wkz" event={"ID":"c7392e59-f889-4a0e-bcf0-95a7ba7da181","Type":"ContainerDied","Data":"a9801b5062818440f5a325652d8563850ff823cb5bef3c6f55648ff4b956cc63"}
Sep 10 15:16:05.259382 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:05.259362    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l" event={"ID":"81669089-b97a-4d9f-907d-4d62da491cac","Type":"ContainerStarted","Data":"22bcffae6a8f089dd2c3e02a3dd7a2adcbc4e4c505b30dc65146c79601ad04ab"}
Sep 10 15:16:05.259626 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:05.259606    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:16:05.259737 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:05.259661    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:16:05.275262 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:05.275236    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:16:05.301612 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:05.301566    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l" podStartSLOduration=8.795345065 podStartE2EDuration="27.301550296s" totalImagesPullingTime="18.506205231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:16:05.301155849 +0000 UTC m=+27.716951668" watchObservedRunningTime="2026-09-10 15:16:05.301550296 +0000 UTC m=+27.717346115"
Sep 10 15:16:06.119908 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:06.119708    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:06.120040 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:06.119761    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:16:06.120040 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:06.119995    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:16:06.120130 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:06.119826    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:16:06.120130 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:06.120057    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:16:06.120193 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:06.120159    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:16:06.262167 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:06.262138    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:16:06.278079 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:06.278051    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:16:06.558079 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:06.558049    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wkpgh"]
Sep 10 15:16:06.558238 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:06.558173    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:16:06.558300 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:06.558282    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:16:06.560979 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:06.560951    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dqbgv"]
Sep 10 15:16:06.561099 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:06.561048    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:06.561182 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:06.561161    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:16:06.561540 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:06.561517    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5kk8k"]
Sep 10 15:16:06.561637 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:06.561624    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:16:06.561741 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:06.561709    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:16:07.266171 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:07.266136    2573 generic.go:356] "Generic (PLEG): container finished" podID="c7392e59-f889-4a0e-bcf0-95a7ba7da181" containerID="aece61d49d3047fe704fd2d3b9b89f0ac81573d77ef9822cdb241d2d35ddbe44" exitCode=0
Sep 10 15:16:07.266556 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:07.266226    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7wkz" event={"ID":"c7392e59-f889-4a0e-bcf0-95a7ba7da181","Type":"ContainerDied","Data":"aece61d49d3047fe704fd2d3b9b89f0ac81573d77ef9822cdb241d2d35ddbe44"}
Sep 10 15:16:08.120403 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:08.120368    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:08.120506 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:08.120456    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:16:08.120584 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:08.120565    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:16:08.120707 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:08.120686    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:16:08.120771 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:08.120723    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:16:08.120902 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:08.120829    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:16:08.270852 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:08.270818    2573 generic.go:356] "Generic (PLEG): container finished" podID="c7392e59-f889-4a0e-bcf0-95a7ba7da181" containerID="2463eaf292799dc1a8cdbbf063af36e34cd264236f2b86412c138f7dd727cff6" exitCode=0
Sep 10 15:16:08.271221 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:08.270889    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7wkz" event={"ID":"c7392e59-f889-4a0e-bcf0-95a7ba7da181","Type":"ContainerDied","Data":"2463eaf292799dc1a8cdbbf063af36e34cd264236f2b86412c138f7dd727cff6"}
Sep 10 15:16:10.120239 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:10.119696    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:16:10.120239 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:10.119726    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:16:10.120239 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:10.119699    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:10.120239 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:10.119863    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:16:10.120239 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:10.119937    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:16:10.120239 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:10.120095    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:16:11.868384 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:11.868347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:16:11.868913 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:11.868418    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:16:11.868913 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:11.868516    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:16:11.868913 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:11.868528    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:16:11.868913 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:11.868583    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs podName:190adfd7-a800-4b59-b03c-fd0508ada6de nodeName:}" failed. No retries permitted until 2026-09-10 15:16:43.868568169 +0000 UTC m=+66.284363971 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs") pod "network-metrics-daemon-5kk8k" (UID: "190adfd7-a800-4b59-b03c-fd0508ada6de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:16:11.868913 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:11.868600    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret podName:3f858dfb-38ba-44b7-8b7d-b38cfb655428 nodeName:}" failed. No retries permitted until 2026-09-10 15:16:43.868591834 +0000 UTC m=+66.284387632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret") pod "global-pull-secret-syncer-wkpgh" (UID: "3f858dfb-38ba-44b7-8b7d-b38cfb655428") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:16:11.968945 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:11.968896    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759rr\" (UniqueName: \"kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr\") pod \"network-check-target-dqbgv\" (UID: \"42483446-4d7f-4a0d-96c7-7771f0b5a14c\") " pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:11.969136 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:11.969086    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:16:11.969136 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:11.969106    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:16:11.969136 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:11.969118    2573 projected.go:196] Error preparing data for projected volume kube-api-access-759rr for pod openshift-network-diagnostics/network-check-target-dqbgv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:16:11.969261 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:11.969165    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr podName:42483446-4d7f-4a0d-96c7-7771f0b5a14c nodeName:}" failed. No retries permitted until 2026-09-10 15:16:43.969151824 +0000 UTC m=+66.384947620 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-759rr" (UniqueName: "kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr") pod "network-check-target-dqbgv" (UID: "42483446-4d7f-4a0d-96c7-7771f0b5a14c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:16:12.120010 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.119977    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:12.120210 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:12.120102    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqbgv" podUID="42483446-4d7f-4a0d-96c7-7771f0b5a14c"
Sep 10 15:16:12.120210 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.119977    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:16:12.120210 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:12.120200    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kk8k" podUID="190adfd7-a800-4b59-b03c-fd0508ada6de"
Sep 10 15:16:12.120210 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.119977    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:16:12.120428 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:12.120266    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkpgh" podUID="3f858dfb-38ba-44b7-8b7d-b38cfb655428"
Sep 10 15:16:12.844925 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.844890    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-115.ec2.internal" event="NodeReady"
Sep 10 15:16:12.845108 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.845056    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 15:16:12.881271 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.881228    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"]
Sep 10 15:16:12.913882 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.911406    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78"]
Sep 10 15:16:12.914982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.914647    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"
Sep 10 15:16:12.917226 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.917199    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 15:16:12.917635 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.917602    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 15:16:12.918014 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.917990    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 15:16:12.918574 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.918551    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 15:16:12.929011 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.928987    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-579df544f9-wlzfv"]
Sep 10 15:16:12.929126 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.929114    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78"
Sep 10 15:16:12.931005 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.930985    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 15:16:12.931127 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.931055    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jtm4b\""
Sep 10 15:16:12.949053 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.949028    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"]
Sep 10 15:16:12.949182 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.949160    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:12.951234 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.951205    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 15:16:12.951628 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.951608    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sxzrd\""
Sep 10 15:16:12.951933 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.951915    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 15:16:12.952495 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.952356    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 15:16:12.956643 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.956624    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 15:16:12.969176 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.969149    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"]
Sep 10 15:16:12.969286 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.969192    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78"]
Sep 10 15:16:12.969286 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.969213    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z8tnd"]
Sep 10 15:16:12.969369 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.969283    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:12.971521 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.971499    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 15:16:12.971694 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.971673    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 15:16:12.971768 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.971732    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 15:16:12.971863 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.971662    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 15:16:12.977035 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.977014    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6dbef59e-d171-4b46-a0ff-ac4bc165bff3-klusterlet-config\") pod \"klusterlet-addon-workmgr-69848b66d9-4wzll\" (UID: \"6dbef59e-d171-4b46-a0ff-ac4bc165bff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"
Sep 10 15:16:12.977149 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.977060    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgvf2\" (UniqueName: \"kubernetes.io/projected/6dbef59e-d171-4b46-a0ff-ac4bc165bff3-kube-api-access-rgvf2\") pod \"klusterlet-addon-workmgr-69848b66d9-4wzll\" (UID: \"6dbef59e-d171-4b46-a0ff-ac4bc165bff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"
Sep 10 15:16:12.977149 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.977126    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dbef59e-d171-4b46-a0ff-ac4bc165bff3-tmp\") pod \"klusterlet-addon-workmgr-69848b66d9-4wzll\" (UID: \"6dbef59e-d171-4b46-a0ff-ac4bc165bff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"
Sep 10 15:16:12.990126 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.990096    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wmlgv"]
Sep 10 15:16:12.990261 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.990233    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z8tnd"
Sep 10 15:16:12.992470 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.992446    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 15:16:12.992470 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.992464    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-46mx6\""
Sep 10 15:16:12.992644 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.992571    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 15:16:12.992644 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:12.992613    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 15:16:13.007769 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.007736    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-579df544f9-wlzfv"]
Sep 10 15:16:13.007920 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.007782    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z8tnd"]
Sep 10 15:16:13.007920 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.007823    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"]
Sep 10 15:16:13.007920 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.007846    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wmlgv"]
Sep 10 15:16:13.007920 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.007870    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:13.010027 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.010005    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 15:16:13.010196 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.010150    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 15:16:13.010281 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.010235    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4q2px\""
Sep 10 15:16:13.077884 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.077847    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a33951-8222-4e14-9630-2b4b11140aaf-tmp-dir\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:13.078083 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.077894    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg22n\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-kube-api-access-qg22n\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.078083 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.077987    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dbef59e-d171-4b46-a0ff-ac4bc165bff3-tmp\") pod \"klusterlet-addon-workmgr-69848b66d9-4wzll\" (UID: \"6dbef59e-d171-4b46-a0ff-ac4bc165bff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"
Sep 10 15:16:13.078083 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078029    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcd493f9-5da2-403c-b48f-c32fb81e4571-ca-trust-extracted\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.078083 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078060    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-certificates\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.078290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078099    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a33951-8222-4e14-9630-2b4b11140aaf-config-volume\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:13.078290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078121    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fed2f3de-c85c-46c3-959a-aeebd24257e4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-578d8b768b-hzv78\" (UID: \"fed2f3de-c85c-46c3-959a-aeebd24257e4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78"
Sep 10 15:16:13.078290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078147    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sxsd\" (UniqueName: \"kubernetes.io/projected/fed2f3de-c85c-46c3-959a-aeebd24257e4-kube-api-access-7sxsd\") pod \"managed-serviceaccount-addon-agent-578d8b768b-hzv78\" (UID: \"fed2f3de-c85c-46c3-959a-aeebd24257e4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78"
Sep 10 15:16:13.078290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078181    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:13.078290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078198    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbmx7\" (UniqueName: \"kubernetes.io/projected/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-kube-api-access-mbmx7\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.078290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078236    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-ca\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.078290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078276    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcd493f9-5da2-403c-b48f-c32fb81e4571-installation-pull-secrets\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.078634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078305    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-bound-sa-token\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.078634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078384    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vmwt\" (UniqueName: \"kubernetes.io/projected/07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a-kube-api-access-2vmwt\") pod \"ingress-canary-z8tnd\" (UID: \"07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a\") " pod="openshift-ingress-canary/ingress-canary-z8tnd"
Sep 10 15:16:13.078634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078404    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfw7q\" (UniqueName: \"kubernetes.io/projected/d6a33951-8222-4e14-9630-2b4b11140aaf-kube-api-access-pfw7q\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:13.078634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078405    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dbef59e-d171-4b46-a0ff-ac4bc165bff3-tmp\") pod \"klusterlet-addon-workmgr-69848b66d9-4wzll\" (UID: \"6dbef59e-d171-4b46-a0ff-ac4bc165bff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"
Sep 10 15:16:13.078634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078418    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcd493f9-5da2-403c-b48f-c32fb81e4571-image-registry-private-configuration\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.078634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078449    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.078634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6dbef59e-d171-4b46-a0ff-ac4bc165bff3-klusterlet-config\") pod \"klusterlet-addon-workmgr-69848b66d9-4wzll\" (UID: \"6dbef59e-d171-4b46-a0ff-ac4bc165bff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"
Sep 10 15:16:13.078634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078535    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.078634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078558    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.078634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078575    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.078634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078604    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-hub\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.079176 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078644    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgvf2\" (UniqueName: \"kubernetes.io/projected/6dbef59e-d171-4b46-a0ff-ac4bc165bff3-kube-api-access-rgvf2\") pod \"klusterlet-addon-workmgr-69848b66d9-4wzll\" (UID: \"6dbef59e-d171-4b46-a0ff-ac4bc165bff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"
Sep 10 15:16:13.079176 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078733    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcd493f9-5da2-403c-b48f-c32fb81e4571-trusted-ca\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.079176 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.078758    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a-cert\") pod \"ingress-canary-z8tnd\" (UID: \"07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a\") " pod="openshift-ingress-canary/ingress-canary-z8tnd"
Sep 10 15:16:13.083351 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.083328    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6dbef59e-d171-4b46-a0ff-ac4bc165bff3-klusterlet-config\") pod \"klusterlet-addon-workmgr-69848b66d9-4wzll\" (UID: \"6dbef59e-d171-4b46-a0ff-ac4bc165bff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"
Sep 10 15:16:13.087268 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.087240    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgvf2\" (UniqueName: \"kubernetes.io/projected/6dbef59e-d171-4b46-a0ff-ac4bc165bff3-kube-api-access-rgvf2\") pod \"klusterlet-addon-workmgr-69848b66d9-4wzll\" (UID: \"6dbef59e-d171-4b46-a0ff-ac4bc165bff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"
Sep 10 15:16:13.179681 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.179597    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcd493f9-5da2-403c-b48f-c32fb81e4571-ca-trust-extracted\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.179681 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.179630    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-certificates\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.179681 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.179656    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a33951-8222-4e14-9630-2b4b11140aaf-config-volume\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:13.179681 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.179681    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fed2f3de-c85c-46c3-959a-aeebd24257e4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-578d8b768b-hzv78\" (UID: \"fed2f3de-c85c-46c3-959a-aeebd24257e4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78"
Sep 10 15:16:13.179982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.179706    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sxsd\" (UniqueName: \"kubernetes.io/projected/fed2f3de-c85c-46c3-959a-aeebd24257e4-kube-api-access-7sxsd\") pod \"managed-serviceaccount-addon-agent-578d8b768b-hzv78\" (UID: \"fed2f3de-c85c-46c3-959a-aeebd24257e4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78"
Sep 10 15:16:13.179982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.179734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:13.179982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.179757    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbmx7\" (UniqueName: \"kubernetes.io/projected/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-kube-api-access-mbmx7\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.179982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.179783    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-ca\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.179982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.179830    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcd493f9-5da2-403c-b48f-c32fb81e4571-installation-pull-secrets\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.179982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.179855    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-bound-sa-token\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.179982 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:13.179889    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:16:13.179982 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:13.179962    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls podName:d6a33951-8222-4e14-9630-2b4b11140aaf nodeName:}" failed. No retries permitted until 2026-09-10 15:16:13.679940442 +0000 UTC m=+36.095736256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls") pod "dns-default-wmlgv" (UID: "d6a33951-8222-4e14-9630-2b4b11140aaf") : secret "dns-default-metrics-tls" not found
Sep 10 15:16:13.180283 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180069    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcd493f9-5da2-403c-b48f-c32fb81e4571-ca-trust-extracted\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.180283 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.179898    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vmwt\" (UniqueName: \"kubernetes.io/projected/07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a-kube-api-access-2vmwt\") pod \"ingress-canary-z8tnd\" (UID: \"07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a\") " pod="openshift-ingress-canary/ingress-canary-z8tnd"
Sep 10 15:16:13.180283 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180114    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfw7q\" (UniqueName: \"kubernetes.io/projected/d6a33951-8222-4e14-9630-2b4b11140aaf-kube-api-access-pfw7q\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:13.180283 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180143    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcd493f9-5da2-403c-b48f-c32fb81e4571-image-registry-private-configuration\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.180283 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180172    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.180283 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180215    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.180283 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180240    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.180283 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180252    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a33951-8222-4e14-9630-2b4b11140aaf-config-volume\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:13.180283 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180263    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.180804 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-hub\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.180804 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180365    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcd493f9-5da2-403c-b48f-c32fb81e4571-trusted-ca\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.180804 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180394    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a-cert\") pod \"ingress-canary-z8tnd\" (UID: \"07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a\") " pod="openshift-ingress-canary/ingress-canary-z8tnd"
Sep 10 15:16:13.180804 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180418    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a33951-8222-4e14-9630-2b4b11140aaf-tmp-dir\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:13.180804 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180452    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg22n\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-kube-api-access-qg22n\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.180804 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.180689    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-certificates\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.180804 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:13.180740    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:16:13.180804 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:13.180751    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579df544f9-wlzfv: secret "image-registry-tls" not found
Sep 10 15:16:13.181325 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:13.180874    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls podName:bcd493f9-5da2-403c-b48f-c32fb81e4571 nodeName:}" failed. No retries permitted until 2026-09-10 15:16:13.680850823 +0000 UTC m=+36.096646626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls") pod "image-registry-579df544f9-wlzfv" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571") : secret "image-registry-tls" not found
Sep 10 15:16:13.181846 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.181819    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcd493f9-5da2-403c-b48f-c32fb81e4571-trusted-ca\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.183558 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.183531    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fed2f3de-c85c-46c3-959a-aeebd24257e4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-578d8b768b-hzv78\" (UID: \"fed2f3de-c85c-46c3-959a-aeebd24257e4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78"
Sep 10 15:16:13.183654 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.183636    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.183839 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.183817    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a33951-8222-4e14-9630-2b4b11140aaf-tmp-dir\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:13.183920 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.183887    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcd493f9-5da2-403c-b48f-c32fb81e4571-image-registry-private-configuration\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.184375 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.184351    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-ca\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.184447 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.184407    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.184785 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.184761    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a-cert\") pod \"ingress-canary-z8tnd\" (UID: \"07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a\") " pod="openshift-ingress-canary/ingress-canary-z8tnd"
Sep 10 15:16:13.185032 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.185008    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcd493f9-5da2-403c-b48f-c32fb81e4571-installation-pull-secrets\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.185251 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.185234    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-hub\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.186025 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.186003    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.191255 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.191230    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sxsd\" (UniqueName: \"kubernetes.io/projected/fed2f3de-c85c-46c3-959a-aeebd24257e4-kube-api-access-7sxsd\") pod \"managed-serviceaccount-addon-agent-578d8b768b-hzv78\" (UID: \"fed2f3de-c85c-46c3-959a-aeebd24257e4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78"
Sep 10 15:16:13.191759 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.191738    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vmwt\" (UniqueName: \"kubernetes.io/projected/07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a-kube-api-access-2vmwt\") pod \"ingress-canary-z8tnd\" (UID: \"07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a\") " pod="openshift-ingress-canary/ingress-canary-z8tnd"
Sep 10 15:16:13.191887 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.191775    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-bound-sa-token\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.192782 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.192760    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg22n\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-kube-api-access-qg22n\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.193108 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.193083    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbmx7\" (UniqueName: \"kubernetes.io/projected/d540c9c4-a41f-45c2-bbbd-f52c5bee38a1-kube-api-access-mbmx7\") pod \"cluster-proxy-proxy-agent-54454bfbc-p548k\" (UID: \"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.195179 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.195157    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfw7q\" (UniqueName: \"kubernetes.io/projected/d6a33951-8222-4e14-9630-2b4b11140aaf-kube-api-access-pfw7q\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:13.229206 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.229176    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"
Sep 10 15:16:13.243905 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.243878    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78"
Sep 10 15:16:13.279849 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.279823    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:16:13.313856 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.313822    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z8tnd"
Sep 10 15:16:13.684155 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.684081    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:13.684407 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:13.684275    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:16:13.684407 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:13.684408    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579df544f9-wlzfv: secret "image-registry-tls" not found
Sep 10 15:16:13.684534 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.684424    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:13.684534 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:13.684458    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls podName:bcd493f9-5da2-403c-b48f-c32fb81e4571 nodeName:}" failed. No retries permitted until 2026-09-10 15:16:14.684443051 +0000 UTC m=+37.100238847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls") pod "image-registry-579df544f9-wlzfv" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571") : secret "image-registry-tls" not found
Sep 10 15:16:13.684534 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:13.684532    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:16:13.684662 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:13.684591    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls podName:d6a33951-8222-4e14-9630-2b4b11140aaf nodeName:}" failed. No retries permitted until 2026-09-10 15:16:14.684575112 +0000 UTC m=+37.100370931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls") pod "dns-default-wmlgv" (UID: "d6a33951-8222-4e14-9630-2b4b11140aaf") : secret "dns-default-metrics-tls" not found
Sep 10 15:16:13.703665 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.703532    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z8tnd"]
Sep 10 15:16:13.705511 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.705485    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78"]
Sep 10 15:16:13.706516 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.706456    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"]
Sep 10 15:16:13.707158 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:13.707136    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"]
Sep 10 15:16:13.708194 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:16:13.708174    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbef59e_d171_4b46_a0ff_ac4bc165bff3.slice/crio-e0071773ae60dda180b83675696b6bed987674734ca53d292f5af167ab162ec0 WatchSource:0}: Error finding container e0071773ae60dda180b83675696b6bed987674734ca53d292f5af167ab162ec0: Status 404 returned error can't find the container with id e0071773ae60dda180b83675696b6bed987674734ca53d292f5af167ab162ec0
Sep 10 15:16:13.709030 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:16:13.709007    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfed2f3de_c85c_46c3_959a_aeebd24257e4.slice/crio-d057668b2e3e0f7802b7747297c77da9b745d24200083aab89a226fcb6a25cb5 WatchSource:0}: Error finding container d057668b2e3e0f7802b7747297c77da9b745d24200083aab89a226fcb6a25cb5: Status 404 returned error can't find the container with id d057668b2e3e0f7802b7747297c77da9b745d24200083aab89a226fcb6a25cb5
Sep 10 15:16:13.709656 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:16:13.709620    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd540c9c4_a41f_45c2_bbbd_f52c5bee38a1.slice/crio-6ac4cb6448acd4e9c4b3eccbece58db9ef2ce3a172a8056106dfc1f04225af74 WatchSource:0}: Error finding container 6ac4cb6448acd4e9c4b3eccbece58db9ef2ce3a172a8056106dfc1f04225af74: Status 404 returned error can't find the container with id 6ac4cb6448acd4e9c4b3eccbece58db9ef2ce3a172a8056106dfc1f04225af74
Sep 10 15:16:13.710739 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:16:13.710721    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b0c4f8_0d00_40cc_a8b6_eb84f9546b3a.slice/crio-7f4e2d658f2f93d50b47c13777b2ea8071a43c5a31ac2b1da04f64fbbc538674 WatchSource:0}: Error finding container 7f4e2d658f2f93d50b47c13777b2ea8071a43c5a31ac2b1da04f64fbbc538674: Status 404 returned error can't find the container with id 7f4e2d658f2f93d50b47c13777b2ea8071a43c5a31ac2b1da04f64fbbc538674
Sep 10 15:16:14.120085 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.120044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:16:14.120851 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.120044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:16:14.120851 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.120044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:14.124443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.124242    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 15:16:14.124443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.124315    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 15:16:14.124597 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.124321    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-96bxt\""
Sep 10 15:16:14.124597 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.124338    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d5wmh\""
Sep 10 15:16:14.124597 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.124338    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 15:16:14.124597 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.124382    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 15:16:14.289322 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.289235    2573 generic.go:356] "Generic (PLEG): container finished" podID="c7392e59-f889-4a0e-bcf0-95a7ba7da181" containerID="bf2f3cc399074a2699bde1c2abe1986b1800daf24b4f1c0e0845a68906524036" exitCode=0
Sep 10 15:16:14.289322 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.289304    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7wkz" event={"ID":"c7392e59-f889-4a0e-bcf0-95a7ba7da181","Type":"ContainerDied","Data":"bf2f3cc399074a2699bde1c2abe1986b1800daf24b4f1c0e0845a68906524036"}
Sep 10 15:16:14.290523 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.290498    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z8tnd" event={"ID":"07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a","Type":"ContainerStarted","Data":"7f4e2d658f2f93d50b47c13777b2ea8071a43c5a31ac2b1da04f64fbbc538674"}
Sep 10 15:16:14.291710 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.291689    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll" event={"ID":"6dbef59e-d171-4b46-a0ff-ac4bc165bff3","Type":"ContainerStarted","Data":"e0071773ae60dda180b83675696b6bed987674734ca53d292f5af167ab162ec0"}
Sep 10 15:16:14.292872 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.292845    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78" event={"ID":"fed2f3de-c85c-46c3-959a-aeebd24257e4","Type":"ContainerStarted","Data":"d057668b2e3e0f7802b7747297c77da9b745d24200083aab89a226fcb6a25cb5"}
Sep 10 15:16:14.293919 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.293900    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k" event={"ID":"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1","Type":"ContainerStarted","Data":"6ac4cb6448acd4e9c4b3eccbece58db9ef2ce3a172a8056106dfc1f04225af74"}
Sep 10 15:16:14.697029 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.696060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:14.697029 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:14.696200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:14.697029 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:14.696331    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:16:14.697029 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:14.696394    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls podName:d6a33951-8222-4e14-9630-2b4b11140aaf nodeName:}" failed. No retries permitted until 2026-09-10 15:16:16.696374884 +0000 UTC m=+39.112170684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls") pod "dns-default-wmlgv" (UID: "d6a33951-8222-4e14-9630-2b4b11140aaf") : secret "dns-default-metrics-tls" not found
Sep 10 15:16:14.697029 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:14.696878    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:16:14.697029 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:14.696898    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579df544f9-wlzfv: secret "image-registry-tls" not found
Sep 10 15:16:14.697029 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:14.696947    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls podName:bcd493f9-5da2-403c-b48f-c32fb81e4571 nodeName:}" failed. No retries permitted until 2026-09-10 15:16:16.696931869 +0000 UTC m=+39.112727668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls") pod "image-registry-579df544f9-wlzfv" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571") : secret "image-registry-tls" not found
Sep 10 15:16:15.307102 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:15.307063    2573 generic.go:356] "Generic (PLEG): container finished" podID="c7392e59-f889-4a0e-bcf0-95a7ba7da181" containerID="e5acacbca0691c7c6ad2bdc5d1b6def4b68372ee6a98f37f08679483c688e913" exitCode=0
Sep 10 15:16:15.307539 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:15.307131    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7wkz" event={"ID":"c7392e59-f889-4a0e-bcf0-95a7ba7da181","Type":"ContainerDied","Data":"e5acacbca0691c7c6ad2bdc5d1b6def4b68372ee6a98f37f08679483c688e913"}
Sep 10 15:16:16.322194 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:16.322151    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7wkz" event={"ID":"c7392e59-f889-4a0e-bcf0-95a7ba7da181","Type":"ContainerStarted","Data":"68d9b6fbb877ff862d4bc54caf888df26a4b571e12994f5cea241cbb555efb0a"}
Sep 10 15:16:16.346265 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:16.346022    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x7wkz" podStartSLOduration=7.39007701 podStartE2EDuration="38.346000087s" totalImagesPullingTime="30.955923077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:15:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:16:16.345233747 +0000 UTC m=+38.761029588" watchObservedRunningTime="2026-09-10 15:16:16.346000087 +0000 UTC m=+38.761795907"
Sep 10 15:16:16.716310 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:16.716278    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:16.716501 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:16.716349    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:16.716501 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:16.716480    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:16:16.716501 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:16.716494    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579df544f9-wlzfv: secret "image-registry-tls" not found
Sep 10 15:16:16.716649 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:16.716557    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls podName:bcd493f9-5da2-403c-b48f-c32fb81e4571 nodeName:}" failed. No retries permitted until 2026-09-10 15:16:20.716536787 +0000 UTC m=+43.132332800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls") pod "image-registry-579df544f9-wlzfv" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571") : secret "image-registry-tls" not found
Sep 10 15:16:16.717001 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:16.716978    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:16:16.717122 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:16.717036    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls podName:d6a33951-8222-4e14-9630-2b4b11140aaf nodeName:}" failed. No retries permitted until 2026-09-10 15:16:20.717019461 +0000 UTC m=+43.132815271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls") pod "dns-default-wmlgv" (UID: "d6a33951-8222-4e14-9630-2b4b11140aaf") : secret "dns-default-metrics-tls" not found
Sep 10 15:16:20.751599 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:20.751556    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:20.752082 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:20.751660    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:20.752082 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:20.751756    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:16:20.752082 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:20.751778    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:16:20.752082 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:20.751781    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579df544f9-wlzfv: secret "image-registry-tls" not found
Sep 10 15:16:20.752082 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:20.751869    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls podName:d6a33951-8222-4e14-9630-2b4b11140aaf nodeName:}" failed. No retries permitted until 2026-09-10 15:16:28.751849716 +0000 UTC m=+51.167645513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls") pod "dns-default-wmlgv" (UID: "d6a33951-8222-4e14-9630-2b4b11140aaf") : secret "dns-default-metrics-tls" not found
Sep 10 15:16:20.752082 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:20.751889    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls podName:bcd493f9-5da2-403c-b48f-c32fb81e4571 nodeName:}" failed. No retries permitted until 2026-09-10 15:16:28.75187968 +0000 UTC m=+51.167675478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls") pod "image-registry-579df544f9-wlzfv" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571") : secret "image-registry-tls" not found
Sep 10 15:16:23.343284 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:23.343252    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll" event={"ID":"6dbef59e-d171-4b46-a0ff-ac4bc165bff3","Type":"ContainerStarted","Data":"20df446b8191631392e73e343221c03152c24561af3ece999f317c72bdf46874"}
Sep 10 15:16:23.343726 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:23.343438    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"
Sep 10 15:16:23.345114 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:23.345069    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78" event={"ID":"fed2f3de-c85c-46c3-959a-aeebd24257e4","Type":"ContainerStarted","Data":"9000e83e36af589eb02c6b1bb2e2e30538ac03d98b579bd358c974b66e251feb"}
Sep 10 15:16:23.345482 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:23.345460    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll"
Sep 10 15:16:23.347257 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:23.347236    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k" event={"ID":"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1","Type":"ContainerStarted","Data":"53b1763e1f5e7003ab23b33aa386886bd0270e02daa5acc7ad369446ad1606e6"}
Sep 10 15:16:23.347365 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:23.347268    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k" event={"ID":"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1","Type":"ContainerStarted","Data":"623548214ee96d89b79114335ea46208a3c7def97d725550c4b98d2209d075d1"}
Sep 10 15:16:23.347365 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:23.347284    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k" event={"ID":"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1","Type":"ContainerStarted","Data":"3a9cfab44109409048237d6c4101acaec1eca8d5b7d8f38c3afa3b275f78d265"}
Sep 10 15:16:23.348628 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:23.348590    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z8tnd" event={"ID":"07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a","Type":"ContainerStarted","Data":"444079f74775ab442f375cbf372f3442154f4334c9e61a119e590e8589b77195"}
Sep 10 15:16:23.389536 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:23.389484    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69848b66d9-4wzll" podStartSLOduration=34.369367991 podStartE2EDuration="43.389469056s" totalImagesPullingTime="9.020101065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:16:23.372511509 +0000 UTC m=+45.788307329" watchObservedRunningTime="2026-09-10 15:16:23.389469056 +0000 UTC m=+45.805264875"
Sep 10 15:16:23.406167 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:23.406124    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z8tnd" podStartSLOduration=2.388656409 podStartE2EDuration="11.406114328s" totalImagesPullingTime="9.017457919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:16:23.40571073 +0000 UTC m=+45.821506553" watchObservedRunningTime="2026-09-10 15:16:23.406114328 +0000 UTC m=+45.821910137"
Sep 10 15:16:23.425198 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:23.425141    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k" podStartSLOduration=34.407742539 podStartE2EDuration="43.425119355s" totalImagesPullingTime="9.017376816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:16:23.423733331 +0000 UTC m=+45.839529151" watchObservedRunningTime="2026-09-10 15:16:23.425119355 +0000 UTC m=+45.840915174"
Sep 10 15:16:23.442415 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:23.442369    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578d8b768b-hzv78" podStartSLOduration=34.417738075 podStartE2EDuration="43.442351895s" totalImagesPullingTime="9.02461382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:16:23.44135145 +0000 UTC m=+45.857147270" watchObservedRunningTime="2026-09-10 15:16:23.442351895 +0000 UTC m=+45.858147714"
Sep 10 15:16:28.817950 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:28.817894    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:28.818357 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:28.818029    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:28.818357 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:28.818058    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:16:28.818357 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:28.818077    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579df544f9-wlzfv: secret "image-registry-tls" not found
Sep 10 15:16:28.818357 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:28.818146    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls podName:bcd493f9-5da2-403c-b48f-c32fb81e4571 nodeName:}" failed. No retries permitted until 2026-09-10 15:16:44.818130908 +0000 UTC m=+67.233926706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls") pod "image-registry-579df544f9-wlzfv" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571") : secret "image-registry-tls" not found
Sep 10 15:16:28.818357 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:28.818190    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:16:28.818357 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:28.818266    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls podName:d6a33951-8222-4e14-9630-2b4b11140aaf nodeName:}" failed. No retries permitted until 2026-09-10 15:16:44.81825085 +0000 UTC m=+67.234046650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls") pod "dns-default-wmlgv" (UID: "d6a33951-8222-4e14-9630-2b4b11140aaf") : secret "dns-default-metrics-tls" not found
Sep 10 15:16:38.282312 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.282278    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mcn9l"
Sep 10 15:16:38.291347 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.291317    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dl4c4"]
Sep 10 15:16:38.295942 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.295917    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.297877 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.297858    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 15:16:38.297877 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.297870    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 15:16:38.298219 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.298203    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 15:16:38.298366 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.298349    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-696sd\""
Sep 10 15:16:38.301714 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.301696    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 15:16:38.311658 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.311635    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dl4c4"]
Sep 10 15:16:38.387557 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.387507    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xxk\" (UniqueName: \"kubernetes.io/projected/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-api-access-x6xxk\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.387742 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.387572    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.387742 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.387598    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b12f58a1-a40c-4024-8411-2334e8d0244f-crio-socket\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.387742 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.387680    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.387896 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.387779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b12f58a1-a40c-4024-8411-2334e8d0244f-data-volume\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.488713 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.488682    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.488892 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.488740    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b12f58a1-a40c-4024-8411-2334e8d0244f-data-volume\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.488892 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:38.488825    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:16:38.488892 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.488845    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xxk\" (UniqueName: \"kubernetes.io/projected/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-api-access-x6xxk\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.488892 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.488885    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.489107 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:38.488899    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm podName:b12f58a1-a40c-4024-8411-2334e8d0244f nodeName:}" failed. No retries permitted until 2026-09-10 15:16:38.9888805 +0000 UTC m=+61.404676316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl4c4" (UID: "b12f58a1-a40c-4024-8411-2334e8d0244f") : configmap "kube-rbac-proxy" not found
Sep 10 15:16:38.489107 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.488938    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b12f58a1-a40c-4024-8411-2334e8d0244f-crio-socket\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.489107 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:38.488965    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:16:38.489107 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:38.489003    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls podName:b12f58a1-a40c-4024-8411-2334e8d0244f nodeName:}" failed. No retries permitted until 2026-09-10 15:16:38.988991571 +0000 UTC m=+61.404787382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl4c4" (UID: "b12f58a1-a40c-4024-8411-2334e8d0244f") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:16:38.489107 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.489033    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b12f58a1-a40c-4024-8411-2334e8d0244f-crio-socket\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.489363 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.489110    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b12f58a1-a40c-4024-8411-2334e8d0244f-data-volume\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.498401 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.498371    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xxk\" (UniqueName: \"kubernetes.io/projected/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-api-access-x6xxk\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.991537 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.991503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.991709 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:38.991600    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:38.991709 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:38.991641    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:16:38.991709 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:38.991688    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:16:38.991839 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:38.991754    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls podName:b12f58a1-a40c-4024-8411-2334e8d0244f nodeName:}" failed. No retries permitted until 2026-09-10 15:16:39.991740289 +0000 UTC m=+62.407536099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl4c4" (UID: "b12f58a1-a40c-4024-8411-2334e8d0244f") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:16:38.991839 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:38.991767    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm podName:b12f58a1-a40c-4024-8411-2334e8d0244f nodeName:}" failed. No retries permitted until 2026-09-10 15:16:39.991760389 +0000 UTC m=+62.407556186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl4c4" (UID: "b12f58a1-a40c-4024-8411-2334e8d0244f") : configmap "kube-rbac-proxy" not found
Sep 10 15:16:39.998956 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:39.998914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:39.999406 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:39.999021    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:39.999406 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:39.999048    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:16:39.999406 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:39.999119    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm podName:b12f58a1-a40c-4024-8411-2334e8d0244f nodeName:}" failed. No retries permitted until 2026-09-10 15:16:41.999101578 +0000 UTC m=+64.414897393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl4c4" (UID: "b12f58a1-a40c-4024-8411-2334e8d0244f") : configmap "kube-rbac-proxy" not found
Sep 10 15:16:39.999406 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:39.999123    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:16:39.999406 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:39.999164    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls podName:b12f58a1-a40c-4024-8411-2334e8d0244f nodeName:}" failed. No retries permitted until 2026-09-10 15:16:41.99915253 +0000 UTC m=+64.414948328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl4c4" (UID: "b12f58a1-a40c-4024-8411-2334e8d0244f") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:16:40.246127 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:40.246097    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kqvj8_04e0b284-688f-4bd7-a282-f9052bb11bf4/dns-node-resolver/0.log"
Sep 10 15:16:41.447432 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:41.447400    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xsmjp_b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d/node-ca/0.log"
Sep 10 15:16:42.014757 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:42.014720    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:42.014953 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:42.014772    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:42.014953 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:42.014865    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:16:42.014953 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:42.014886    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:16:42.014953 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:42.014930    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm podName:b12f58a1-a40c-4024-8411-2334e8d0244f nodeName:}" failed. No retries permitted until 2026-09-10 15:16:46.014913357 +0000 UTC m=+68.430709177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl4c4" (UID: "b12f58a1-a40c-4024-8411-2334e8d0244f") : configmap "kube-rbac-proxy" not found
Sep 10 15:16:42.014953 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:42.014944    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls podName:b12f58a1-a40c-4024-8411-2334e8d0244f nodeName:}" failed. No retries permitted until 2026-09-10 15:16:46.014937833 +0000 UTC m=+68.430733630 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl4c4" (UID: "b12f58a1-a40c-4024-8411-2334e8d0244f") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:16:42.248768 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:42.248734    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z8tnd_07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a/serve-healthcheck-canary/0.log"
Sep 10 15:16:43.929456 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:43.929415    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:16:43.929864 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:43.929482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:16:43.932287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:43.932259    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 15:16:43.932430 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:43.932333    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 15:16:43.940051 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:43.940035    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 15:16:43.940130 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:43.940100    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs podName:190adfd7-a800-4b59-b03c-fd0508ada6de nodeName:}" failed. No retries permitted until 2026-09-10 15:17:47.940084242 +0000 UTC m=+130.355880039 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs") pod "network-metrics-daemon-5kk8k" (UID: "190adfd7-a800-4b59-b03c-fd0508ada6de") : secret "metrics-daemon-secret" not found
Sep 10 15:16:43.942354 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:43.942338    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f858dfb-38ba-44b7-8b7d-b38cfb655428-original-pull-secret\") pod \"global-pull-secret-syncer-wkpgh\" (UID: \"3f858dfb-38ba-44b7-8b7d-b38cfb655428\") " pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:16:44.030082 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:44.030033    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759rr\" (UniqueName: \"kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr\") pod \"network-check-target-dqbgv\" (UID: \"42483446-4d7f-4a0d-96c7-7771f0b5a14c\") " pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:44.032753 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:44.032732    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 15:16:44.043101 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:44.043072    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 15:16:44.053572 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:44.053550    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-759rr\" (UniqueName: \"kubernetes.io/projected/42483446-4d7f-4a0d-96c7-7771f0b5a14c-kube-api-access-759rr\") pod \"network-check-target-dqbgv\" (UID: \"42483446-4d7f-4a0d-96c7-7771f0b5a14c\") " pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:44.137810 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:44.137769    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkpgh"
Sep 10 15:16:44.144290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:44.144272    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-96bxt\""
Sep 10 15:16:44.152134 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:44.152111    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:44.411875 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:44.411848    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wkpgh"]
Sep 10 15:16:44.414741 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:16:44.414713    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f858dfb_38ba_44b7_8b7d_b38cfb655428.slice/crio-b52fd0c4b5605b1c68800d396e793b7d651f8021dbae8fe7ad402c63cb1ae4f4 WatchSource:0}: Error finding container b52fd0c4b5605b1c68800d396e793b7d651f8021dbae8fe7ad402c63cb1ae4f4: Status 404 returned error can't find the container with id b52fd0c4b5605b1c68800d396e793b7d651f8021dbae8fe7ad402c63cb1ae4f4
Sep 10 15:16:44.439351 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:44.439295    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dqbgv"]
Sep 10 15:16:44.442338 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:16:44.442316    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42483446_4d7f_4a0d_96c7_7771f0b5a14c.slice/crio-45d1cfacaf2da8e777cebfc90c42f1c7881643b4275e05de4cbb202eeaaef488 WatchSource:0}: Error finding container 45d1cfacaf2da8e777cebfc90c42f1c7881643b4275e05de4cbb202eeaaef488: Status 404 returned error can't find the container with id 45d1cfacaf2da8e777cebfc90c42f1c7881643b4275e05de4cbb202eeaaef488
Sep 10 15:16:44.837201 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:44.837163    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:16:44.837350 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:44.837223    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:16:44.837350 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:44.837340    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:16:44.837434 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:44.837353    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:16:44.837434 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:44.837357    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579df544f9-wlzfv: secret "image-registry-tls" not found
Sep 10 15:16:44.837434 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:44.837417    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls podName:d6a33951-8222-4e14-9630-2b4b11140aaf nodeName:}" failed. No retries permitted until 2026-09-10 15:17:16.837398614 +0000 UTC m=+99.253194425 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls") pod "dns-default-wmlgv" (UID: "d6a33951-8222-4e14-9630-2b4b11140aaf") : secret "dns-default-metrics-tls" not found
Sep 10 15:16:44.837528 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:44.837437    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls podName:bcd493f9-5da2-403c-b48f-c32fb81e4571 nodeName:}" failed. No retries permitted until 2026-09-10 15:17:16.837426776 +0000 UTC m=+99.253222578 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls") pod "image-registry-579df544f9-wlzfv" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571") : secret "image-registry-tls" not found
Sep 10 15:16:45.418467 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:45.418424    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dqbgv" event={"ID":"42483446-4d7f-4a0d-96c7-7771f0b5a14c","Type":"ContainerStarted","Data":"45d1cfacaf2da8e777cebfc90c42f1c7881643b4275e05de4cbb202eeaaef488"}
Sep 10 15:16:45.420176 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:45.420147    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wkpgh" event={"ID":"3f858dfb-38ba-44b7-8b7d-b38cfb655428","Type":"ContainerStarted","Data":"b52fd0c4b5605b1c68800d396e793b7d651f8021dbae8fe7ad402c63cb1ae4f4"}
Sep 10 15:16:46.048754 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:46.048706    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:46.048981 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:46.048819    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:46.048981 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:46.048906    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:16:46.048981 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:46.048960    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:16:46.049156 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:46.048993    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls podName:b12f58a1-a40c-4024-8411-2334e8d0244f nodeName:}" failed. No retries permitted until 2026-09-10 15:16:54.048969924 +0000 UTC m=+76.464765721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl4c4" (UID: "b12f58a1-a40c-4024-8411-2334e8d0244f") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:16:46.049156 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:46.049020    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm podName:b12f58a1-a40c-4024-8411-2334e8d0244f nodeName:}" failed. No retries permitted until 2026-09-10 15:16:54.049004072 +0000 UTC m=+76.464799872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl4c4" (UID: "b12f58a1-a40c-4024-8411-2334e8d0244f") : configmap "kube-rbac-proxy" not found
Sep 10 15:16:48.433303 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:48.432863    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dqbgv" event={"ID":"42483446-4d7f-4a0d-96c7-7771f0b5a14c","Type":"ContainerStarted","Data":"806513cd6835447c1ecb5bfeec4d2fbc8fbaea826c0e3d7d477b0ce6f7d4ff75"}
Sep 10 15:16:48.433303 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:48.433029    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:16:48.450820 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:48.450756    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dqbgv" podStartSLOduration=66.894895046 podStartE2EDuration="1m10.450740396s" totalImagesPullingTime="3.55584535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:16:48.449771044 +0000 UTC m=+70.865566863" watchObservedRunningTime="2026-09-10 15:16:48.450740396 +0000 UTC m=+70.866536206"
Sep 10 15:16:50.440623 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:50.440590    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wkpgh" event={"ID":"3f858dfb-38ba-44b7-8b7d-b38cfb655428","Type":"ContainerStarted","Data":"464d471d397b104db5a9acb67b9ec358193ea977c800f6e9965e4bc31f2348c3"}
Sep 10 15:16:50.459098 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:50.459044    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wkpgh" podStartSLOduration=66.981363699 podStartE2EDuration="1m12.459027008s" totalImagesPullingTime="5.477663309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:16:50.457534854 +0000 UTC m=+72.873330672" watchObservedRunningTime="2026-09-10 15:16:50.459027008 +0000 UTC m=+72.874822826"
Sep 10 15:16:54.112356 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:54.112312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:54.112840 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:54.112367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:16:54.112840 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:54.112463    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:16:54.112840 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:16:54.112548    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls podName:b12f58a1-a40c-4024-8411-2334e8d0244f nodeName:}" failed. No retries permitted until 2026-09-10 15:17:10.112531413 +0000 UTC m=+92.528327214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl4c4" (UID: "b12f58a1-a40c-4024-8411-2334e8d0244f") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:16:54.113077 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:16:54.112856    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b12f58a1-a40c-4024-8411-2334e8d0244f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:17:10.136774 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:10.136672    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:17:10.139085 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:10.139064    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b12f58a1-a40c-4024-8411-2334e8d0244f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl4c4\" (UID: \"b12f58a1-a40c-4024-8411-2334e8d0244f\") " pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:17:10.404881 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:10.404776    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dl4c4"
Sep 10 15:17:10.675066 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:10.674989    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dl4c4"]
Sep 10 15:17:10.678223 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:17:10.678195    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12f58a1_a40c_4024_8411_2334e8d0244f.slice/crio-455c81a34a78679604272dbac36ea63cc31fdbdbcd253ff94322cedebf3381d9 WatchSource:0}: Error finding container 455c81a34a78679604272dbac36ea63cc31fdbdbcd253ff94322cedebf3381d9: Status 404 returned error can't find the container with id 455c81a34a78679604272dbac36ea63cc31fdbdbcd253ff94322cedebf3381d9
Sep 10 15:17:11.505203 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:11.505163    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dl4c4" event={"ID":"b12f58a1-a40c-4024-8411-2334e8d0244f","Type":"ContainerStarted","Data":"6ba164d0e1b0b234456efbe104a4c6a39151ac6777edfd61a1ee3e1b6dfaccce"}
Sep 10 15:17:11.505553 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:11.505243    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dl4c4" event={"ID":"b12f58a1-a40c-4024-8411-2334e8d0244f","Type":"ContainerStarted","Data":"455c81a34a78679604272dbac36ea63cc31fdbdbcd253ff94322cedebf3381d9"}
Sep 10 15:17:12.510239 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:12.510199    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dl4c4" event={"ID":"b12f58a1-a40c-4024-8411-2334e8d0244f","Type":"ContainerStarted","Data":"9ed5c2dd515850c51a9b2272d56f1a9096e0958d1ffaaf8ca91f9db929d80d87"}
Sep 10 15:17:13.515374 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:13.515336    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dl4c4" event={"ID":"b12f58a1-a40c-4024-8411-2334e8d0244f","Type":"ContainerStarted","Data":"4e5a5ef4c6b3f988b9e4709c0c509ddf6dd9fe837ad06b98d3643f706f916c52"}
Sep 10 15:17:13.534655 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:13.534603    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dl4c4" podStartSLOduration=32.965228685 podStartE2EDuration="35.534588397s" totalImagesPullingTime="2.569359712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:16:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:17:13.533566186 +0000 UTC m=+95.949362006" watchObservedRunningTime="2026-09-10 15:17:13.534588397 +0000 UTC m=+95.950384222"
Sep 10 15:17:15.192076 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.192038    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kwgtl"]
Sep 10 15:17:15.194103 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.194080    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.196261 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.196236    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 15:17:15.197857 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.197819    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 15:17:15.197857 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.197836    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 15:17:15.198042 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.197892    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 15:17:15.198042 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.197827    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 15:17:15.198168 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.198127    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m2csv\""
Sep 10 15:17:15.198479 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.198465    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 15:17:15.282448 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.282413    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-wtmp\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.282448 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.282448    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91a736cd-69d5-4a64-9eda-566b8cc61889-metrics-client-ca\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.282646 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.282472    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-accelerators-collector-config\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.282646 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.282577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91a736cd-69d5-4a64-9eda-566b8cc61889-sys\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.282749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.282645    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-textfile\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.282749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.282710    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-tls\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.282879 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.282760    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91a736cd-69d5-4a64-9eda-566b8cc61889-root\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.282879 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.282832    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.282971 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.282883    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbsdl\" (UniqueName: \"kubernetes.io/projected/91a736cd-69d5-4a64-9eda-566b8cc61889-kube-api-access-qbsdl\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.383398 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.383360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91a736cd-69d5-4a64-9eda-566b8cc61889-sys\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.383398 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.383403    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-textfile\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.383641 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.383421    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-tls\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.383641 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.383438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91a736cd-69d5-4a64-9eda-566b8cc61889-root\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.383641 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.383461    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.383641 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.383475    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91a736cd-69d5-4a64-9eda-566b8cc61889-sys\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.383641 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.383499    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbsdl\" (UniqueName: \"kubernetes.io/projected/91a736cd-69d5-4a64-9eda-566b8cc61889-kube-api-access-qbsdl\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.383641 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.383509    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91a736cd-69d5-4a64-9eda-566b8cc61889-root\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.383641 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.383587    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-wtmp\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.383641 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.383611    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91a736cd-69d5-4a64-9eda-566b8cc61889-metrics-client-ca\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.383641 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.383635    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-accelerators-collector-config\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.384075 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.383761    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-wtmp\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.384075 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.383786    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-textfile\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.384179 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.384147    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-accelerators-collector-config\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.384369 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.384200    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91a736cd-69d5-4a64-9eda-566b8cc61889-metrics-client-ca\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.386096 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.386073    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-tls\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.386311 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.386290    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91a736cd-69d5-4a64-9eda-566b8cc61889-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.396395 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.396368    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbsdl\" (UniqueName: \"kubernetes.io/projected/91a736cd-69d5-4a64-9eda-566b8cc61889-kube-api-access-qbsdl\") pod \"node-exporter-kwgtl\" (UID: \"91a736cd-69d5-4a64-9eda-566b8cc61889\") " pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.502962 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.502881    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kwgtl"
Sep 10 15:17:15.512557 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:17:15.512525    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a736cd_69d5_4a64_9eda_566b8cc61889.slice/crio-37c21a3d92214a60b01f95de3612e17e1a4bf1a036b625ae88785bb0d65cbdb3 WatchSource:0}: Error finding container 37c21a3d92214a60b01f95de3612e17e1a4bf1a036b625ae88785bb0d65cbdb3: Status 404 returned error can't find the container with id 37c21a3d92214a60b01f95de3612e17e1a4bf1a036b625ae88785bb0d65cbdb3
Sep 10 15:17:15.522722 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:15.522697    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwgtl" event={"ID":"91a736cd-69d5-4a64-9eda-566b8cc61889","Type":"ContainerStarted","Data":"37c21a3d92214a60b01f95de3612e17e1a4bf1a036b625ae88785bb0d65cbdb3"}
Sep 10 15:17:16.896076 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:16.896042    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:17:16.896438 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:16.896090    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:17:16.898467 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:16.898433    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6a33951-8222-4e14-9630-2b4b11140aaf-metrics-tls\") pod \"dns-default-wmlgv\" (UID: \"d6a33951-8222-4e14-9630-2b4b11140aaf\") " pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:17:16.898727 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:16.898705    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls\") pod \"image-registry-579df544f9-wlzfv\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") " pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:17:16.921514 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:16.921485    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4q2px\""
Sep 10 15:17:16.930285 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:16.930265    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:17:17.162877 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:17.162782    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sxzrd\""
Sep 10 15:17:17.171033 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:17.171014    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:17:17.233294 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:17.233267    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wmlgv"]
Sep 10 15:17:17.234101 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:17:17.234076    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a33951_8222_4e14_9630_2b4b11140aaf.slice/crio-3f2954267012ac65d1a73c12e3a7a4baa56b45fd936fbd1b6f783fb7a189a9bc WatchSource:0}: Error finding container 3f2954267012ac65d1a73c12e3a7a4baa56b45fd936fbd1b6f783fb7a189a9bc: Status 404 returned error can't find the container with id 3f2954267012ac65d1a73c12e3a7a4baa56b45fd936fbd1b6f783fb7a189a9bc
Sep 10 15:17:17.450157 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:17.450051    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-579df544f9-wlzfv"]
Sep 10 15:17:17.453782 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:17:17.453754    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd493f9_5da2_403c_b48f_c32fb81e4571.slice/crio-1e99274426225659383a78f86f1d5ab47879a71fada114dc3cdaeab1a824da17 WatchSource:0}: Error finding container 1e99274426225659383a78f86f1d5ab47879a71fada114dc3cdaeab1a824da17: Status 404 returned error can't find the container with id 1e99274426225659383a78f86f1d5ab47879a71fada114dc3cdaeab1a824da17
Sep 10 15:17:17.529549 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:17.529515    2573 generic.go:356] "Generic (PLEG): container finished" podID="91a736cd-69d5-4a64-9eda-566b8cc61889" containerID="9bfa30bc41a646a5c0696a7f174f4db337e302310699cff161fe1376b989ab32" exitCode=0
Sep 10 15:17:17.529728 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:17.529582    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwgtl" event={"ID":"91a736cd-69d5-4a64-9eda-566b8cc61889","Type":"ContainerDied","Data":"9bfa30bc41a646a5c0696a7f174f4db337e302310699cff161fe1376b989ab32"}
Sep 10 15:17:17.530769 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:17.530747    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wmlgv" event={"ID":"d6a33951-8222-4e14-9630-2b4b11140aaf","Type":"ContainerStarted","Data":"3f2954267012ac65d1a73c12e3a7a4baa56b45fd936fbd1b6f783fb7a189a9bc"}
Sep 10 15:17:17.532105 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:17.532049    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579df544f9-wlzfv" event={"ID":"bcd493f9-5da2-403c-b48f-c32fb81e4571","Type":"ContainerStarted","Data":"1e99274426225659383a78f86f1d5ab47879a71fada114dc3cdaeab1a824da17"}
Sep 10 15:17:18.538331 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:18.538288    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwgtl" event={"ID":"91a736cd-69d5-4a64-9eda-566b8cc61889","Type":"ContainerStarted","Data":"d67491b018a522c63f69ecf683613ac99573612bb6c85ffb5ad622113858c812"}
Sep 10 15:17:18.538837 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:18.538353    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwgtl" event={"ID":"91a736cd-69d5-4a64-9eda-566b8cc61889","Type":"ContainerStarted","Data":"c8c713bf1b99f2138af841d7339c1109d157ff0bb58c23a38328d3258cecd550"}
Sep 10 15:17:18.540144 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:18.540112    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579df544f9-wlzfv" event={"ID":"bcd493f9-5da2-403c-b48f-c32fb81e4571","Type":"ContainerStarted","Data":"f1f1587ef14c37bce0e4e2dc53122785365078dbb6b182a11b8df6ad8bd55364"}
Sep 10 15:17:18.540324 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:18.540309    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:17:18.559645 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:18.559596    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kwgtl" podStartSLOduration=2.446160718 podStartE2EDuration="3.559582855s" totalImagesPullingTime="1.044871494s" totalInitContainerRuntime="68.550643ms" isStatefulPod=true podCreationTimestamp="2026-09-10 15:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:17:18.559188065 +0000 UTC m=+100.974983888" watchObservedRunningTime="2026-09-10 15:17:18.559582855 +0000 UTC m=+100.975378673"
Sep 10 15:17:18.579072 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:18.579013    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-579df544f9-wlzfv" podStartSLOduration=100.578995826 podStartE2EDuration="1m40.578995826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:17:18.578263983 +0000 UTC m=+100.994059801" watchObservedRunningTime="2026-09-10 15:17:18.578995826 +0000 UTC m=+100.994791659"
Sep 10 15:17:19.438866 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:19.438838    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dqbgv"
Sep 10 15:17:19.544854 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:19.544816    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wmlgv" event={"ID":"d6a33951-8222-4e14-9630-2b4b11140aaf","Type":"ContainerStarted","Data":"cedd903ae1b39bcbc6c17810e3c5690034cd8b4c2c8cb0403f276b495d701399"}
Sep 10 15:17:19.545330 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:19.544911    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wmlgv" event={"ID":"d6a33951-8222-4e14-9630-2b4b11140aaf","Type":"ContainerStarted","Data":"866cf7fefec3e10b6e8837dbe415ddd7e9b468940b8bcc0090049e901485b459"}
Sep 10 15:17:20.548474 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:20.548445    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:17:26.294928 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:26.294875    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wmlgv" podStartSLOduration=72.507026053 podStartE2EDuration="1m14.294857898s" totalImagesPullingTime="1.787831845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:17:19.562081842 +0000 UTC m=+101.977877661" watchObservedRunningTime="2026-09-10 15:17:26.294857898 +0000 UTC m=+108.710653716"
Sep 10 15:17:26.295380 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:26.295205    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-579df544f9-wlzfv"]
Sep 10 15:17:30.554397 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:30.554367    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wmlgv"
Sep 10 15:17:36.301497 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:36.301471    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:17:48.030476 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:48.030436    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:17:48.032919 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:48.032891    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/190adfd7-a800-4b59-b03c-fd0508ada6de-metrics-certs\") pod \"network-metrics-daemon-5kk8k\" (UID: \"190adfd7-a800-4b59-b03c-fd0508ada6de\") " pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:17:48.332456 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:48.332426    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d5wmh\""
Sep 10 15:17:48.340220 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:48.340197    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kk8k"
Sep 10 15:17:48.610176 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:48.610102    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5kk8k"]
Sep 10 15:17:48.613470 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:17:48.613442    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190adfd7_a800_4b59_b03c_fd0508ada6de.slice/crio-64074c285a9d679eef64f0aac31cb6dc9d805e11d8c1790db3632be18e87a547 WatchSource:0}: Error finding container 64074c285a9d679eef64f0aac31cb6dc9d805e11d8c1790db3632be18e87a547: Status 404 returned error can't find the container with id 64074c285a9d679eef64f0aac31cb6dc9d805e11d8c1790db3632be18e87a547
Sep 10 15:17:48.637647 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:48.637620    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kk8k" event={"ID":"190adfd7-a800-4b59-b03c-fd0508ada6de","Type":"ContainerStarted","Data":"64074c285a9d679eef64f0aac31cb6dc9d805e11d8c1790db3632be18e87a547"}
Sep 10 15:17:50.646880 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:50.646845    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kk8k" event={"ID":"190adfd7-a800-4b59-b03c-fd0508ada6de","Type":"ContainerStarted","Data":"ce615284c904bb66d03452523e60ed4dad2459e4569f1f0f8e6f9c8dca0dbf37"}
Sep 10 15:17:50.647246 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:50.646899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kk8k" event={"ID":"190adfd7-a800-4b59-b03c-fd0508ada6de","Type":"ContainerStarted","Data":"8b9d501e3cceaec65f1d623fe52a60692904d6dd0531268ae8b9c3d734a3c0ce"}
Sep 10 15:17:50.665136 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:50.665087    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5kk8k" podStartSLOduration=131.541491214 podStartE2EDuration="2m12.665071446s" totalImagesPullingTime="1.123580232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:17:50.663983011 +0000 UTC m=+133.079778829" watchObservedRunningTime="2026-09-10 15:17:50.665071446 +0000 UTC m=+133.080867294"
Sep 10 15:17:51.314719 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.314651    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-579df544f9-wlzfv" podUID="bcd493f9-5da2-403c-b48f-c32fb81e4571" containerName="registry" containerID="cri-o://f1f1587ef14c37bce0e4e2dc53122785365078dbb6b182a11b8df6ad8bd55364" gracePeriod=30
Sep 10 15:17:51.557478 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.557452    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:17:51.651544 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.651504    2573 generic.go:356] "Generic (PLEG): container finished" podID="bcd493f9-5da2-403c-b48f-c32fb81e4571" containerID="f1f1587ef14c37bce0e4e2dc53122785365078dbb6b182a11b8df6ad8bd55364" exitCode=0
Sep 10 15:17:51.651967 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.651554    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579df544f9-wlzfv"
Sep 10 15:17:51.651967 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.651591    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579df544f9-wlzfv" event={"ID":"bcd493f9-5da2-403c-b48f-c32fb81e4571","Type":"ContainerDied","Data":"f1f1587ef14c37bce0e4e2dc53122785365078dbb6b182a11b8df6ad8bd55364"}
Sep 10 15:17:51.651967 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.651646    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579df544f9-wlzfv" event={"ID":"bcd493f9-5da2-403c-b48f-c32fb81e4571","Type":"ContainerDied","Data":"1e99274426225659383a78f86f1d5ab47879a71fada114dc3cdaeab1a824da17"}
Sep 10 15:17:51.652488 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.652475    2573 scope.go:117] "RemoveContainer" containerID="f1f1587ef14c37bce0e4e2dc53122785365078dbb6b182a11b8df6ad8bd55364"
Sep 10 15:17:51.655948 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.655932    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-certificates\") pod \"bcd493f9-5da2-403c-b48f-c32fb81e4571\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") "
Sep 10 15:17:51.656062 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.655960    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-bound-sa-token\") pod \"bcd493f9-5da2-403c-b48f-c32fb81e4571\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") "
Sep 10 15:17:51.656062 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.655995    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcd493f9-5da2-403c-b48f-c32fb81e4571-trusted-ca\") pod \"bcd493f9-5da2-403c-b48f-c32fb81e4571\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") "
Sep 10 15:17:51.656062 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.656018    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg22n\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-kube-api-access-qg22n\") pod \"bcd493f9-5da2-403c-b48f-c32fb81e4571\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") "
Sep 10 15:17:51.656062 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.656057    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcd493f9-5da2-403c-b48f-c32fb81e4571-installation-pull-secrets\") pod \"bcd493f9-5da2-403c-b48f-c32fb81e4571\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") "
Sep 10 15:17:51.656270 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.656080    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls\") pod \"bcd493f9-5da2-403c-b48f-c32fb81e4571\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") "
Sep 10 15:17:51.656270 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.656122    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcd493f9-5da2-403c-b48f-c32fb81e4571-image-registry-private-configuration\") pod \"bcd493f9-5da2-403c-b48f-c32fb81e4571\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") "
Sep 10 15:17:51.656270 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.656165    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcd493f9-5da2-403c-b48f-c32fb81e4571-ca-trust-extracted\") pod \"bcd493f9-5da2-403c-b48f-c32fb81e4571\" (UID: \"bcd493f9-5da2-403c-b48f-c32fb81e4571\") "
Sep 10 15:17:51.656426 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.656404    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bcd493f9-5da2-403c-b48f-c32fb81e4571" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:17:51.656589 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.656560    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd493f9-5da2-403c-b48f-c32fb81e4571-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bcd493f9-5da2-403c-b48f-c32fb81e4571" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:17:51.659022 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.658984    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bcd493f9-5da2-403c-b48f-c32fb81e4571" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:17:51.659132 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.659050    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bcd493f9-5da2-403c-b48f-c32fb81e4571" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:17:51.659132 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.659062    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-kube-api-access-qg22n" (OuterVolumeSpecName: "kube-api-access-qg22n") pod "bcd493f9-5da2-403c-b48f-c32fb81e4571" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571"). InnerVolumeSpecName "kube-api-access-qg22n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:17:51.659290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.659266    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd493f9-5da2-403c-b48f-c32fb81e4571-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bcd493f9-5da2-403c-b48f-c32fb81e4571" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:17:51.659361 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.659317    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd493f9-5da2-403c-b48f-c32fb81e4571-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bcd493f9-5da2-403c-b48f-c32fb81e4571" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:17:51.662948 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.662926    2573 scope.go:117] "RemoveContainer" containerID="f1f1587ef14c37bce0e4e2dc53122785365078dbb6b182a11b8df6ad8bd55364"
Sep 10 15:17:51.663645 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:17:51.663622    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f1587ef14c37bce0e4e2dc53122785365078dbb6b182a11b8df6ad8bd55364\": container with ID starting with f1f1587ef14c37bce0e4e2dc53122785365078dbb6b182a11b8df6ad8bd55364 not found: ID does not exist" containerID="f1f1587ef14c37bce0e4e2dc53122785365078dbb6b182a11b8df6ad8bd55364"
Sep 10 15:17:51.663719 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.663652    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f1587ef14c37bce0e4e2dc53122785365078dbb6b182a11b8df6ad8bd55364"} err="failed to get container status \"f1f1587ef14c37bce0e4e2dc53122785365078dbb6b182a11b8df6ad8bd55364\": rpc error: code = NotFound desc = could not find container \"f1f1587ef14c37bce0e4e2dc53122785365078dbb6b182a11b8df6ad8bd55364\": container with ID starting with f1f1587ef14c37bce0e4e2dc53122785365078dbb6b182a11b8df6ad8bd55364 not found: ID does not exist"
Sep 10 15:17:51.666407 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.666385    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd493f9-5da2-403c-b48f-c32fb81e4571-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bcd493f9-5da2-403c-b48f-c32fb81e4571" (UID: "bcd493f9-5da2-403c-b48f-c32fb81e4571"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:17:51.757142 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.757100    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg22n\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-kube-api-access-qg22n\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:17:51.757142 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.757135    2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcd493f9-5da2-403c-b48f-c32fb81e4571-installation-pull-secrets\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:17:51.757142 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.757147    2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:17:51.757383 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.757157    2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcd493f9-5da2-403c-b48f-c32fb81e4571-image-registry-private-configuration\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:17:51.757383 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.757165    2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcd493f9-5da2-403c-b48f-c32fb81e4571-ca-trust-extracted\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:17:51.757383 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.757182    2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcd493f9-5da2-403c-b48f-c32fb81e4571-registry-certificates\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:17:51.757383 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.757196    2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcd493f9-5da2-403c-b48f-c32fb81e4571-bound-sa-token\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:17:51.757383 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.757210    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcd493f9-5da2-403c-b48f-c32fb81e4571-trusted-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:17:51.972184 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.972151    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-579df544f9-wlzfv"]
Sep 10 15:17:51.979732 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:51.979708    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-579df544f9-wlzfv"]
Sep 10 15:17:52.124557 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:52.124528    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcd493f9-5da2-403c-b48f-c32fb81e4571" path="/var/lib/kubelet/pods/bcd493f9-5da2-403c-b48f-c32fb81e4571/volumes"
Sep 10 15:17:53.281498 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:17:53.281450    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k" podUID="d540c9c4-a41f-45c2-bbbd-f52c5bee38a1" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 10 15:18:03.281285 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:18:03.281249    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k" podUID="d540c9c4-a41f-45c2-bbbd-f52c5bee38a1" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 10 15:18:13.281596 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:18:13.281558    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k" podUID="d540c9c4-a41f-45c2-bbbd-f52c5bee38a1" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 10 15:18:13.282093 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:18:13.281628    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k"
Sep 10 15:18:13.282147 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:18:13.282111    2573 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"53b1763e1f5e7003ab23b33aa386886bd0270e02daa5acc7ad369446ad1606e6"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 10 15:18:13.282184 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:18:13.282149    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k" podUID="d540c9c4-a41f-45c2-bbbd-f52c5bee38a1" containerName="service-proxy" containerID="cri-o://53b1763e1f5e7003ab23b33aa386886bd0270e02daa5acc7ad369446ad1606e6" gracePeriod=30
Sep 10 15:18:13.725221 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:18:13.725181    2573 generic.go:356] "Generic (PLEG): container finished" podID="d540c9c4-a41f-45c2-bbbd-f52c5bee38a1" containerID="53b1763e1f5e7003ab23b33aa386886bd0270e02daa5acc7ad369446ad1606e6" exitCode=2
Sep 10 15:18:13.725404 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:18:13.725255    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k" event={"ID":"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1","Type":"ContainerDied","Data":"53b1763e1f5e7003ab23b33aa386886bd0270e02daa5acc7ad369446ad1606e6"}
Sep 10 15:18:13.725404 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:18:13.725301    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54454bfbc-p548k" event={"ID":"d540c9c4-a41f-45c2-bbbd-f52c5bee38a1","Type":"ContainerStarted","Data":"e712b63cd5bd89ca17370fa14c779f6ad5471ddd7644ac30cb4e48038b04b89f"}
Sep 10 15:20:01.496714 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.496635    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vj767"]
Sep 10 15:20:01.497196 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.496933    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcd493f9-5da2-403c-b48f-c32fb81e4571" containerName="registry"
Sep 10 15:20:01.497196 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.496949    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcd493f9-5da2-403c-b48f-c32fb81e4571" containerName="registry"
Sep 10 15:20:01.497196 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.496988    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcd493f9-5da2-403c-b48f-c32fb81e4571" containerName="registry"
Sep 10 15:20:01.499507 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.499490    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vj767"
Sep 10 15:20:01.501690 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.501669    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 15:20:01.502459 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.502440    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 15:20:01.502525 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.502471    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-cvql2\""
Sep 10 15:20:01.510990 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.510968    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vj767"]
Sep 10 15:20:01.543187 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.543157    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnvpz\" (UniqueName: \"kubernetes.io/projected/d908dcc5-8150-49e8-9eb9-cd29274a2edf-kube-api-access-gnvpz\") pod \"cert-manager-cainjector-559cdd8c87-vj767\" (UID: \"d908dcc5-8150-49e8-9eb9-cd29274a2edf\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vj767"
Sep 10 15:20:01.543293 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.543201    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d908dcc5-8150-49e8-9eb9-cd29274a2edf-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vj767\" (UID: \"d908dcc5-8150-49e8-9eb9-cd29274a2edf\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vj767"
Sep 10 15:20:01.644042 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.644005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnvpz\" (UniqueName: \"kubernetes.io/projected/d908dcc5-8150-49e8-9eb9-cd29274a2edf-kube-api-access-gnvpz\") pod \"cert-manager-cainjector-559cdd8c87-vj767\" (UID: \"d908dcc5-8150-49e8-9eb9-cd29274a2edf\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vj767"
Sep 10 15:20:01.644206 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.644048    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d908dcc5-8150-49e8-9eb9-cd29274a2edf-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vj767\" (UID: \"d908dcc5-8150-49e8-9eb9-cd29274a2edf\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vj767"
Sep 10 15:20:01.652907 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.652870    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d908dcc5-8150-49e8-9eb9-cd29274a2edf-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vj767\" (UID: \"d908dcc5-8150-49e8-9eb9-cd29274a2edf\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vj767"
Sep 10 15:20:01.653014 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.652920    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnvpz\" (UniqueName: \"kubernetes.io/projected/d908dcc5-8150-49e8-9eb9-cd29274a2edf-kube-api-access-gnvpz\") pod \"cert-manager-cainjector-559cdd8c87-vj767\" (UID: \"d908dcc5-8150-49e8-9eb9-cd29274a2edf\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vj767"
Sep 10 15:20:01.808118 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:01.808034    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vj767"
Sep 10 15:20:02.075979 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:02.075950    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vj767"]
Sep 10 15:20:02.079276 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:20:02.079241    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd908dcc5_8150_49e8_9eb9_cd29274a2edf.slice/crio-4b064225f7637695c9c18700015219962769e4aa5a25dff21b17e6221eb31a07 WatchSource:0}: Error finding container 4b064225f7637695c9c18700015219962769e4aa5a25dff21b17e6221eb31a07: Status 404 returned error can't find the container with id 4b064225f7637695c9c18700015219962769e4aa5a25dff21b17e6221eb31a07
Sep 10 15:20:03.073732 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:03.073697    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vj767" event={"ID":"d908dcc5-8150-49e8-9eb9-cd29274a2edf","Type":"ContainerStarted","Data":"4b064225f7637695c9c18700015219962769e4aa5a25dff21b17e6221eb31a07"}
Sep 10 15:20:06.087317 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:06.087276    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vj767" event={"ID":"d908dcc5-8150-49e8-9eb9-cd29274a2edf","Type":"ContainerStarted","Data":"1571b0847dc7ada1cb380a872e6c86cb92e985810f1aae7d021199548ce309ea"}
Sep 10 15:20:10.579595 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.579533    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vj767" podStartSLOduration=6.495971586 podStartE2EDuration="9.579513127s" totalImagesPullingTime="3.083541541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:20:06.108460347 +0000 UTC m=+268.524256167" watchObservedRunningTime="2026-09-10 15:20:10.579513127 +0000 UTC m=+272.995308970"
Sep 10 15:20:10.580448 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.580428    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w"]
Sep 10 15:20:10.583299 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.583282    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w"
Sep 10 15:20:10.585500 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.585476    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:20:10.585632 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.585519    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 15:20:10.585632 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.585539    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-nl7vk\""
Sep 10 15:20:10.593774 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.593752    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w"]
Sep 10 15:20:10.703550 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.703514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc41b164-0531-49a0-8f6a-eb7da8e60c54-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6x82w\" (UID: \"dc41b164-0531-49a0-8f6a-eb7da8e60c54\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w"
Sep 10 15:20:10.703718 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.703563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bztcw\" (UniqueName: \"kubernetes.io/projected/dc41b164-0531-49a0-8f6a-eb7da8e60c54-kube-api-access-bztcw\") pod \"openshift-lws-operator-fd8ccff4c-6x82w\" (UID: \"dc41b164-0531-49a0-8f6a-eb7da8e60c54\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w"
Sep 10 15:20:10.803875 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.803839    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc41b164-0531-49a0-8f6a-eb7da8e60c54-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6x82w\" (UID: \"dc41b164-0531-49a0-8f6a-eb7da8e60c54\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w"
Sep 10 15:20:10.804015 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.803888    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bztcw\" (UniqueName: \"kubernetes.io/projected/dc41b164-0531-49a0-8f6a-eb7da8e60c54-kube-api-access-bztcw\") pod \"openshift-lws-operator-fd8ccff4c-6x82w\" (UID: \"dc41b164-0531-49a0-8f6a-eb7da8e60c54\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w"
Sep 10 15:20:10.804263 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.804240    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc41b164-0531-49a0-8f6a-eb7da8e60c54-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6x82w\" (UID: \"dc41b164-0531-49a0-8f6a-eb7da8e60c54\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w"
Sep 10 15:20:10.813091 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.813066    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bztcw\" (UniqueName: \"kubernetes.io/projected/dc41b164-0531-49a0-8f6a-eb7da8e60c54-kube-api-access-bztcw\") pod \"openshift-lws-operator-fd8ccff4c-6x82w\" (UID: \"dc41b164-0531-49a0-8f6a-eb7da8e60c54\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w"
Sep 10 15:20:10.891676 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:10.891586    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w"
Sep 10 15:20:11.162198 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:11.162115    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w"]
Sep 10 15:20:11.165674 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:20:11.165642    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc41b164_0531_49a0_8f6a_eb7da8e60c54.slice/crio-ab0f99e91a364f830f72361dc89a4a34f762405326ff625ebfc945a656a45ee0 WatchSource:0}: Error finding container ab0f99e91a364f830f72361dc89a4a34f762405326ff625ebfc945a656a45ee0: Status 404 returned error can't find the container with id ab0f99e91a364f830f72361dc89a4a34f762405326ff625ebfc945a656a45ee0
Sep 10 15:20:12.107838 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:12.107784    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w" event={"ID":"dc41b164-0531-49a0-8f6a-eb7da8e60c54","Type":"ContainerStarted","Data":"ab0f99e91a364f830f72361dc89a4a34f762405326ff625ebfc945a656a45ee0"}
Sep 10 15:20:14.116139 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:14.116100    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w" event={"ID":"dc41b164-0531-49a0-8f6a-eb7da8e60c54","Type":"ContainerStarted","Data":"92b2f1572a7d2cfadbc8cbd6dea459be211865dc27c72ea41e1e02ea4ee5b971"}
Sep 10 15:20:14.134557 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:14.134504    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6x82w" podStartSLOduration=1.369806818 podStartE2EDuration="4.134486776s" totalImagesPullingTime="2.764679958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:20:14.133338062 +0000 UTC m=+276.549133881" watchObservedRunningTime="2026-09-10 15:20:14.134486776 +0000 UTC m=+276.550282599"
Sep 10 15:20:32.109808 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.109766    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"]
Sep 10 15:20:32.112117 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.112101    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.114538 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.114511    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 15:20:32.115767 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.115732    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 15:20:32.115941 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.115927    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wbskq\""
Sep 10 15:20:32.117844 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.117824    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 15:20:32.117949 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.117916    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 15:20:32.136141 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.136108    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"]
Sep 10 15:20:32.252591 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.252547    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvkr\" (UniqueName: \"kubernetes.io/projected/65312fa5-0192-4130-81c8-2e9b54329d00-kube-api-access-tdvkr\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.252591 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.252597    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65312fa5-0192-4130-81c8-2e9b54329d00-manifests\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.252862 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.252617    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65312fa5-0192-4130-81c8-2e9b54329d00-tmp\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.252862 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.252668    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65312fa5-0192-4130-81c8-2e9b54329d00-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.252862 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.252700    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65312fa5-0192-4130-81c8-2e9b54329d00-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.353409 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.353371    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65312fa5-0192-4130-81c8-2e9b54329d00-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.353409 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.353417    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvkr\" (UniqueName: \"kubernetes.io/projected/65312fa5-0192-4130-81c8-2e9b54329d00-kube-api-access-tdvkr\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.353660 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.353437    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65312fa5-0192-4130-81c8-2e9b54329d00-manifests\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.353660 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.353457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65312fa5-0192-4130-81c8-2e9b54329d00-tmp\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.353660 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.353597    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65312fa5-0192-4130-81c8-2e9b54329d00-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.353967 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.353943    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65312fa5-0192-4130-81c8-2e9b54329d00-manifests\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.355961 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.355935    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65312fa5-0192-4130-81c8-2e9b54329d00-tmp\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.356203 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.356184    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65312fa5-0192-4130-81c8-2e9b54329d00-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.356260 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.356197    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65312fa5-0192-4130-81c8-2e9b54329d00-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.362474 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.362417    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvkr\" (UniqueName: \"kubernetes.io/projected/65312fa5-0192-4130-81c8-2e9b54329d00-kube-api-access-tdvkr\") pod \"opendatahub-operator-controller-manager-76f847f9cd-mtl9p\" (UID: \"65312fa5-0192-4130-81c8-2e9b54329d00\") " pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.422648 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.422612    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:32.716826 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:32.716601    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"]
Sep 10 15:20:32.719425 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:20:32.719401    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65312fa5_0192_4130_81c8_2e9b54329d00.slice/crio-49d1260a22b610c869ccc33c274e2856f79363f7f98018faf14539a3725d044f WatchSource:0}: Error finding container 49d1260a22b610c869ccc33c274e2856f79363f7f98018faf14539a3725d044f: Status 404 returned error can't find the container with id 49d1260a22b610c869ccc33c274e2856f79363f7f98018faf14539a3725d044f
Sep 10 15:20:33.178606 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:33.178569    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p" event={"ID":"65312fa5-0192-4130-81c8-2e9b54329d00","Type":"ContainerStarted","Data":"49d1260a22b610c869ccc33c274e2856f79363f7f98018faf14539a3725d044f"}
Sep 10 15:20:36.192889 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:36.192846    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p" event={"ID":"65312fa5-0192-4130-81c8-2e9b54329d00","Type":"ContainerStarted","Data":"12ca5e72f0ac0d7844acdd404ab6fbeb69bed037235034ee9f08a0a266897e99"}
Sep 10 15:20:37.197723 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:37.197685    2573 generic.go:356] "Generic (PLEG): container finished" podID="65312fa5-0192-4130-81c8-2e9b54329d00" containerID="12ca5e72f0ac0d7844acdd404ab6fbeb69bed037235034ee9f08a0a266897e99" exitCode=0
Sep 10 15:20:37.198128 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:37.197743    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p" event={"ID":"65312fa5-0192-4130-81c8-2e9b54329d00","Type":"ContainerDied","Data":"12ca5e72f0ac0d7844acdd404ab6fbeb69bed037235034ee9f08a0a266897e99"}
Sep 10 15:20:38.033313 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:38.033245    2573 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 15:20:38.203355 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:38.203314    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p" event={"ID":"65312fa5-0192-4130-81c8-2e9b54329d00","Type":"ContainerStarted","Data":"cdde8a40dfb5d2ae0ae51a80bcdbb30a1af06d661ade9140cf43f31176ed9422"}
Sep 10 15:20:38.212305 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:38.203473    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:38.238423 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:38.238366    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p" podStartSLOduration=1.528078364 podStartE2EDuration="6.238344617s" totalImagesPullingTime="3.685815543s" totalInitContainerRuntime="1.02445071s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:20:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:20:38.237684031 +0000 UTC m=+300.653479861" watchObservedRunningTime="2026-09-10 15:20:38.238344617 +0000 UTC m=+300.654140436"
Sep 10 15:20:49.209708 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:49.209670    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76f847f9cd-mtl9p"
Sep 10 15:20:52.107770 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.107738    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"]
Sep 10 15:20:52.114583 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.114566    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"
Sep 10 15:20:52.128424 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.128403    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 15:20:52.129857 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.129840    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 15:20:52.140152 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.140127    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 15:20:52.141260 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.141242    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 15:20:52.151577 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.151554    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7fz72\""
Sep 10 15:20:52.152963 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.152938    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"]
Sep 10 15:20:52.220850 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.220814    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzptr\" (UniqueName: \"kubernetes.io/projected/181199ba-a7a1-4031-a5ac-8b8eb120b694-kube-api-access-fzptr\") pod \"kube-auth-proxy-84bd548dcf-gt6gm\" (UID: \"181199ba-a7a1-4031-a5ac-8b8eb120b694\") " pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"
Sep 10 15:20:52.221038 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.220898    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/181199ba-a7a1-4031-a5ac-8b8eb120b694-tls-certs\") pod \"kube-auth-proxy-84bd548dcf-gt6gm\" (UID: \"181199ba-a7a1-4031-a5ac-8b8eb120b694\") " pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"
Sep 10 15:20:52.221038 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.220955    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/181199ba-a7a1-4031-a5ac-8b8eb120b694-tmp\") pod \"kube-auth-proxy-84bd548dcf-gt6gm\" (UID: \"181199ba-a7a1-4031-a5ac-8b8eb120b694\") " pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"
Sep 10 15:20:52.321974 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.321934    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/181199ba-a7a1-4031-a5ac-8b8eb120b694-tls-certs\") pod \"kube-auth-proxy-84bd548dcf-gt6gm\" (UID: \"181199ba-a7a1-4031-a5ac-8b8eb120b694\") " pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"
Sep 10 15:20:52.322172 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.321987    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/181199ba-a7a1-4031-a5ac-8b8eb120b694-tmp\") pod \"kube-auth-proxy-84bd548dcf-gt6gm\" (UID: \"181199ba-a7a1-4031-a5ac-8b8eb120b694\") " pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"
Sep 10 15:20:52.322172 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.322015    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzptr\" (UniqueName: \"kubernetes.io/projected/181199ba-a7a1-4031-a5ac-8b8eb120b694-kube-api-access-fzptr\") pod \"kube-auth-proxy-84bd548dcf-gt6gm\" (UID: \"181199ba-a7a1-4031-a5ac-8b8eb120b694\") " pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"
Sep 10 15:20:52.324496 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.324466    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/181199ba-a7a1-4031-a5ac-8b8eb120b694-tmp\") pod \"kube-auth-proxy-84bd548dcf-gt6gm\" (UID: \"181199ba-a7a1-4031-a5ac-8b8eb120b694\") " pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"
Sep 10 15:20:52.324708 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.324689    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/181199ba-a7a1-4031-a5ac-8b8eb120b694-tls-certs\") pod \"kube-auth-proxy-84bd548dcf-gt6gm\" (UID: \"181199ba-a7a1-4031-a5ac-8b8eb120b694\") " pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"
Sep 10 15:20:52.381343 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.381267    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzptr\" (UniqueName: \"kubernetes.io/projected/181199ba-a7a1-4031-a5ac-8b8eb120b694-kube-api-access-fzptr\") pod \"kube-auth-proxy-84bd548dcf-gt6gm\" (UID: \"181199ba-a7a1-4031-a5ac-8b8eb120b694\") " pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"
Sep 10 15:20:52.423418 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.423382    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"
Sep 10 15:20:52.714107 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.714016    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm"]
Sep 10 15:20:52.717073 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:20:52.717041    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181199ba_a7a1_4031_a5ac_8b8eb120b694.slice/crio-b1e4f3a3e7d3482bf8c398b3701242e32d859d12b099440f87561067fdaa7084 WatchSource:0}: Error finding container b1e4f3a3e7d3482bf8c398b3701242e32d859d12b099440f87561067fdaa7084: Status 404 returned error can't find the container with id b1e4f3a3e7d3482bf8c398b3701242e32d859d12b099440f87561067fdaa7084
Sep 10 15:20:52.718931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:52.718911    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:20:53.257588 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:53.257550    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm" event={"ID":"181199ba-a7a1-4031-a5ac-8b8eb120b694","Type":"ContainerStarted","Data":"b1e4f3a3e7d3482bf8c398b3701242e32d859d12b099440f87561067fdaa7084"}
Sep 10 15:20:53.902588 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:53.902549    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"]
Sep 10 15:20:53.905115 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:53.905080    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:53.909497 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:53.908213    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 15:20:53.909497 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:53.908973    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 15:20:53.909497 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:53.909188    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-d5vrh\""
Sep 10 15:20:53.909497 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:53.909262    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 15:20:53.910044 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:53.910006    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 15:20:53.930912 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:53.930884    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"]
Sep 10 15:20:54.036368 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.036325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.036566 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.036373    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64170c5f-e00d-47da-96ce-120e191dbb1a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.036566 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.036415    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdsrn\" (UniqueName: \"kubernetes.io/projected/64170c5f-e00d-47da-96ce-120e191dbb1a-kube-api-access-mdsrn\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.036566 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.036521    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.036566 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.036554    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.036752 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.036579    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.036752 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.036642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.137603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.137542    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.137603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.137597    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.137866 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.137631    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.137866 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.137677    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.137866 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.137716    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.137866 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.137744    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64170c5f-e00d-47da-96ce-120e191dbb1a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.137866 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.137776    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdsrn\" (UniqueName: \"kubernetes.io/projected/64170c5f-e00d-47da-96ce-120e191dbb1a-kube-api-access-mdsrn\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.138225 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.138198    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.141438 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.141390    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.142200 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.141862    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.142200 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.142165    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.145278 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.145230    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64170c5f-e00d-47da-96ce-120e191dbb1a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.147932 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.147883    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdsrn\" (UniqueName: \"kubernetes.io/projected/64170c5f-e00d-47da-96ce-120e191dbb1a-kube-api-access-mdsrn\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.148347 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.148326    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bvprw\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.218374 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.218271    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:54.520136 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:54.520049    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"]
Sep 10 15:20:54.982152 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:20:54.982114    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64170c5f_e00d_47da_96ce_120e191dbb1a.slice/crio-8cc572aba9ab11e173aa7a7c5401e39229792ecb9fff640978d03399e719fd02 WatchSource:0}: Error finding container 8cc572aba9ab11e173aa7a7c5401e39229792ecb9fff640978d03399e719fd02: Status 404 returned error can't find the container with id 8cc572aba9ab11e173aa7a7c5401e39229792ecb9fff640978d03399e719fd02
Sep 10 15:20:55.267899 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:55.267783    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw" event={"ID":"64170c5f-e00d-47da-96ce-120e191dbb1a","Type":"ContainerStarted","Data":"8cc572aba9ab11e173aa7a7c5401e39229792ecb9fff640978d03399e719fd02"}
Sep 10 15:20:56.275360 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:56.275266    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm" event={"ID":"181199ba-a7a1-4031-a5ac-8b8eb120b694","Type":"ContainerStarted","Data":"46d98791dfba264090e4ee969021d1054a3411887d5d8d4205d8e02fed462e6a"}
Sep 10 15:20:56.299490 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:56.299414    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84bd548dcf-gt6gm" podStartSLOduration=1.372992566 podStartE2EDuration="4.299393098s" totalImagesPullingTime="2.926400532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:20:56.299078716 +0000 UTC m=+318.714874555" watchObservedRunningTime="2026-09-10 15:20:56.299393098 +0000 UTC m=+318.715188942"
Sep 10 15:20:57.580390 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:57.580350    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 10 15:20:57.580667 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:57.580422    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 10 15:20:58.285935 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:58.285882    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw" event={"ID":"64170c5f-e00d-47da-96ce-120e191dbb1a","Type":"ContainerStarted","Data":"104308670aec90475beb99a9cc7c4bfad30274c5ee6c045d5d4300008a6f8a9e"}
Sep 10 15:20:58.286117 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:58.286064    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:20:58.287349 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:58.287316    2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-bvprw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 15:20:58.287449 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:58.287380    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw" podUID="64170c5f-e00d-47da-96ce-120e191dbb1a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 15:20:58.313565 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:58.313491    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw" podStartSLOduration=2.717909915 podStartE2EDuration="5.313475293s" totalImagesPullingTime="2.595565378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:20:58.313446202 +0000 UTC m=+320.729242035" watchObservedRunningTime="2026-09-10 15:20:58.313475293 +0000 UTC m=+320.729271112"
Sep 10 15:20:59.289870 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:20:59.289838    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:21:14.467138 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.467100    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"]
Sep 10 15:21:14.475475 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.475450    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"
Sep 10 15:21:14.479692 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.479665    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 15:21:14.479897 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.479882    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8jd96\""
Sep 10 15:21:14.486492 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.486469    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59318b3f-2caa-4029-907d-63beb057d67e-cert\") pod \"kserve-controller-manager-d6f4b65b8-tc64f\" (UID: \"59318b3f-2caa-4029-907d-63beb057d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"
Sep 10 15:21:14.486578 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.486513    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw42t\" (UniqueName: \"kubernetes.io/projected/59318b3f-2caa-4029-907d-63beb057d67e-kube-api-access-fw42t\") pod \"kserve-controller-manager-d6f4b65b8-tc64f\" (UID: \"59318b3f-2caa-4029-907d-63beb057d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"
Sep 10 15:21:14.497025 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.497000    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"]
Sep 10 15:21:14.503313 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.503290    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29"]
Sep 10 15:21:14.505338 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.505318    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29"
Sep 10 15:21:14.510285 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.510264    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-h57mw\""
Sep 10 15:21:14.515864 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.515845    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 15:21:14.527530 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.527509    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29"]
Sep 10 15:21:14.587375 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.587342    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2d57895-c3bd-4629-baf0-b80543c9f60f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zzt29\" (UID: \"d2d57895-c3bd-4629-baf0-b80543c9f60f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29"
Sep 10 15:21:14.587534 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.587399    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw42t\" (UniqueName: \"kubernetes.io/projected/59318b3f-2caa-4029-907d-63beb057d67e-kube-api-access-fw42t\") pod \"kserve-controller-manager-d6f4b65b8-tc64f\" (UID: \"59318b3f-2caa-4029-907d-63beb057d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"
Sep 10 15:21:14.587534 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.587491    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59318b3f-2caa-4029-907d-63beb057d67e-cert\") pod \"kserve-controller-manager-d6f4b65b8-tc64f\" (UID: \"59318b3f-2caa-4029-907d-63beb057d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"
Sep 10 15:21:14.587534 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.587523    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85z2g\" (UniqueName: \"kubernetes.io/projected/d2d57895-c3bd-4629-baf0-b80543c9f60f-kube-api-access-85z2g\") pod \"llmisvc-controller-manager-777c6bd7d5-zzt29\" (UID: \"d2d57895-c3bd-4629-baf0-b80543c9f60f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29"
Sep 10 15:21:14.587650 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:21:14.587634    2573 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 10 15:21:14.587705 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:21:14.587694    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59318b3f-2caa-4029-907d-63beb057d67e-cert podName:59318b3f-2caa-4029-907d-63beb057d67e nodeName:}" failed. No retries permitted until 2026-09-10 15:21:15.087677634 +0000 UTC m=+337.503473448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/59318b3f-2caa-4029-907d-63beb057d67e-cert") pod "kserve-controller-manager-d6f4b65b8-tc64f" (UID: "59318b3f-2caa-4029-907d-63beb057d67e") : secret "kserve-webhook-server-cert" not found
Sep 10 15:21:14.604901 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.604867    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw42t\" (UniqueName: \"kubernetes.io/projected/59318b3f-2caa-4029-907d-63beb057d67e-kube-api-access-fw42t\") pod \"kserve-controller-manager-d6f4b65b8-tc64f\" (UID: \"59318b3f-2caa-4029-907d-63beb057d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"
Sep 10 15:21:14.688840 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.688785    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85z2g\" (UniqueName: \"kubernetes.io/projected/d2d57895-c3bd-4629-baf0-b80543c9f60f-kube-api-access-85z2g\") pod \"llmisvc-controller-manager-777c6bd7d5-zzt29\" (UID: \"d2d57895-c3bd-4629-baf0-b80543c9f60f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29"
Sep 10 15:21:14.688840 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.688841    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2d57895-c3bd-4629-baf0-b80543c9f60f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zzt29\" (UID: \"d2d57895-c3bd-4629-baf0-b80543c9f60f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29"
Sep 10 15:21:14.691415 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.691386    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2d57895-c3bd-4629-baf0-b80543c9f60f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zzt29\" (UID: \"d2d57895-c3bd-4629-baf0-b80543c9f60f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29"
Sep 10 15:21:14.705112 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.705087    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85z2g\" (UniqueName: \"kubernetes.io/projected/d2d57895-c3bd-4629-baf0-b80543c9f60f-kube-api-access-85z2g\") pod \"llmisvc-controller-manager-777c6bd7d5-zzt29\" (UID: \"d2d57895-c3bd-4629-baf0-b80543c9f60f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29"
Sep 10 15:21:14.815652 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:14.815564    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29"
Sep 10 15:21:15.051123 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.051093    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"]
Sep 10 15:21:15.051319 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.051295    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw" podUID="64170c5f-e00d-47da-96ce-120e191dbb1a" containerName="discovery" containerID="cri-o://104308670aec90475beb99a9cc7c4bfad30274c5ee6c045d5d4300008a6f8a9e" gracePeriod=30
Sep 10 15:21:15.090570 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.090541    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29"]
Sep 10 15:21:15.092489 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.092465    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59318b3f-2caa-4029-907d-63beb057d67e-cert\") pod \"kserve-controller-manager-d6f4b65b8-tc64f\" (UID: \"59318b3f-2caa-4029-907d-63beb057d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"
Sep 10 15:21:15.094977 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.094957    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59318b3f-2caa-4029-907d-63beb057d67e-cert\") pod \"kserve-controller-manager-d6f4b65b8-tc64f\" (UID: \"59318b3f-2caa-4029-907d-63beb057d67e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"
Sep 10 15:21:15.138626 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:21:15.138594    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd2d57895_c3bd_4629_baf0_b80543c9f60f.slice/crio-1cac80d22bf654f8a5f5008839330a48260adfee5e868b074bf00e7a7015e806 WatchSource:0}: Error finding container 1cac80d22bf654f8a5f5008839330a48260adfee5e868b074bf00e7a7015e806: Status 404 returned error can't find the container with id 1cac80d22bf654f8a5f5008839330a48260adfee5e868b074bf00e7a7015e806
Sep 10 15:21:15.283204 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.283180    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:21:15.348642 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.348558    2573 generic.go:356] "Generic (PLEG): container finished" podID="64170c5f-e00d-47da-96ce-120e191dbb1a" containerID="104308670aec90475beb99a9cc7c4bfad30274c5ee6c045d5d4300008a6f8a9e" exitCode=0
Sep 10 15:21:15.348642 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.348603    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"
Sep 10 15:21:15.348883 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.348651    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw" event={"ID":"64170c5f-e00d-47da-96ce-120e191dbb1a","Type":"ContainerDied","Data":"104308670aec90475beb99a9cc7c4bfad30274c5ee6c045d5d4300008a6f8a9e"}
Sep 10 15:21:15.348883 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.348695    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw" event={"ID":"64170c5f-e00d-47da-96ce-120e191dbb1a","Type":"ContainerDied","Data":"8cc572aba9ab11e173aa7a7c5401e39229792ecb9fff640978d03399e719fd02"}
Sep 10 15:21:15.348883 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.348714    2573 scope.go:117] "RemoveContainer" containerID="104308670aec90475beb99a9cc7c4bfad30274c5ee6c045d5d4300008a6f8a9e"
Sep 10 15:21:15.354200 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.354056    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29" event={"ID":"d2d57895-c3bd-4629-baf0-b80543c9f60f","Type":"ContainerStarted","Data":"1cac80d22bf654f8a5f5008839330a48260adfee5e868b074bf00e7a7015e806"}
Sep 10 15:21:15.358878 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.358859    2573 scope.go:117] "RemoveContainer" containerID="104308670aec90475beb99a9cc7c4bfad30274c5ee6c045d5d4300008a6f8a9e"
Sep 10 15:21:15.359154 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:21:15.359130    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104308670aec90475beb99a9cc7c4bfad30274c5ee6c045d5d4300008a6f8a9e\": container with ID starting with 104308670aec90475beb99a9cc7c4bfad30274c5ee6c045d5d4300008a6f8a9e not found: ID does not exist" containerID="104308670aec90475beb99a9cc7c4bfad30274c5ee6c045d5d4300008a6f8a9e"
Sep 10 15:21:15.359197 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.359157    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104308670aec90475beb99a9cc7c4bfad30274c5ee6c045d5d4300008a6f8a9e"} err="failed to get container status \"104308670aec90475beb99a9cc7c4bfad30274c5ee6c045d5d4300008a6f8a9e\": rpc error: code = NotFound desc = could not find container \"104308670aec90475beb99a9cc7c4bfad30274c5ee6c045d5d4300008a6f8a9e\": container with ID starting with 104308670aec90475beb99a9cc7c4bfad30274c5ee6c045d5d4300008a6f8a9e not found: ID does not exist"
Sep 10 15:21:15.385465 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.385448    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"
Sep 10 15:21:15.394979 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.394919    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-token\") pod \"64170c5f-e00d-47da-96ce-120e191dbb1a\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") "
Sep 10 15:21:15.394979 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.394956    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-cacerts\") pod \"64170c5f-e00d-47da-96ce-120e191dbb1a\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") "
Sep 10 15:21:15.395123 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.394989    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64170c5f-e00d-47da-96ce-120e191dbb1a-local-certs\") pod \"64170c5f-e00d-47da-96ce-120e191dbb1a\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") "
Sep 10 15:21:15.395123 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.395111    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-csr-ca-configmap\") pod \"64170c5f-e00d-47da-96ce-120e191dbb1a\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") "
Sep 10 15:21:15.395228 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.395165    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdsrn\" (UniqueName: \"kubernetes.io/projected/64170c5f-e00d-47da-96ce-120e191dbb1a-kube-api-access-mdsrn\") pod \"64170c5f-e00d-47da-96ce-120e191dbb1a\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") "
Sep 10 15:21:15.395228 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.395197    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-kubeconfig\") pod \"64170c5f-e00d-47da-96ce-120e191dbb1a\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") "
Sep 10 15:21:15.395228 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.395215    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-csr-dns-cert\") pod \"64170c5f-e00d-47da-96ce-120e191dbb1a\" (UID: \"64170c5f-e00d-47da-96ce-120e191dbb1a\") "
Sep 10 15:21:15.395496 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.395453    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "64170c5f-e00d-47da-96ce-120e191dbb1a" (UID: "64170c5f-e00d-47da-96ce-120e191dbb1a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:21:15.397406 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.397381    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-cacerts" (OuterVolumeSpecName: "cacerts") pod "64170c5f-e00d-47da-96ce-120e191dbb1a" (UID: "64170c5f-e00d-47da-96ce-120e191dbb1a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:21:15.397593 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.397568    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64170c5f-e00d-47da-96ce-120e191dbb1a-local-certs" (OuterVolumeSpecName: "local-certs") pod "64170c5f-e00d-47da-96ce-120e191dbb1a" (UID: "64170c5f-e00d-47da-96ce-120e191dbb1a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:21:15.397593 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.397572    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-token" (OuterVolumeSpecName: "istio-token") pod "64170c5f-e00d-47da-96ce-120e191dbb1a" (UID: "64170c5f-e00d-47da-96ce-120e191dbb1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:21:15.397980 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.397957    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "64170c5f-e00d-47da-96ce-120e191dbb1a" (UID: "64170c5f-e00d-47da-96ce-120e191dbb1a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:21:15.397980 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.397967    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64170c5f-e00d-47da-96ce-120e191dbb1a-kube-api-access-mdsrn" (OuterVolumeSpecName: "kube-api-access-mdsrn") pod "64170c5f-e00d-47da-96ce-120e191dbb1a" (UID: "64170c5f-e00d-47da-96ce-120e191dbb1a"). InnerVolumeSpecName "kube-api-access-mdsrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:21:15.398258 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.398242    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "64170c5f-e00d-47da-96ce-120e191dbb1a" (UID: "64170c5f-e00d-47da-96ce-120e191dbb1a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:21:15.496086 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.496046    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdsrn\" (UniqueName: \"kubernetes.io/projected/64170c5f-e00d-47da-96ce-120e191dbb1a-kube-api-access-mdsrn\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:21:15.496086 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.496078    2573 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-kubeconfig\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:21:15.496086 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.496088    2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-csr-dns-cert\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:21:15.496086 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.496096    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-token\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:21:15.496529 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.496105    2573 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/64170c5f-e00d-47da-96ce-120e191dbb1a-cacerts\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:21:15.496529 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.496112    2573 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/64170c5f-e00d-47da-96ce-120e191dbb1a-local-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:21:15.496529 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.496120    2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/64170c5f-e00d-47da-96ce-120e191dbb1a-istio-csr-ca-configmap\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:21:15.662785 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.662753    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"]
Sep 10 15:21:15.665483 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:21:15.665457    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59318b3f_2caa_4029_907d_63beb057d67e.slice/crio-f2d384ff6efb66ae4e1e9ecef92c70a7d3598c95fcf6d2d5be674c31d0906a86 WatchSource:0}: Error finding container f2d384ff6efb66ae4e1e9ecef92c70a7d3598c95fcf6d2d5be674c31d0906a86: Status 404 returned error can't find the container with id f2d384ff6efb66ae4e1e9ecef92c70a7d3598c95fcf6d2d5be674c31d0906a86
Sep 10 15:21:15.681659 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.681634    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"]
Sep 10 15:21:15.698211 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:15.698180    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bvprw"]
Sep 10 15:21:16.125948 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:16.125869    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64170c5f-e00d-47da-96ce-120e191dbb1a" path="/var/lib/kubelet/pods/64170c5f-e00d-47da-96ce-120e191dbb1a/volumes"
Sep 10 15:21:16.360396 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:16.360364    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f" event={"ID":"59318b3f-2caa-4029-907d-63beb057d67e","Type":"ContainerStarted","Data":"f2d384ff6efb66ae4e1e9ecef92c70a7d3598c95fcf6d2d5be674c31d0906a86"}
Sep 10 15:21:19.375177 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:19.375142    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f" event={"ID":"59318b3f-2caa-4029-907d-63beb057d67e","Type":"ContainerStarted","Data":"f173c8502bb796c7d8c5638bba92284c0913e462f49f0d17376bec2e47890079"}
Sep 10 15:21:19.375610 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:19.375307    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"
Sep 10 15:21:19.381446 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:19.381418    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29" event={"ID":"d2d57895-c3bd-4629-baf0-b80543c9f60f","Type":"ContainerStarted","Data":"bf4c9462be933006bc88c06fe7a344c8be468262d384ec4d710630b29c98bc1b"}
Sep 10 15:21:19.381596 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:19.381488    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29"
Sep 10 15:21:19.404161 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:19.404116    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f" podStartSLOduration=2.364954386 podStartE2EDuration="5.404103555s" totalImagesPullingTime="3.039149169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:21:19.402519048 +0000 UTC m=+341.818314869" watchObservedRunningTime="2026-09-10 15:21:19.404103555 +0000 UTC m=+341.819899376"
Sep 10 15:21:50.388185 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:50.388152    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29"
Sep 10 15:21:50.391285 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:50.391263    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tc64f"
Sep 10 15:21:50.412997 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:21:50.412946    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zzt29" podStartSLOduration=32.864881092 podStartE2EDuration="36.412930362s" totalImagesPullingTime="3.54804927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:21:19.450928394 +0000 UTC m=+341.866724213" watchObservedRunningTime="2026-09-10 15:21:50.412930362 +0000 UTC m=+372.828726235"
Sep 10 15:22:19.413392 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:19.413351    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-jtnw6"]
Sep 10 15:22:19.413818 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:19.413618    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64170c5f-e00d-47da-96ce-120e191dbb1a" containerName="discovery"
Sep 10 15:22:19.413818 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:19.413629    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="64170c5f-e00d-47da-96ce-120e191dbb1a" containerName="discovery"
Sep 10 15:22:19.413818 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:19.413673    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="64170c5f-e00d-47da-96ce-120e191dbb1a" containerName="discovery"
Sep 10 15:22:19.416770 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:19.416753    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-jtnw6"
Sep 10 15:22:19.419475 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:19.419452    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 15:22:19.419475 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:19.419452    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-g7mzf\""
Sep 10 15:22:19.419660 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:19.419494    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 15:22:19.428032 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:19.428010    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-jtnw6"]
Sep 10 15:22:19.483711 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:19.483667    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49c46\" (UniqueName: \"kubernetes.io/projected/d5604c15-0aca-435f-ae29-db31db0eb6ea-kube-api-access-49c46\") pod \"authorino-operator-76b478bd78-jtnw6\" (UID: \"d5604c15-0aca-435f-ae29-db31db0eb6ea\") " pod="kuadrant-system/authorino-operator-76b478bd78-jtnw6"
Sep 10 15:22:19.585055 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:19.585019    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49c46\" (UniqueName: \"kubernetes.io/projected/d5604c15-0aca-435f-ae29-db31db0eb6ea-kube-api-access-49c46\") pod \"authorino-operator-76b478bd78-jtnw6\" (UID: \"d5604c15-0aca-435f-ae29-db31db0eb6ea\") " pod="kuadrant-system/authorino-operator-76b478bd78-jtnw6"
Sep 10 15:22:19.594586 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:19.594565    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49c46\" (UniqueName: \"kubernetes.io/projected/d5604c15-0aca-435f-ae29-db31db0eb6ea-kube-api-access-49c46\") pod \"authorino-operator-76b478bd78-jtnw6\" (UID: \"d5604c15-0aca-435f-ae29-db31db0eb6ea\") " pod="kuadrant-system/authorino-operator-76b478bd78-jtnw6"
Sep 10 15:22:19.726844 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:19.726718    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-jtnw6"
Sep 10 15:22:20.015908 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:20.015884    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-jtnw6"]
Sep 10 15:22:20.018166 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:22:20.018141    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5604c15_0aca_435f_ae29_db31db0eb6ea.slice/crio-aadfb2b22e4213b7d2e4cc4c482b8874c31d2b4e0a58fdd267472f3e00668fe8 WatchSource:0}: Error finding container aadfb2b22e4213b7d2e4cc4c482b8874c31d2b4e0a58fdd267472f3e00668fe8: Status 404 returned error can't find the container with id aadfb2b22e4213b7d2e4cc4c482b8874c31d2b4e0a58fdd267472f3e00668fe8
Sep 10 15:22:20.617759 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:20.617720    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-jtnw6" event={"ID":"d5604c15-0aca-435f-ae29-db31db0eb6ea","Type":"ContainerStarted","Data":"aadfb2b22e4213b7d2e4cc4c482b8874c31d2b4e0a58fdd267472f3e00668fe8"}
Sep 10 15:22:21.632351 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:21.632323    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5tz85"]
Sep 10 15:22:21.635430 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:21.635401    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5tz85"
Sep 10 15:22:21.638517 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:21.638484    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zv84l\""
Sep 10 15:22:21.638660 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:21.638544    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 15:22:21.652269 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:21.651743    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5tz85"]
Sep 10 15:22:21.699027 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:21.698986    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbqb7\" (UniqueName: \"kubernetes.io/projected/c34331c8-06d5-4b13-8901-36325d5c2a6e-kube-api-access-gbqb7\") pod \"dns-operator-controller-manager-7b84cd7c9c-5tz85\" (UID: \"c34331c8-06d5-4b13-8901-36325d5c2a6e\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5tz85"
Sep 10 15:22:21.800335 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:21.800299    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbqb7\" (UniqueName: \"kubernetes.io/projected/c34331c8-06d5-4b13-8901-36325d5c2a6e-kube-api-access-gbqb7\") pod \"dns-operator-controller-manager-7b84cd7c9c-5tz85\" (UID: \"c34331c8-06d5-4b13-8901-36325d5c2a6e\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5tz85"
Sep 10 15:22:21.818973 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:21.818936    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbqb7\" (UniqueName: \"kubernetes.io/projected/c34331c8-06d5-4b13-8901-36325d5c2a6e-kube-api-access-gbqb7\") pod \"dns-operator-controller-manager-7b84cd7c9c-5tz85\" (UID: \"c34331c8-06d5-4b13-8901-36325d5c2a6e\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5tz85"
Sep 10 15:22:21.954538 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:21.954457    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5tz85"
Sep 10 15:22:23.135262 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:23.135180    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5tz85"]
Sep 10 15:22:23.138393 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:22:23.138366    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34331c8_06d5_4b13_8901_36325d5c2a6e.slice/crio-7c886b0062d23e8e1c0481fccd3666fb41e9b95897f4ded148062a9ce3b8e6b2 WatchSource:0}: Error finding container 7c886b0062d23e8e1c0481fccd3666fb41e9b95897f4ded148062a9ce3b8e6b2: Status 404 returned error can't find the container with id 7c886b0062d23e8e1c0481fccd3666fb41e9b95897f4ded148062a9ce3b8e6b2
Sep 10 15:22:23.634178 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:23.634136    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-jtnw6" event={"ID":"d5604c15-0aca-435f-ae29-db31db0eb6ea","Type":"ContainerStarted","Data":"582ac1da4c9b34a7b6ab18fa24abf7dbd5c0ab429c4a54635708ae2b02b27e1e"}
Sep 10 15:22:23.634392 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:23.634264    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-jtnw6"
Sep 10 15:22:23.635469 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:23.635448    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5tz85" event={"ID":"c34331c8-06d5-4b13-8901-36325d5c2a6e","Type":"ContainerStarted","Data":"7c886b0062d23e8e1c0481fccd3666fb41e9b95897f4ded148062a9ce3b8e6b2"}
Sep 10 15:22:23.660630 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:23.660578    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-jtnw6" podStartSLOduration=1.749295655 podStartE2EDuration="4.660562705s" totalImagesPullingTime="2.91126705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:22:23.6589774 +0000 UTC m=+406.074773210" watchObservedRunningTime="2026-09-10 15:22:23.660562705 +0000 UTC m=+406.076358525"
Sep 10 15:22:25.648395 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:25.648357    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5tz85" event={"ID":"c34331c8-06d5-4b13-8901-36325d5c2a6e","Type":"ContainerStarted","Data":"c2bff10a1fc6aaabefb5cd826df12694db7938a79f0170bed08d1df5f7b4b460"}
Sep 10 15:22:25.648821 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:25.648481    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5tz85"
Sep 10 15:22:25.680090 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:25.680027    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5tz85" podStartSLOduration=2.88636101 podStartE2EDuration="4.680007758s" totalImagesPullingTime="1.793646748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:22:25.677946929 +0000 UTC m=+408.093742747" watchObservedRunningTime="2026-09-10 15:22:25.680007758 +0000 UTC m=+408.095803576"
Sep 10 15:22:26.195189 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:26.195154    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"]
Sep 10 15:22:26.198347 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:26.198331    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:26.204498 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:26.204474    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-g5nj9\""
Sep 10 15:22:26.213693 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:26.213670    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"]
Sep 10 15:22:26.233849 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:26.233814    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" (UID: \"9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:26.234001 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:26.233876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxjh\" (UniqueName: \"kubernetes.io/projected/9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323-kube-api-access-xmxjh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" (UID: \"9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:26.334557 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:26.334519    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" (UID: \"9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:26.334690 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:26.334584    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxjh\" (UniqueName: \"kubernetes.io/projected/9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323-kube-api-access-xmxjh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" (UID: \"9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:26.334944 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:26.334925    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" (UID: \"9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:26.346740 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:26.346710    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxjh\" (UniqueName: \"kubernetes.io/projected/9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323-kube-api-access-xmxjh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" (UID: \"9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:26.513485 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:26.513396    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:26.794492 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:26.794446    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"]
Sep 10 15:22:26.797286 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:22:26.797255    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f63dcd2_64eb_4a99_9d2a_9f0b2e0e8323.slice/crio-3d3782b9474d6ab1f7f85b9ebb05736bb8e931887f52f37805258fa5c5222c8a WatchSource:0}: Error finding container 3d3782b9474d6ab1f7f85b9ebb05736bb8e931887f52f37805258fa5c5222c8a: Status 404 returned error can't find the container with id 3d3782b9474d6ab1f7f85b9ebb05736bb8e931887f52f37805258fa5c5222c8a
Sep 10 15:22:27.661109 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:27.661063    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d" event={"ID":"9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323","Type":"ContainerStarted","Data":"3d3782b9474d6ab1f7f85b9ebb05736bb8e931887f52f37805258fa5c5222c8a"}
Sep 10 15:22:28.440750 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:28.440712    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"]
Sep 10 15:22:28.444470 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:28.444294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:28.446903 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:28.446858    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-px2cx\""
Sep 10 15:22:28.452359 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:28.452328    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxkl\" (UniqueName: \"kubernetes.io/projected/b0ce9546-427a-4f9f-8800-3b2cefbc471d-kube-api-access-tpxkl\") pod \"limitador-operator-controller-manager-74495c69f8-88qk4\" (UID: \"b0ce9546-427a-4f9f-8800-3b2cefbc471d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:28.457084 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:28.457024    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"]
Sep 10 15:22:28.553406 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:28.553370    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxkl\" (UniqueName: \"kubernetes.io/projected/b0ce9546-427a-4f9f-8800-3b2cefbc471d-kube-api-access-tpxkl\") pod \"limitador-operator-controller-manager-74495c69f8-88qk4\" (UID: \"b0ce9546-427a-4f9f-8800-3b2cefbc471d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:28.574805 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:28.574758    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxkl\" (UniqueName: \"kubernetes.io/projected/b0ce9546-427a-4f9f-8800-3b2cefbc471d-kube-api-access-tpxkl\") pod \"limitador-operator-controller-manager-74495c69f8-88qk4\" (UID: \"b0ce9546-427a-4f9f-8800-3b2cefbc471d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:28.757318 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:28.757232    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:29.040477 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:29.040402    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"]
Sep 10 15:22:29.044212 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:22:29.044168    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ce9546_427a_4f9f_8800_3b2cefbc471d.slice/crio-461b3f0983979bcb44a4cc0efab9043419ad2313b658ae9799e80718382141fd WatchSource:0}: Error finding container 461b3f0983979bcb44a4cc0efab9043419ad2313b658ae9799e80718382141fd: Status 404 returned error can't find the container with id 461b3f0983979bcb44a4cc0efab9043419ad2313b658ae9799e80718382141fd
Sep 10 15:22:29.673535 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:29.673501    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4" event={"ID":"b0ce9546-427a-4f9f-8800-3b2cefbc471d","Type":"ContainerStarted","Data":"461b3f0983979bcb44a4cc0efab9043419ad2313b658ae9799e80718382141fd"}
Sep 10 15:22:31.691516 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:31.691412    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d" event={"ID":"9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323","Type":"ContainerStarted","Data":"83664fea6093ec8053b7f1b9fc8a168cd50745184d70b9259a68ff8f864600c2"}
Sep 10 15:22:31.691925 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:31.691524    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:31.693105 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:31.693078    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4" event={"ID":"b0ce9546-427a-4f9f-8800-3b2cefbc471d","Type":"ContainerStarted","Data":"6a946b8d3dd8e5624df5417573ec91604afd186572a9ad3ee75fb29c37d0dda3"}
Sep 10 15:22:31.693213 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:31.693191    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:31.722809 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:31.722733    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d" podStartSLOduration=1.8632218520000001 podStartE2EDuration="5.722719454s" totalImagesPullingTime="3.859497602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:22:31.721937729 +0000 UTC m=+414.137733548" watchObservedRunningTime="2026-09-10 15:22:31.722719454 +0000 UTC m=+414.138515272"
Sep 10 15:22:31.758480 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:31.758429    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4" podStartSLOduration=1.372565869 podStartE2EDuration="3.758409497s" totalImagesPullingTime="2.385843628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:22:31.757368064 +0000 UTC m=+414.173163885" watchObservedRunningTime="2026-09-10 15:22:31.758409497 +0000 UTC m=+414.174205317"
Sep 10 15:22:34.642896 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:34.642867    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-jtnw6"
Sep 10 15:22:36.656044 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:36.656014    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5tz85"
Sep 10 15:22:42.702431 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:42.702397    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:42.702888 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:42.702626    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:43.974458 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:43.974422    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"]
Sep 10 15:22:43.977890 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:43.977873    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:43.990151 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:43.990128    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"]
Sep 10 15:22:44.086311 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.086268    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjm24\" (UniqueName: \"kubernetes.io/projected/1ddfea82-5ca7-4580-9af7-769fe477a8a1-kube-api-access-zjm24\") pod \"kuadrant-operator-controller-manager-5f685c496f-c22l8\" (UID: \"1ddfea82-5ca7-4580-9af7-769fe477a8a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:44.086311 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.086314    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ddfea82-5ca7-4580-9af7-769fe477a8a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-c22l8\" (UID: \"1ddfea82-5ca7-4580-9af7-769fe477a8a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:44.187372 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.187333    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjm24\" (UniqueName: \"kubernetes.io/projected/1ddfea82-5ca7-4580-9af7-769fe477a8a1-kube-api-access-zjm24\") pod \"kuadrant-operator-controller-manager-5f685c496f-c22l8\" (UID: \"1ddfea82-5ca7-4580-9af7-769fe477a8a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:44.187539 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.187376    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ddfea82-5ca7-4580-9af7-769fe477a8a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-c22l8\" (UID: \"1ddfea82-5ca7-4580-9af7-769fe477a8a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:44.187811 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.187767    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ddfea82-5ca7-4580-9af7-769fe477a8a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-c22l8\" (UID: \"1ddfea82-5ca7-4580-9af7-769fe477a8a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:44.196727 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.196702    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjm24\" (UniqueName: \"kubernetes.io/projected/1ddfea82-5ca7-4580-9af7-769fe477a8a1-kube-api-access-zjm24\") pod \"kuadrant-operator-controller-manager-5f685c496f-c22l8\" (UID: \"1ddfea82-5ca7-4580-9af7-769fe477a8a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:44.287663 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.287574    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:44.566189 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.566160    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"]
Sep 10 15:22:44.569069 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:22:44.569044    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ddfea82_5ca7_4580_9af7_769fe477a8a1.slice/crio-db8531ba53f6bcd7b1ea410a4510569cc4195df332efe4ca8b7e86c6b9acb4e3 WatchSource:0}: Error finding container db8531ba53f6bcd7b1ea410a4510569cc4195df332efe4ca8b7e86c6b9acb4e3: Status 404 returned error can't find the container with id db8531ba53f6bcd7b1ea410a4510569cc4195df332efe4ca8b7e86c6b9acb4e3
Sep 10 15:22:44.753308 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.753267    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8" event={"ID":"1ddfea82-5ca7-4580-9af7-769fe477a8a1","Type":"ContainerStarted","Data":"f88b5d60295b5422278e7f1ba7dae4df4fa92ce66d8980541c0bd6a8df48e963"}
Sep 10 15:22:44.753489 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.753324    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8" event={"ID":"1ddfea82-5ca7-4580-9af7-769fe477a8a1","Type":"ContainerStarted","Data":"db8531ba53f6bcd7b1ea410a4510569cc4195df332efe4ca8b7e86c6b9acb4e3"}
Sep 10 15:22:44.753489 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.753371    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:44.778591 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.778536    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8" podStartSLOduration=1.778516432 podStartE2EDuration="1.778516432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:22:44.776575275 +0000 UTC m=+427.192371094" watchObservedRunningTime="2026-09-10 15:22:44.778516432 +0000 UTC m=+427.194312252"
Sep 10 15:22:44.940548 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.940504    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"]
Sep 10 15:22:44.955319 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.955292    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"]
Sep 10 15:22:44.966479 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.966450    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"]
Sep 10 15:22:44.966663 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.966636    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" containerName="manager" containerID="cri-o://83664fea6093ec8053b7f1b9fc8a168cd50745184d70b9259a68ff8f864600c2" gracePeriod=2
Sep 10 15:22:44.983112 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.983077    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"]
Sep 10 15:22:44.985584 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.985558    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"]
Sep 10 15:22:44.985894 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.985879    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" containerName="manager"
Sep 10 15:22:44.985955 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.985895    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" containerName="manager"
Sep 10 15:22:44.985955 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.985926    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ddfea82-5ca7-4580-9af7-769fe477a8a1" containerName="manager"
Sep 10 15:22:44.985955 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.985933    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ddfea82-5ca7-4580-9af7-769fe477a8a1" containerName="manager"
Sep 10 15:22:44.986042 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.985990    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ddfea82-5ca7-4580-9af7-769fe477a8a1" containerName="manager"
Sep 10 15:22:44.986042 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.985999    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" containerName="manager"
Sep 10 15:22:44.988695 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:44.988678    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"
Sep 10 15:22:45.012837 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.012810    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"]
Sep 10 15:22:45.017406 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.017381    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"]
Sep 10 15:22:45.017716 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.017662    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" containerName="manager" containerID="cri-o://6a946b8d3dd8e5624df5417573ec91604afd186572a9ad3ee75fb29c37d0dda3" gracePeriod=2
Sep 10 15:22:45.023441 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.023415    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"]
Sep 10 15:22:45.029063 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.029039    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"]
Sep 10 15:22:45.029348 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.029335    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" containerName="manager"
Sep 10 15:22:45.029400 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.029350    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" containerName="manager"
Sep 10 15:22:45.029400 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.029395    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" containerName="manager"
Sep 10 15:22:45.032083 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.032066    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"
Sep 10 15:22:45.055840 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.055723    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:45.056715 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.056669    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"]
Sep 10 15:22:45.058605 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.058235    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:45.060539 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.060488    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-88qk4\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:45.060667 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.060587    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-62jls"]
Sep 10 15:22:45.064259 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.064236    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-62jls"
Sep 10 15:22:45.090635 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.090605    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-62jls"]
Sep 10 15:22:45.092332 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.092186    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:45.093836 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.093780    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e83ef995-aa99-462e-9b20-7bec95464032-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k6gzw\" (UID: \"e83ef995-aa99-462e-9b20-7bec95464032\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"
Sep 10 15:22:45.093941 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.093882    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8e3f3c72-bc7e-40bd-9719-8358842cec03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-jzqb6\" (UID: \"8e3f3c72-bc7e-40bd-9719-8358842cec03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"
Sep 10 15:22:45.093941 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.093922    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll4kr\" (UniqueName: \"kubernetes.io/projected/e83ef995-aa99-462e-9b20-7bec95464032-kube-api-access-ll4kr\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k6gzw\" (UID: \"e83ef995-aa99-462e-9b20-7bec95464032\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"
Sep 10 15:22:45.094051 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.093977    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k95bw\" (UniqueName: \"kubernetes.io/projected/8e3f3c72-bc7e-40bd-9719-8358842cec03-kube-api-access-k95bw\") pod \"kuadrant-operator-controller-manager-5f685c496f-jzqb6\" (UID: \"8e3f3c72-bc7e-40bd-9719-8358842cec03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"
Sep 10 15:22:45.117306 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.117263    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-88qk4\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:45.194652 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.194511    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e83ef995-aa99-462e-9b20-7bec95464032-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k6gzw\" (UID: \"e83ef995-aa99-462e-9b20-7bec95464032\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"
Sep 10 15:22:45.194652 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.194580    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8e3f3c72-bc7e-40bd-9719-8358842cec03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-jzqb6\" (UID: \"8e3f3c72-bc7e-40bd-9719-8358842cec03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"
Sep 10 15:22:45.194652 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.194618    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qrt\" (UniqueName: \"kubernetes.io/projected/87213ea5-dbc7-46f0-abfc-613d537ad421-kube-api-access-b9qrt\") pod \"limitador-operator-controller-manager-74495c69f8-62jls\" (UID: \"87213ea5-dbc7-46f0-abfc-613d537ad421\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-62jls"
Sep 10 15:22:45.195011 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.194663    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll4kr\" (UniqueName: \"kubernetes.io/projected/e83ef995-aa99-462e-9b20-7bec95464032-kube-api-access-ll4kr\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k6gzw\" (UID: \"e83ef995-aa99-462e-9b20-7bec95464032\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"
Sep 10 15:22:45.195011 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.194737    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k95bw\" (UniqueName: \"kubernetes.io/projected/8e3f3c72-bc7e-40bd-9719-8358842cec03-kube-api-access-k95bw\") pod \"kuadrant-operator-controller-manager-5f685c496f-jzqb6\" (UID: \"8e3f3c72-bc7e-40bd-9719-8358842cec03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"
Sep 10 15:22:45.195468 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.195417    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8e3f3c72-bc7e-40bd-9719-8358842cec03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-jzqb6\" (UID: \"8e3f3c72-bc7e-40bd-9719-8358842cec03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"
Sep 10 15:22:45.195553 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.195477    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e83ef995-aa99-462e-9b20-7bec95464032-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k6gzw\" (UID: \"e83ef995-aa99-462e-9b20-7bec95464032\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"
Sep 10 15:22:45.205868 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.205845    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll4kr\" (UniqueName: \"kubernetes.io/projected/e83ef995-aa99-462e-9b20-7bec95464032-kube-api-access-ll4kr\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k6gzw\" (UID: \"e83ef995-aa99-462e-9b20-7bec95464032\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"
Sep 10 15:22:45.205961 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.205910    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k95bw\" (UniqueName: \"kubernetes.io/projected/8e3f3c72-bc7e-40bd-9719-8358842cec03-kube-api-access-k95bw\") pod \"kuadrant-operator-controller-manager-5f685c496f-jzqb6\" (UID: \"8e3f3c72-bc7e-40bd-9719-8358842cec03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"
Sep 10 15:22:45.252854 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.252830    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:45.254825 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.254807    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:45.255756 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.255737    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:45.257701 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.257682    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-88qk4\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:45.259879 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.259854    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-88qk4\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:45.261910 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.261882    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:45.295602 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.295574    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323-extensions-socket-volume\") pod \"9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323\" (UID: \"9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323\") "
Sep 10 15:22:45.295760 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.295625    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmxjh\" (UniqueName: \"kubernetes.io/projected/9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323-kube-api-access-xmxjh\") pod \"9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323\" (UID: \"9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323\") "
Sep 10 15:22:45.295760 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.295741    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qrt\" (UniqueName: \"kubernetes.io/projected/87213ea5-dbc7-46f0-abfc-613d537ad421-kube-api-access-b9qrt\") pod \"limitador-operator-controller-manager-74495c69f8-62jls\" (UID: \"87213ea5-dbc7-46f0-abfc-613d537ad421\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-62jls"
Sep 10 15:22:45.295974 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.295951    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" (UID: "9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:22:45.297921 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.297899    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323-kube-api-access-xmxjh" (OuterVolumeSpecName: "kube-api-access-xmxjh") pod "9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" (UID: "9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323"). InnerVolumeSpecName "kube-api-access-xmxjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:22:45.304848 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.304820    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qrt\" (UniqueName: \"kubernetes.io/projected/87213ea5-dbc7-46f0-abfc-613d537ad421-kube-api-access-b9qrt\") pod \"limitador-operator-controller-manager-74495c69f8-62jls\" (UID: \"87213ea5-dbc7-46f0-abfc-613d537ad421\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-62jls"
Sep 10 15:22:45.396760 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.396734    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpxkl\" (UniqueName: \"kubernetes.io/projected/b0ce9546-427a-4f9f-8800-3b2cefbc471d-kube-api-access-tpxkl\") pod \"b0ce9546-427a-4f9f-8800-3b2cefbc471d\" (UID: \"b0ce9546-427a-4f9f-8800-3b2cefbc471d\") "
Sep 10 15:22:45.396962 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.396926    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmxjh\" (UniqueName: \"kubernetes.io/projected/9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323-kube-api-access-xmxjh\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:22:45.396962 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.396942    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323-extensions-socket-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:22:45.398944 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.398918    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ce9546-427a-4f9f-8800-3b2cefbc471d-kube-api-access-tpxkl" (OuterVolumeSpecName: "kube-api-access-tpxkl") pod "b0ce9546-427a-4f9f-8800-3b2cefbc471d" (UID: "b0ce9546-427a-4f9f-8800-3b2cefbc471d"). InnerVolumeSpecName "kube-api-access-tpxkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:22:45.406063 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.406044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"
Sep 10 15:22:45.412905 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.412878    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"
Sep 10 15:22:45.419355 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.418874    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-62jls"
Sep 10 15:22:45.497817 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.497727    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpxkl\" (UniqueName: \"kubernetes.io/projected/b0ce9546-427a-4f9f-8800-3b2cefbc471d-kube-api-access-tpxkl\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:22:45.699556 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.699526    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"]
Sep 10 15:22:45.701550 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:22:45.701525    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3f3c72_bc7e_40bd_9719_8358842cec03.slice/crio-97d7c17cd78018c6440cf865cbf132e2fd56d310c1358c6edb0b0e2a2890344a WatchSource:0}: Error finding container 97d7c17cd78018c6440cf865cbf132e2fd56d310c1358c6edb0b0e2a2890344a: Status 404 returned error can't find the container with id 97d7c17cd78018c6440cf865cbf132e2fd56d310c1358c6edb0b0e2a2890344a
Sep 10 15:22:45.722957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.722928    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-62jls"]
Sep 10 15:22:45.724253 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.724231    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"]
Sep 10 15:22:45.733997 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:22:45.733971    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87213ea5_dbc7_46f0_abfc_613d537ad421.slice/crio-da9455ed209c851ad1218669a7a35c8e409ffbb1c33db903c4a2bb3851f05862 WatchSource:0}: Error finding container da9455ed209c851ad1218669a7a35c8e409ffbb1c33db903c4a2bb3851f05862: Status 404 returned error can't find the container with id da9455ed209c851ad1218669a7a35c8e409ffbb1c33db903c4a2bb3851f05862
Sep 10 15:22:45.734197 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:22:45.734178    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83ef995_aa99_462e_9b20_7bec95464032.slice/crio-22e0a530f4f141259fb15a17799ad79aee696d9186d8f66c09dc9b3768eacc21 WatchSource:0}: Error finding container 22e0a530f4f141259fb15a17799ad79aee696d9186d8f66c09dc9b3768eacc21: Status 404 returned error can't find the container with id 22e0a530f4f141259fb15a17799ad79aee696d9186d8f66c09dc9b3768eacc21
Sep 10 15:22:45.760574 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.760543    2573 generic.go:356] "Generic (PLEG): container finished" podID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" containerID="83664fea6093ec8053b7f1b9fc8a168cd50745184d70b9259a68ff8f864600c2" exitCode=0
Sep 10 15:22:45.760718 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.760590    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:45.760718 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.760670    2573 scope.go:117] "RemoveContainer" containerID="83664fea6093ec8053b7f1b9fc8a168cd50745184d70b9259a68ff8f864600c2"
Sep 10 15:22:45.763203 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.763097    2573 generic.go:356] "Generic (PLEG): container finished" podID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" containerID="6a946b8d3dd8e5624df5417573ec91604afd186572a9ad3ee75fb29c37d0dda3" exitCode=0
Sep 10 15:22:45.763203 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.763152    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:45.763947 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.763871    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:45.765240 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.765215    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-62jls" event={"ID":"87213ea5-dbc7-46f0-abfc-613d537ad421","Type":"ContainerStarted","Data":"da9455ed209c851ad1218669a7a35c8e409ffbb1c33db903c4a2bb3851f05862"}
Sep 10 15:22:45.766469 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.766445    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-88qk4\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:45.768423 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.768358    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw" event={"ID":"e83ef995-aa99-462e-9b20-7bec95464032","Type":"ContainerStarted","Data":"22e0a530f4f141259fb15a17799ad79aee696d9186d8f66c09dc9b3768eacc21"}
Sep 10 15:22:45.768761 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.768582    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:45.771876 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.771401    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6" event={"ID":"8e3f3c72-bc7e-40bd-9719-8358842cec03","Type":"ContainerStarted","Data":"97d7c17cd78018c6440cf865cbf132e2fd56d310c1358c6edb0b0e2a2890344a"}
Sep 10 15:22:45.771876 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.771599    2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8" container="manager"
Sep 10 15:22:45.772355 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.772102    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-88qk4\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:45.773876 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.773842    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:45.775015 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.774718    2573 scope.go:117] "RemoveContainer" containerID="83664fea6093ec8053b7f1b9fc8a168cd50745184d70b9259a68ff8f864600c2"
Sep 10 15:22:45.775015 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.775133    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83664fea6093ec8053b7f1b9fc8a168cd50745184d70b9259a68ff8f864600c2\": container with ID starting with 83664fea6093ec8053b7f1b9fc8a168cd50745184d70b9259a68ff8f864600c2 not found: ID does not exist" containerID="83664fea6093ec8053b7f1b9fc8a168cd50745184d70b9259a68ff8f864600c2"
Sep 10 15:22:45.775015 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.775165    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83664fea6093ec8053b7f1b9fc8a168cd50745184d70b9259a68ff8f864600c2"} err="failed to get container status \"83664fea6093ec8053b7f1b9fc8a168cd50745184d70b9259a68ff8f864600c2\": rpc error: code = NotFound desc = could not find container \"83664fea6093ec8053b7f1b9fc8a168cd50745184d70b9259a68ff8f864600c2\": container with ID starting with 83664fea6093ec8053b7f1b9fc8a168cd50745184d70b9259a68ff8f864600c2 not found: ID does not exist"
Sep 10 15:22:45.775015 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.775213    2573 scope.go:117] "RemoveContainer" containerID="6a946b8d3dd8e5624df5417573ec91604afd186572a9ad3ee75fb29c37d0dda3"
Sep 10 15:22:45.776340 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.775670    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-88qk4\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:45.778004 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.777537    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-c22l8\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="1ddfea82-5ca7-4580-9af7-769fe477a8a1" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:45.782056 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.782024    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:45.783994 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.783969    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-88qk4\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:45.785571 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.785551    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-c22l8\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="1ddfea82-5ca7-4580-9af7-769fe477a8a1" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:45.790372 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.789739    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-88qk4\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-88qk4"
Sep 10 15:22:45.790372 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.789860    2573 scope.go:117] "RemoveContainer" containerID="6a946b8d3dd8e5624df5417573ec91604afd186572a9ad3ee75fb29c37d0dda3"
Sep 10 15:22:45.796550 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.790938    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a946b8d3dd8e5624df5417573ec91604afd186572a9ad3ee75fb29c37d0dda3\": container with ID starting with 6a946b8d3dd8e5624df5417573ec91604afd186572a9ad3ee75fb29c37d0dda3 not found: ID does not exist" containerID="6a946b8d3dd8e5624df5417573ec91604afd186572a9ad3ee75fb29c37d0dda3"
Sep 10 15:22:45.796550 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:45.791336    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a946b8d3dd8e5624df5417573ec91604afd186572a9ad3ee75fb29c37d0dda3"} err="failed to get container status \"6a946b8d3dd8e5624df5417573ec91604afd186572a9ad3ee75fb29c37d0dda3\": rpc error: code = NotFound desc = could not find container \"6a946b8d3dd8e5624df5417573ec91604afd186572a9ad3ee75fb29c37d0dda3\": container with ID starting with 6a946b8d3dd8e5624df5417573ec91604afd186572a9ad3ee75fb29c37d0dda3 not found: ID does not exist"
Sep 10 15:22:45.796974 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.796750    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-c22l8\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="1ddfea82-5ca7-4580-9af7-769fe477a8a1" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:45.801538 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:45.801513    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xp88d\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xp88d"
Sep 10 15:22:46.125891 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:46.125851    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323" path="/var/lib/kubelet/pods/9f63dcd2-64eb-4a99-9d2a-9f0b2e0e8323/volumes"
Sep 10 15:22:46.126367 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:46.126319    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ce9546-427a-4f9f-8800-3b2cefbc471d" path="/var/lib/kubelet/pods/b0ce9546-427a-4f9f-8800-3b2cefbc471d/volumes"
Sep 10 15:22:46.781004 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:46.780962    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-62jls" event={"ID":"87213ea5-dbc7-46f0-abfc-613d537ad421","Type":"ContainerStarted","Data":"7f895bd1712867af6d776cc28bdcac8c7723e8a3f5077e55e288ba13acfaaa8c"}
Sep 10 15:22:46.781212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:46.781087    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-62jls"
Sep 10 15:22:46.782957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:46.782913    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8" podUID="1ddfea82-5ca7-4580-9af7-769fe477a8a1" containerName="manager" containerID="cri-o://f88b5d60295b5422278e7f1ba7dae4df4fa92ce66d8980541c0bd6a8df48e963" gracePeriod=2
Sep 10 15:22:46.784710 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:46.784683    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw" event={"ID":"e83ef995-aa99-462e-9b20-7bec95464032","Type":"ContainerStarted","Data":"cc77e419274686616fbf8bf49c4fcb20a644fdcbc447a4b2069ccb67d9bfa9c8"}
Sep 10 15:22:46.784835 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:46.784735    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"
Sep 10 15:22:46.786844 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:46.786760    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6" event={"ID":"8e3f3c72-bc7e-40bd-9719-8358842cec03","Type":"ContainerStarted","Data":"bcf63e8361dbd2b60566f046154ff2cf3ee2c76eafeb7ca9550c29bd1ba1fedd"}
Sep 10 15:22:46.786975 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:46.786870    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"
Sep 10 15:22:46.871599 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:46.871519    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-62jls" podStartSLOduration=1.871496461 podStartE2EDuration="1.871496461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:22:46.870132248 +0000 UTC m=+429.285928069" watchObservedRunningTime="2026-09-10 15:22:46.871496461 +0000 UTC m=+429.287292281"
Sep 10 15:22:46.946973 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:46.946918    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6" podStartSLOduration=2.946902037 podStartE2EDuration="2.946902037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:22:46.944876704 +0000 UTC m=+429.360672523" watchObservedRunningTime="2026-09-10 15:22:46.946902037 +0000 UTC m=+429.362697856"
Sep 10 15:22:46.982327 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:46.982279    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw" podStartSLOduration=2.982261178 podStartE2EDuration="2.982261178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:22:46.980594953 +0000 UTC m=+429.396390772" watchObservedRunningTime="2026-09-10 15:22:46.982261178 +0000 UTC m=+429.398056996"
Sep 10 15:22:47.028252 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:47.028228    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:47.109428 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:47.109397    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ddfea82-5ca7-4580-9af7-769fe477a8a1-extensions-socket-volume\") pod \"1ddfea82-5ca7-4580-9af7-769fe477a8a1\" (UID: \"1ddfea82-5ca7-4580-9af7-769fe477a8a1\") "
Sep 10 15:22:47.109603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:47.109480    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjm24\" (UniqueName: \"kubernetes.io/projected/1ddfea82-5ca7-4580-9af7-769fe477a8a1-kube-api-access-zjm24\") pod \"1ddfea82-5ca7-4580-9af7-769fe477a8a1\" (UID: \"1ddfea82-5ca7-4580-9af7-769fe477a8a1\") "
Sep 10 15:22:47.109825 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:47.109780    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddfea82-5ca7-4580-9af7-769fe477a8a1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1ddfea82-5ca7-4580-9af7-769fe477a8a1" (UID: "1ddfea82-5ca7-4580-9af7-769fe477a8a1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:22:47.111668 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:47.111647    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddfea82-5ca7-4580-9af7-769fe477a8a1-kube-api-access-zjm24" (OuterVolumeSpecName: "kube-api-access-zjm24") pod "1ddfea82-5ca7-4580-9af7-769fe477a8a1" (UID: "1ddfea82-5ca7-4580-9af7-769fe477a8a1"). InnerVolumeSpecName "kube-api-access-zjm24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:22:47.210030 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:47.209996    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ddfea82-5ca7-4580-9af7-769fe477a8a1-extensions-socket-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:22:47.210030 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:47.210026    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjm24\" (UniqueName: \"kubernetes.io/projected/1ddfea82-5ca7-4580-9af7-769fe477a8a1-kube-api-access-zjm24\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:22:47.794129 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:47.794088    2573 generic.go:356] "Generic (PLEG): container finished" podID="1ddfea82-5ca7-4580-9af7-769fe477a8a1" containerID="f88b5d60295b5422278e7f1ba7dae4df4fa92ce66d8980541c0bd6a8df48e963" exitCode=0
Sep 10 15:22:47.794292 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:47.794135    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:47.794292 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:47.794186    2573 scope.go:117] "RemoveContainer" containerID="f88b5d60295b5422278e7f1ba7dae4df4fa92ce66d8980541c0bd6a8df48e963"
Sep 10 15:22:47.796692 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:47.796662    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-c22l8\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="1ddfea82-5ca7-4580-9af7-769fe477a8a1" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:47.802668 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:47.802653    2573 scope.go:117] "RemoveContainer" containerID="f88b5d60295b5422278e7f1ba7dae4df4fa92ce66d8980541c0bd6a8df48e963"
Sep 10 15:22:47.802957 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:47.802934    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88b5d60295b5422278e7f1ba7dae4df4fa92ce66d8980541c0bd6a8df48e963\": container with ID starting with f88b5d60295b5422278e7f1ba7dae4df4fa92ce66d8980541c0bd6a8df48e963 not found: ID does not exist" containerID="f88b5d60295b5422278e7f1ba7dae4df4fa92ce66d8980541c0bd6a8df48e963"
Sep 10 15:22:47.803009 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:47.802966    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88b5d60295b5422278e7f1ba7dae4df4fa92ce66d8980541c0bd6a8df48e963"} err="failed to get container status \"f88b5d60295b5422278e7f1ba7dae4df4fa92ce66d8980541c0bd6a8df48e963\": rpc error: code = NotFound desc = could not find container \"f88b5d60295b5422278e7f1ba7dae4df4fa92ce66d8980541c0bd6a8df48e963\": container with ID starting with f88b5d60295b5422278e7f1ba7dae4df4fa92ce66d8980541c0bd6a8df48e963 not found: ID does not exist"
Sep 10 15:22:47.803935 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:47.803907    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-c22l8\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="1ddfea82-5ca7-4580-9af7-769fe477a8a1" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:48.123514 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:48.123475    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-c22l8\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="1ddfea82-5ca7-4580-9af7-769fe477a8a1" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-c22l8"
Sep 10 15:22:48.125454 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:48.125422    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ddfea82-5ca7-4580-9af7-769fe477a8a1" path="/var/lib/kubelet/pods/1ddfea82-5ca7-4580-9af7-769fe477a8a1/volumes"
Sep 10 15:22:57.796722 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:57.796690    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"
Sep 10 15:22:57.797343 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:57.796923    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-62jls"
Sep 10 15:22:57.797343 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:57.796999    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"
Sep 10 15:22:57.888221 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:57.888191    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"]
Sep 10 15:22:57.888404 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:57.888381    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw" podUID="e83ef995-aa99-462e-9b20-7bec95464032" containerName="manager" containerID="cri-o://cc77e419274686616fbf8bf49c4fcb20a644fdcbc447a4b2069ccb67d9bfa9c8" gracePeriod=10
Sep 10 15:22:58.139445 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.139421    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"
Sep 10 15:22:58.194319 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.194289    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"]
Sep 10 15:22:58.194643 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.194627    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83ef995-aa99-462e-9b20-7bec95464032" containerName="manager"
Sep 10 15:22:58.194723 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.194647    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83ef995-aa99-462e-9b20-7bec95464032" containerName="manager"
Sep 10 15:22:58.194723 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.194718    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83ef995-aa99-462e-9b20-7bec95464032" containerName="manager"
Sep 10 15:22:58.196542 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.196521    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"
Sep 10 15:22:58.206703 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.206682    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e83ef995-aa99-462e-9b20-7bec95464032-extensions-socket-volume\") pod \"e83ef995-aa99-462e-9b20-7bec95464032\" (UID: \"e83ef995-aa99-462e-9b20-7bec95464032\") "
Sep 10 15:22:58.206839 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.206751    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll4kr\" (UniqueName: \"kubernetes.io/projected/e83ef995-aa99-462e-9b20-7bec95464032-kube-api-access-ll4kr\") pod \"e83ef995-aa99-462e-9b20-7bec95464032\" (UID: \"e83ef995-aa99-462e-9b20-7bec95464032\") "
Sep 10 15:22:58.207088 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.207064    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83ef995-aa99-462e-9b20-7bec95464032-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e83ef995-aa99-462e-9b20-7bec95464032" (UID: "e83ef995-aa99-462e-9b20-7bec95464032"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:22:58.208977 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.208954    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83ef995-aa99-462e-9b20-7bec95464032-kube-api-access-ll4kr" (OuterVolumeSpecName: "kube-api-access-ll4kr") pod "e83ef995-aa99-462e-9b20-7bec95464032" (UID: "e83ef995-aa99-462e-9b20-7bec95464032"). InnerVolumeSpecName "kube-api-access-ll4kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:22:58.217740 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.217716    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"]
Sep 10 15:22:58.307434 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.307344    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/661edcdb-1c8f-42d6-a05a-fcf274343e9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bqdfm\" (UID: \"661edcdb-1c8f-42d6-a05a-fcf274343e9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"
Sep 10 15:22:58.307434 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.307408    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zssnv\" (UniqueName: \"kubernetes.io/projected/661edcdb-1c8f-42d6-a05a-fcf274343e9e-kube-api-access-zssnv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bqdfm\" (UID: \"661edcdb-1c8f-42d6-a05a-fcf274343e9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"
Sep 10 15:22:58.307621 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.307445    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e83ef995-aa99-462e-9b20-7bec95464032-extensions-socket-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:22:58.307621 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.307458    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll4kr\" (UniqueName: \"kubernetes.io/projected/e83ef995-aa99-462e-9b20-7bec95464032-kube-api-access-ll4kr\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:22:58.408308 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.408266    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zssnv\" (UniqueName: \"kubernetes.io/projected/661edcdb-1c8f-42d6-a05a-fcf274343e9e-kube-api-access-zssnv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bqdfm\" (UID: \"661edcdb-1c8f-42d6-a05a-fcf274343e9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"
Sep 10 15:22:58.408488 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.408351    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/661edcdb-1c8f-42d6-a05a-fcf274343e9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bqdfm\" (UID: \"661edcdb-1c8f-42d6-a05a-fcf274343e9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"
Sep 10 15:22:58.408723 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.408701    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/661edcdb-1c8f-42d6-a05a-fcf274343e9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bqdfm\" (UID: \"661edcdb-1c8f-42d6-a05a-fcf274343e9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"
Sep 10 15:22:58.420333 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.420309    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zssnv\" (UniqueName: \"kubernetes.io/projected/661edcdb-1c8f-42d6-a05a-fcf274343e9e-kube-api-access-zssnv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bqdfm\" (UID: \"661edcdb-1c8f-42d6-a05a-fcf274343e9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"
Sep 10 15:22:58.506692 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.506643    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"
Sep 10 15:22:58.782298 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.782141    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"]
Sep 10 15:22:58.785108 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:22:58.785076    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661edcdb_1c8f_42d6_a05a_fcf274343e9e.slice/crio-131f1399da7f8d49bb3c58e7e2a9f96dfe264a7c5ccd5b6af2c82a1b52f70587 WatchSource:0}: Error finding container 131f1399da7f8d49bb3c58e7e2a9f96dfe264a7c5ccd5b6af2c82a1b52f70587: Status 404 returned error can't find the container with id 131f1399da7f8d49bb3c58e7e2a9f96dfe264a7c5ccd5b6af2c82a1b52f70587
Sep 10 15:22:58.849706 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.849677    2573 generic.go:356] "Generic (PLEG): container finished" podID="e83ef995-aa99-462e-9b20-7bec95464032" containerID="cc77e419274686616fbf8bf49c4fcb20a644fdcbc447a4b2069ccb67d9bfa9c8" exitCode=0
Sep 10 15:22:58.850081 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.849723    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"
Sep 10 15:22:58.850081 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.849755    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw" event={"ID":"e83ef995-aa99-462e-9b20-7bec95464032","Type":"ContainerDied","Data":"cc77e419274686616fbf8bf49c4fcb20a644fdcbc447a4b2069ccb67d9bfa9c8"}
Sep 10 15:22:58.850081 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.849837    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw" event={"ID":"e83ef995-aa99-462e-9b20-7bec95464032","Type":"ContainerDied","Data":"22e0a530f4f141259fb15a17799ad79aee696d9186d8f66c09dc9b3768eacc21"}
Sep 10 15:22:58.850081 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.849867    2573 scope.go:117] "RemoveContainer" containerID="cc77e419274686616fbf8bf49c4fcb20a644fdcbc447a4b2069ccb67d9bfa9c8"
Sep 10 15:22:58.851562 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.851535    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm" event={"ID":"661edcdb-1c8f-42d6-a05a-fcf274343e9e","Type":"ContainerStarted","Data":"4e982c643f405ac78a3eaf136639878d2949884f22dfe4679c672b6a1f142e93"}
Sep 10 15:22:58.851650 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.851578    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm" event={"ID":"661edcdb-1c8f-42d6-a05a-fcf274343e9e","Type":"ContainerStarted","Data":"131f1399da7f8d49bb3c58e7e2a9f96dfe264a7c5ccd5b6af2c82a1b52f70587"}
Sep 10 15:22:58.851691 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.851680    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"
Sep 10 15:22:58.858534 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.858514    2573 scope.go:117] "RemoveContainer" containerID="cc77e419274686616fbf8bf49c4fcb20a644fdcbc447a4b2069ccb67d9bfa9c8"
Sep 10 15:22:58.858832 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:22:58.858813    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc77e419274686616fbf8bf49c4fcb20a644fdcbc447a4b2069ccb67d9bfa9c8\": container with ID starting with cc77e419274686616fbf8bf49c4fcb20a644fdcbc447a4b2069ccb67d9bfa9c8 not found: ID does not exist" containerID="cc77e419274686616fbf8bf49c4fcb20a644fdcbc447a4b2069ccb67d9bfa9c8"
Sep 10 15:22:58.858906 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.858843    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc77e419274686616fbf8bf49c4fcb20a644fdcbc447a4b2069ccb67d9bfa9c8"} err="failed to get container status \"cc77e419274686616fbf8bf49c4fcb20a644fdcbc447a4b2069ccb67d9bfa9c8\": rpc error: code = NotFound desc = could not find container \"cc77e419274686616fbf8bf49c4fcb20a644fdcbc447a4b2069ccb67d9bfa9c8\": container with ID starting with cc77e419274686616fbf8bf49c4fcb20a644fdcbc447a4b2069ccb67d9bfa9c8 not found: ID does not exist"
Sep 10 15:22:58.882493 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.882447    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm" podStartSLOduration=0.882432328 podStartE2EDuration="882.432328ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:22:58.88122181 +0000 UTC m=+441.297017631" watchObservedRunningTime="2026-09-10 15:22:58.882432328 +0000 UTC m=+441.298228147"
Sep 10 15:22:58.907829 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.903545    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"]
Sep 10 15:22:58.907829 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:22:58.907100    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k6gzw"]
Sep 10 15:23:00.125013 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:00.124972    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e83ef995-aa99-462e-9b20-7bec95464032" path="/var/lib/kubelet/pods/e83ef995-aa99-462e-9b20-7bec95464032/volumes"
Sep 10 15:23:09.859920 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:09.859890    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"
Sep 10 15:23:09.913055 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:09.913026    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"]
Sep 10 15:23:09.913261 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:09.913221    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6" podUID="8e3f3c72-bc7e-40bd-9719-8358842cec03" containerName="manager" containerID="cri-o://bcf63e8361dbd2b60566f046154ff2cf3ee2c76eafeb7ca9550c29bd1ba1fedd" gracePeriod=10
Sep 10 15:23:10.157834 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.157806    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"
Sep 10 15:23:10.305940 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.305902    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8e3f3c72-bc7e-40bd-9719-8358842cec03-extensions-socket-volume\") pod \"8e3f3c72-bc7e-40bd-9719-8358842cec03\" (UID: \"8e3f3c72-bc7e-40bd-9719-8358842cec03\") "
Sep 10 15:23:10.306115 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.306011    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k95bw\" (UniqueName: \"kubernetes.io/projected/8e3f3c72-bc7e-40bd-9719-8358842cec03-kube-api-access-k95bw\") pod \"8e3f3c72-bc7e-40bd-9719-8358842cec03\" (UID: \"8e3f3c72-bc7e-40bd-9719-8358842cec03\") "
Sep 10 15:23:10.306340 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.306319    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3f3c72-bc7e-40bd-9719-8358842cec03-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8e3f3c72-bc7e-40bd-9719-8358842cec03" (UID: "8e3f3c72-bc7e-40bd-9719-8358842cec03"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:23:10.308174 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.308147    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3f3c72-bc7e-40bd-9719-8358842cec03-kube-api-access-k95bw" (OuterVolumeSpecName: "kube-api-access-k95bw") pod "8e3f3c72-bc7e-40bd-9719-8358842cec03" (UID: "8e3f3c72-bc7e-40bd-9719-8358842cec03"). InnerVolumeSpecName "kube-api-access-k95bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:23:10.406959 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.406878    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8e3f3c72-bc7e-40bd-9719-8358842cec03-extensions-socket-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:23:10.406959 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.406907    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k95bw\" (UniqueName: \"kubernetes.io/projected/8e3f3c72-bc7e-40bd-9719-8358842cec03-kube-api-access-k95bw\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:23:10.909016 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.908977    2573 generic.go:356] "Generic (PLEG): container finished" podID="8e3f3c72-bc7e-40bd-9719-8358842cec03" containerID="bcf63e8361dbd2b60566f046154ff2cf3ee2c76eafeb7ca9550c29bd1ba1fedd" exitCode=0
Sep 10 15:23:10.909434 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.909024    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"
Sep 10 15:23:10.909434 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.909040    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6" event={"ID":"8e3f3c72-bc7e-40bd-9719-8358842cec03","Type":"ContainerDied","Data":"bcf63e8361dbd2b60566f046154ff2cf3ee2c76eafeb7ca9550c29bd1ba1fedd"}
Sep 10 15:23:10.909434 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.909090    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6" event={"ID":"8e3f3c72-bc7e-40bd-9719-8358842cec03","Type":"ContainerDied","Data":"97d7c17cd78018c6440cf865cbf132e2fd56d310c1358c6edb0b0e2a2890344a"}
Sep 10 15:23:10.909434 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.909120    2573 scope.go:117] "RemoveContainer" containerID="bcf63e8361dbd2b60566f046154ff2cf3ee2c76eafeb7ca9550c29bd1ba1fedd"
Sep 10 15:23:10.917846 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.917650    2573 scope.go:117] "RemoveContainer" containerID="bcf63e8361dbd2b60566f046154ff2cf3ee2c76eafeb7ca9550c29bd1ba1fedd"
Sep 10 15:23:10.917995 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:23:10.917976    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf63e8361dbd2b60566f046154ff2cf3ee2c76eafeb7ca9550c29bd1ba1fedd\": container with ID starting with bcf63e8361dbd2b60566f046154ff2cf3ee2c76eafeb7ca9550c29bd1ba1fedd not found: ID does not exist" containerID="bcf63e8361dbd2b60566f046154ff2cf3ee2c76eafeb7ca9550c29bd1ba1fedd"
Sep 10 15:23:10.918048 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.918005    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf63e8361dbd2b60566f046154ff2cf3ee2c76eafeb7ca9550c29bd1ba1fedd"} err="failed to get container status \"bcf63e8361dbd2b60566f046154ff2cf3ee2c76eafeb7ca9550c29bd1ba1fedd\": rpc error: code = NotFound desc = could not find container \"bcf63e8361dbd2b60566f046154ff2cf3ee2c76eafeb7ca9550c29bd1ba1fedd\": container with ID starting with bcf63e8361dbd2b60566f046154ff2cf3ee2c76eafeb7ca9550c29bd1ba1fedd not found: ID does not exist"
Sep 10 15:23:10.930049 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.930028    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"]
Sep 10 15:23:10.936669 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:10.936647    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-jzqb6"]
Sep 10 15:23:12.125317 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:12.125282    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e3f3c72-bc7e-40bd-9719-8358842cec03" path="/var/lib/kubelet/pods/8e3f3c72-bc7e-40bd-9719-8358842cec03/volumes"
Sep 10 15:23:30.238880 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.238833    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-th89s"]
Sep 10 15:23:30.239441 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.239421    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e3f3c72-bc7e-40bd-9719-8358842cec03" containerName="manager"
Sep 10 15:23:30.239510 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.239445    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e3f3c72-bc7e-40bd-9719-8358842cec03" containerName="manager"
Sep 10 15:23:30.239602 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.239590    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e3f3c72-bc7e-40bd-9719-8358842cec03" containerName="manager"
Sep 10 15:23:30.241969 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.241945    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-th89s"
Sep 10 15:23:30.244625 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.244604    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 15:23:30.244734 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.244677    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h5hpc\""
Sep 10 15:23:30.249871 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.249850    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-th89s"]
Sep 10 15:23:30.259826 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.259802    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4kdj\" (UniqueName: \"kubernetes.io/projected/30d92134-2b8d-4983-a14e-00a4f7bea1cd-kube-api-access-z4kdj\") pod \"limitador-limitador-8588749977-th89s\" (UID: \"30d92134-2b8d-4983-a14e-00a4f7bea1cd\") " pod="kuadrant-system/limitador-limitador-8588749977-th89s"
Sep 10 15:23:30.259925 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.259858    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/30d92134-2b8d-4983-a14e-00a4f7bea1cd-config-file\") pod \"limitador-limitador-8588749977-th89s\" (UID: \"30d92134-2b8d-4983-a14e-00a4f7bea1cd\") " pod="kuadrant-system/limitador-limitador-8588749977-th89s"
Sep 10 15:23:30.329055 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.329024    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-th89s"]
Sep 10 15:23:30.360783 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.360754    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kdj\" (UniqueName: \"kubernetes.io/projected/30d92134-2b8d-4983-a14e-00a4f7bea1cd-kube-api-access-z4kdj\") pod \"limitador-limitador-8588749977-th89s\" (UID: \"30d92134-2b8d-4983-a14e-00a4f7bea1cd\") " pod="kuadrant-system/limitador-limitador-8588749977-th89s"
Sep 10 15:23:30.360947 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.360811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/30d92134-2b8d-4983-a14e-00a4f7bea1cd-config-file\") pod \"limitador-limitador-8588749977-th89s\" (UID: \"30d92134-2b8d-4983-a14e-00a4f7bea1cd\") " pod="kuadrant-system/limitador-limitador-8588749977-th89s"
Sep 10 15:23:30.361355 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.361338    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/30d92134-2b8d-4983-a14e-00a4f7bea1cd-config-file\") pod \"limitador-limitador-8588749977-th89s\" (UID: \"30d92134-2b8d-4983-a14e-00a4f7bea1cd\") " pod="kuadrant-system/limitador-limitador-8588749977-th89s"
Sep 10 15:23:30.369004 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.368978    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4kdj\" (UniqueName: \"kubernetes.io/projected/30d92134-2b8d-4983-a14e-00a4f7bea1cd-kube-api-access-z4kdj\") pod \"limitador-limitador-8588749977-th89s\" (UID: \"30d92134-2b8d-4983-a14e-00a4f7bea1cd\") " pod="kuadrant-system/limitador-limitador-8588749977-th89s"
Sep 10 15:23:30.553631 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.553543    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-th89s"
Sep 10 15:23:30.826195 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:30.826164    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-th89s"]
Sep 10 15:23:30.829388 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:23:30.829364    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d92134_2b8d_4983_a14e_00a4f7bea1cd.slice/crio-7ecdc520204aea7bee68ad7bbe11028123c8f19c8a9bcb0ebaf835dca712128c WatchSource:0}: Error finding container 7ecdc520204aea7bee68ad7bbe11028123c8f19c8a9bcb0ebaf835dca712128c: Status 404 returned error can't find the container with id 7ecdc520204aea7bee68ad7bbe11028123c8f19c8a9bcb0ebaf835dca712128c
Sep 10 15:23:31.000194 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.000162    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-th89s" event={"ID":"30d92134-2b8d-4983-a14e-00a4f7bea1cd","Type":"ContainerStarted","Data":"7ecdc520204aea7bee68ad7bbe11028123c8f19c8a9bcb0ebaf835dca712128c"}
Sep 10 15:23:31.057171 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.057123    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4xv59"]
Sep 10 15:23:31.060361 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.060333    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4xv59"
Sep 10 15:23:31.063152 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.063130    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ntf8k\""
Sep 10 15:23:31.068429 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.068401    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4xv59"]
Sep 10 15:23:31.167518 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.167438    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smdg\" (UniqueName: \"kubernetes.io/projected/02b9c655-0c44-42d1-876d-e9a992985f37-kube-api-access-9smdg\") pod \"authorino-7bcfd8f9cd-4xv59\" (UID: \"02b9c655-0c44-42d1-876d-e9a992985f37\") " pod="kuadrant-system/authorino-7bcfd8f9cd-4xv59"
Sep 10 15:23:31.172292 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.172261    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-qcl74"]
Sep 10 15:23:31.174535 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.174511    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qcl74"
Sep 10 15:23:31.189391 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.189365    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qcl74"]
Sep 10 15:23:31.268299 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.268261    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc2l5\" (UniqueName: \"kubernetes.io/projected/ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37-kube-api-access-qc2l5\") pod \"authorino-59c974bd48-qcl74\" (UID: \"ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37\") " pod="kuadrant-system/authorino-59c974bd48-qcl74"
Sep 10 15:23:31.268723 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.268317    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smdg\" (UniqueName: \"kubernetes.io/projected/02b9c655-0c44-42d1-876d-e9a992985f37-kube-api-access-9smdg\") pod \"authorino-7bcfd8f9cd-4xv59\" (UID: \"02b9c655-0c44-42d1-876d-e9a992985f37\") " pod="kuadrant-system/authorino-7bcfd8f9cd-4xv59"
Sep 10 15:23:31.278155 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.278125    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smdg\" (UniqueName: \"kubernetes.io/projected/02b9c655-0c44-42d1-876d-e9a992985f37-kube-api-access-9smdg\") pod \"authorino-7bcfd8f9cd-4xv59\" (UID: \"02b9c655-0c44-42d1-876d-e9a992985f37\") " pod="kuadrant-system/authorino-7bcfd8f9cd-4xv59"
Sep 10 15:23:31.369808 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.369734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc2l5\" (UniqueName: \"kubernetes.io/projected/ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37-kube-api-access-qc2l5\") pod \"authorino-59c974bd48-qcl74\" (UID: \"ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37\") " pod="kuadrant-system/authorino-59c974bd48-qcl74"
Sep 10 15:23:31.370624 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.370597    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4xv59"
Sep 10 15:23:31.379470 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.379447    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc2l5\" (UniqueName: \"kubernetes.io/projected/ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37-kube-api-access-qc2l5\") pod \"authorino-59c974bd48-qcl74\" (UID: \"ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37\") " pod="kuadrant-system/authorino-59c974bd48-qcl74"
Sep 10 15:23:31.483710 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.483676    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qcl74"
Sep 10 15:23:31.641837 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.641802    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4xv59"]
Sep 10 15:23:31.644484 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:23:31.644453    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02b9c655_0c44_42d1_876d_e9a992985f37.slice/crio-35f5fbd4e1d44c81bd45016b373063b4441e9ac2228c79fa51cb29f011be270f WatchSource:0}: Error finding container 35f5fbd4e1d44c81bd45016b373063b4441e9ac2228c79fa51cb29f011be270f: Status 404 returned error can't find the container with id 35f5fbd4e1d44c81bd45016b373063b4441e9ac2228c79fa51cb29f011be270f
Sep 10 15:23:31.754371 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:31.754289    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qcl74"]
Sep 10 15:23:31.757970 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:23:31.757944    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecbb71d0_b5f8_4d4f_ae3e_9946f95d3f37.slice/crio-c6feaf7d511d376b7d9a5b452b4a59b4730db06d44c96bd4b94a16868713fba2 WatchSource:0}: Error finding container c6feaf7d511d376b7d9a5b452b4a59b4730db06d44c96bd4b94a16868713fba2: Status 404 returned error can't find the container with id c6feaf7d511d376b7d9a5b452b4a59b4730db06d44c96bd4b94a16868713fba2
Sep 10 15:23:32.005446 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:32.005353    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4xv59" event={"ID":"02b9c655-0c44-42d1-876d-e9a992985f37","Type":"ContainerStarted","Data":"35f5fbd4e1d44c81bd45016b373063b4441e9ac2228c79fa51cb29f011be270f"}
Sep 10 15:23:32.006443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:32.006420    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qcl74" event={"ID":"ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37","Type":"ContainerStarted","Data":"c6feaf7d511d376b7d9a5b452b4a59b4730db06d44c96bd4b94a16868713fba2"}
Sep 10 15:23:33.015872 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:33.015834    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-th89s" event={"ID":"30d92134-2b8d-4983-a14e-00a4f7bea1cd","Type":"ContainerStarted","Data":"4095c0677c9f55fce4d6536e58f776b49751289660e1df60cbd6bc0d7fe900bd"}
Sep 10 15:23:33.016241 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:33.015902    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-th89s"
Sep 10 15:23:33.036663 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:33.036609    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-th89s" podStartSLOduration=1.499125431 podStartE2EDuration="3.03659375s" totalImagesPullingTime="1.537468319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:23:33.034906721 +0000 UTC m=+475.450702539" watchObservedRunningTime="2026-09-10 15:23:33.03659375 +0000 UTC m=+475.452389611"
Sep 10 15:23:37.037657 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:37.037564    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qcl74" event={"ID":"ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37","Type":"ContainerStarted","Data":"4ea2d2df0725b39bbfe48f30385e30ca10f42195070e9d719bd72def0f37513f"}
Sep 10 15:23:37.055766 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:37.055709    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-qcl74" podStartSLOduration=1.074175949 podStartE2EDuration="6.055693376s" totalImagesPullingTime="4.981517427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:23:37.054872246 +0000 UTC m=+479.470668060" watchObservedRunningTime="2026-09-10 15:23:37.055693376 +0000 UTC m=+479.471489189"
Sep 10 15:23:37.085203 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:37.085168    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4xv59"]
Sep 10 15:23:44.022102 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:44.022071    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-th89s"
Sep 10 15:23:44.068514 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:44.068477    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4xv59" event={"ID":"02b9c655-0c44-42d1-876d-e9a992985f37","Type":"ContainerStarted","Data":"eb33ee301db61c8643c298d1f0251e5f38f93edee4735fb054caacd745a145ac"}
Sep 10 15:23:44.068690 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:44.068564    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-4xv59" podUID="02b9c655-0c44-42d1-876d-e9a992985f37" containerName="authorino" containerID="cri-o://eb33ee301db61c8643c298d1f0251e5f38f93edee4735fb054caacd745a145ac" gracePeriod=30
Sep 10 15:23:44.083099 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:44.083048    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-4xv59" podStartSLOduration=1.203802753 podStartE2EDuration="13.083028909s" totalImagesPullingTime="11.879226156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:23:44.082777005 +0000 UTC m=+486.498572836" watchObservedRunningTime="2026-09-10 15:23:44.083028909 +0000 UTC m=+486.498824739"
Sep 10 15:23:44.308700 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:44.308675    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4xv59"
Sep 10 15:23:44.371491 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:44.371446    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9smdg\" (UniqueName: \"kubernetes.io/projected/02b9c655-0c44-42d1-876d-e9a992985f37-kube-api-access-9smdg\") pod \"02b9c655-0c44-42d1-876d-e9a992985f37\" (UID: \"02b9c655-0c44-42d1-876d-e9a992985f37\") "
Sep 10 15:23:44.373625 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:44.373595    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b9c655-0c44-42d1-876d-e9a992985f37-kube-api-access-9smdg" (OuterVolumeSpecName: "kube-api-access-9smdg") pod "02b9c655-0c44-42d1-876d-e9a992985f37" (UID: "02b9c655-0c44-42d1-876d-e9a992985f37"). InnerVolumeSpecName "kube-api-access-9smdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:23:44.472480 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:44.472443    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9smdg\" (UniqueName: \"kubernetes.io/projected/02b9c655-0c44-42d1-876d-e9a992985f37-kube-api-access-9smdg\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:23:45.074440 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:45.074409    2573 generic.go:356] "Generic (PLEG): container finished" podID="02b9c655-0c44-42d1-876d-e9a992985f37" containerID="eb33ee301db61c8643c298d1f0251e5f38f93edee4735fb054caacd745a145ac" exitCode=0
Sep 10 15:23:45.074941 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:45.074467    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4xv59" event={"ID":"02b9c655-0c44-42d1-876d-e9a992985f37","Type":"ContainerDied","Data":"eb33ee301db61c8643c298d1f0251e5f38f93edee4735fb054caacd745a145ac"}
Sep 10 15:23:45.074941 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:45.074478    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4xv59"
Sep 10 15:23:45.074941 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:45.074498    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4xv59" event={"ID":"02b9c655-0c44-42d1-876d-e9a992985f37","Type":"ContainerDied","Data":"35f5fbd4e1d44c81bd45016b373063b4441e9ac2228c79fa51cb29f011be270f"}
Sep 10 15:23:45.074941 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:45.074515    2573 scope.go:117] "RemoveContainer" containerID="eb33ee301db61c8643c298d1f0251e5f38f93edee4735fb054caacd745a145ac"
Sep 10 15:23:45.083006 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:45.082986    2573 scope.go:117] "RemoveContainer" containerID="eb33ee301db61c8643c298d1f0251e5f38f93edee4735fb054caacd745a145ac"
Sep 10 15:23:45.083295 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:23:45.083276    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb33ee301db61c8643c298d1f0251e5f38f93edee4735fb054caacd745a145ac\": container with ID starting with eb33ee301db61c8643c298d1f0251e5f38f93edee4735fb054caacd745a145ac not found: ID does not exist" containerID="eb33ee301db61c8643c298d1f0251e5f38f93edee4735fb054caacd745a145ac"
Sep 10 15:23:45.083340 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:45.083304    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb33ee301db61c8643c298d1f0251e5f38f93edee4735fb054caacd745a145ac"} err="failed to get container status \"eb33ee301db61c8643c298d1f0251e5f38f93edee4735fb054caacd745a145ac\": rpc error: code = NotFound desc = could not find container \"eb33ee301db61c8643c298d1f0251e5f38f93edee4735fb054caacd745a145ac\": container with ID starting with eb33ee301db61c8643c298d1f0251e5f38f93edee4735fb054caacd745a145ac not found: ID does not exist"
Sep 10 15:23:45.094262 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:45.094241    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4xv59"]
Sep 10 15:23:45.097475 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:45.097455    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4xv59"]
Sep 10 15:23:46.125667 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:46.125634    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02b9c655-0c44-42d1-876d-e9a992985f37" path="/var/lib/kubelet/pods/02b9c655-0c44-42d1-876d-e9a992985f37/volumes"
Sep 10 15:23:49.177875 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:49.177835    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-th89s"]
Sep 10 15:23:49.178387 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:49.178023    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-th89s" podUID="30d92134-2b8d-4983-a14e-00a4f7bea1cd" containerName="limitador" containerID="cri-o://4095c0677c9f55fce4d6536e58f776b49751289660e1df60cbd6bc0d7fe900bd" gracePeriod=30
Sep 10 15:23:49.723488 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:49.723462    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-th89s"
Sep 10 15:23:49.812158 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:49.812068    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4kdj\" (UniqueName: \"kubernetes.io/projected/30d92134-2b8d-4983-a14e-00a4f7bea1cd-kube-api-access-z4kdj\") pod \"30d92134-2b8d-4983-a14e-00a4f7bea1cd\" (UID: \"30d92134-2b8d-4983-a14e-00a4f7bea1cd\") "
Sep 10 15:23:49.812158 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:49.812127    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/30d92134-2b8d-4983-a14e-00a4f7bea1cd-config-file\") pod \"30d92134-2b8d-4983-a14e-00a4f7bea1cd\" (UID: \"30d92134-2b8d-4983-a14e-00a4f7bea1cd\") "
Sep 10 15:23:49.812518 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:49.812492    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d92134-2b8d-4983-a14e-00a4f7bea1cd-config-file" (OuterVolumeSpecName: "config-file") pod "30d92134-2b8d-4983-a14e-00a4f7bea1cd" (UID: "30d92134-2b8d-4983-a14e-00a4f7bea1cd"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:23:49.814288 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:49.814264    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d92134-2b8d-4983-a14e-00a4f7bea1cd-kube-api-access-z4kdj" (OuterVolumeSpecName: "kube-api-access-z4kdj") pod "30d92134-2b8d-4983-a14e-00a4f7bea1cd" (UID: "30d92134-2b8d-4983-a14e-00a4f7bea1cd"). InnerVolumeSpecName "kube-api-access-z4kdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:23:49.913322 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:49.913286    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4kdj\" (UniqueName: \"kubernetes.io/projected/30d92134-2b8d-4983-a14e-00a4f7bea1cd-kube-api-access-z4kdj\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:23:49.913322 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:49.913316    2573 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/30d92134-2b8d-4983-a14e-00a4f7bea1cd-config-file\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:23:50.104344 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:50.104308    2573 generic.go:356] "Generic (PLEG): container finished" podID="30d92134-2b8d-4983-a14e-00a4f7bea1cd" containerID="4095c0677c9f55fce4d6536e58f776b49751289660e1df60cbd6bc0d7fe900bd" exitCode=0
Sep 10 15:23:50.104509 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:50.104355    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-th89s"
Sep 10 15:23:50.104509 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:50.104397    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-th89s" event={"ID":"30d92134-2b8d-4983-a14e-00a4f7bea1cd","Type":"ContainerDied","Data":"4095c0677c9f55fce4d6536e58f776b49751289660e1df60cbd6bc0d7fe900bd"}
Sep 10 15:23:50.104509 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:50.104447    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-th89s" event={"ID":"30d92134-2b8d-4983-a14e-00a4f7bea1cd","Type":"ContainerDied","Data":"7ecdc520204aea7bee68ad7bbe11028123c8f19c8a9bcb0ebaf835dca712128c"}
Sep 10 15:23:50.104509 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:50.104468    2573 scope.go:117] "RemoveContainer" containerID="4095c0677c9f55fce4d6536e58f776b49751289660e1df60cbd6bc0d7fe900bd"
Sep 10 15:23:50.113224 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:50.113206    2573 scope.go:117] "RemoveContainer" containerID="4095c0677c9f55fce4d6536e58f776b49751289660e1df60cbd6bc0d7fe900bd"
Sep 10 15:23:50.113506 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:23:50.113484    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4095c0677c9f55fce4d6536e58f776b49751289660e1df60cbd6bc0d7fe900bd\": container with ID starting with 4095c0677c9f55fce4d6536e58f776b49751289660e1df60cbd6bc0d7fe900bd not found: ID does not exist" containerID="4095c0677c9f55fce4d6536e58f776b49751289660e1df60cbd6bc0d7fe900bd"
Sep 10 15:23:50.113603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:50.113510    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4095c0677c9f55fce4d6536e58f776b49751289660e1df60cbd6bc0d7fe900bd"} err="failed to get container status \"4095c0677c9f55fce4d6536e58f776b49751289660e1df60cbd6bc0d7fe900bd\": rpc error: code = NotFound desc = could not find container \"4095c0677c9f55fce4d6536e58f776b49751289660e1df60cbd6bc0d7fe900bd\": container with ID starting with 4095c0677c9f55fce4d6536e58f776b49751289660e1df60cbd6bc0d7fe900bd not found: ID does not exist"
Sep 10 15:23:50.126891 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:50.125847    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-th89s"]
Sep 10 15:23:50.131429 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:50.131409    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-th89s"]
Sep 10 15:23:52.125578 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:52.125548    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30d92134-2b8d-4983-a14e-00a4f7bea1cd" path="/var/lib/kubelet/pods/30d92134-2b8d-4983-a14e-00a4f7bea1cd/volumes"
Sep 10 15:23:53.602986 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.602949    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-x8npx"]
Sep 10 15:23:53.603347 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.603246    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d92134-2b8d-4983-a14e-00a4f7bea1cd" containerName="limitador"
Sep 10 15:23:53.603347 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.603257    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d92134-2b8d-4983-a14e-00a4f7bea1cd" containerName="limitador"
Sep 10 15:23:53.603347 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.603271    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b9c655-0c44-42d1-876d-e9a992985f37" containerName="authorino"
Sep 10 15:23:53.603347 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.603277    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b9c655-0c44-42d1-876d-e9a992985f37" containerName="authorino"
Sep 10 15:23:53.603347 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.603333    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="30d92134-2b8d-4983-a14e-00a4f7bea1cd" containerName="limitador"
Sep 10 15:23:53.603347 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.603341    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b9c655-0c44-42d1-876d-e9a992985f37" containerName="authorino"
Sep 10 15:23:53.605831 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.605813    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x8npx"
Sep 10 15:23:53.609417 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.609389    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:23:53.609996 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.609978    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 15:23:53.610132 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.610033    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kl6t8\""
Sep 10 15:23:53.610132 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.610092    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 15:23:53.618645 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.618619    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-x8npx"]
Sep 10 15:23:53.641516 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.641492    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a314adf2-281b-42d4-8c30-d2d0e00d7e4f-data\") pod \"postgres-69fb4594cf-x8npx\" (UID: \"a314adf2-281b-42d4-8c30-d2d0e00d7e4f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-x8npx"
Sep 10 15:23:53.641644 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.641540    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndlsj\" (UniqueName: \"kubernetes.io/projected/a314adf2-281b-42d4-8c30-d2d0e00d7e4f-kube-api-access-ndlsj\") pod \"postgres-69fb4594cf-x8npx\" (UID: \"a314adf2-281b-42d4-8c30-d2d0e00d7e4f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-x8npx"
Sep 10 15:23:53.742314 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.742274    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a314adf2-281b-42d4-8c30-d2d0e00d7e4f-data\") pod \"postgres-69fb4594cf-x8npx\" (UID: \"a314adf2-281b-42d4-8c30-d2d0e00d7e4f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-x8npx"
Sep 10 15:23:53.742483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.742326    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndlsj\" (UniqueName: \"kubernetes.io/projected/a314adf2-281b-42d4-8c30-d2d0e00d7e4f-kube-api-access-ndlsj\") pod \"postgres-69fb4594cf-x8npx\" (UID: \"a314adf2-281b-42d4-8c30-d2d0e00d7e4f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-x8npx"
Sep 10 15:23:53.742652 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.742632    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a314adf2-281b-42d4-8c30-d2d0e00d7e4f-data\") pod \"postgres-69fb4594cf-x8npx\" (UID: \"a314adf2-281b-42d4-8c30-d2d0e00d7e4f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-x8npx"
Sep 10 15:23:53.750383 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.750352    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndlsj\" (UniqueName: \"kubernetes.io/projected/a314adf2-281b-42d4-8c30-d2d0e00d7e4f-kube-api-access-ndlsj\") pod \"postgres-69fb4594cf-x8npx\" (UID: \"a314adf2-281b-42d4-8c30-d2d0e00d7e4f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-x8npx"
Sep 10 15:23:53.916813 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:53.916695    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x8npx"
Sep 10 15:23:54.187346 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:54.187273    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-x8npx"]
Sep 10 15:23:54.190358 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:23:54.190331    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda314adf2_281b_42d4_8c30_d2d0e00d7e4f.slice/crio-1ea6f099d723bf267822a92f2ae02a2efbbd7398d2633d6b30cced64996e9006 WatchSource:0}: Error finding container 1ea6f099d723bf267822a92f2ae02a2efbbd7398d2633d6b30cced64996e9006: Status 404 returned error can't find the container with id 1ea6f099d723bf267822a92f2ae02a2efbbd7398d2633d6b30cced64996e9006
Sep 10 15:23:55.127074 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:23:55.127038    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x8npx" event={"ID":"a314adf2-281b-42d4-8c30-d2d0e00d7e4f","Type":"ContainerStarted","Data":"1ea6f099d723bf267822a92f2ae02a2efbbd7398d2633d6b30cced64996e9006"}
Sep 10 15:24:00.150020 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:00.149981    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x8npx" event={"ID":"a314adf2-281b-42d4-8c30-d2d0e00d7e4f","Type":"ContainerStarted","Data":"674ed9605ba4b86e1689cd2da71abe30c832e57a352a2da2b068ed2780e1d05d"}
Sep 10 15:24:00.150380 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:00.150133    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x8npx"
Sep 10 15:24:00.166085 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:00.166033    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x8npx" podStartSLOduration=2.120937534 podStartE2EDuration="7.16601466s" totalImagesPullingTime="5.045077126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:23:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:24:00.165452031 +0000 UTC m=+502.581247850" watchObservedRunningTime="2026-09-10 15:24:00.16601466 +0000 UTC m=+502.581810504"
Sep 10 15:24:06.183653 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.183623    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-x8npx"
Sep 10 15:24:06.703874 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.703840    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-nm9np"]
Sep 10 15:24:06.709884 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.709857    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nm9np"
Sep 10 15:24:06.712126 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.712101    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-nm9np"]
Sep 10 15:24:06.839585 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.839553    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-nm9np"]
Sep 10 15:24:06.839813 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:24:06.839773    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-45c55], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-nm9np" podUID="1d9541ba-e4a0-452f-9844-e921d6d413b6"
Sep 10 15:24:06.853224 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.853187    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45c55\" (UniqueName: \"kubernetes.io/projected/1d9541ba-e4a0-452f-9844-e921d6d413b6-kube-api-access-45c55\") pod \"authorino-557f876f69-nm9np\" (UID: \"1d9541ba-e4a0-452f-9844-e921d6d413b6\") " pod="kuadrant-system/authorino-557f876f69-nm9np"
Sep 10 15:24:06.864060 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.864020    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2fdt8"]
Sep 10 15:24:06.868322 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.868303    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2fdt8"
Sep 10 15:24:06.870474 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.870454    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 15:24:06.873180 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.873146    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2fdt8"]
Sep 10 15:24:06.923649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.923615    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2fdt8"]
Sep 10 15:24:06.923904 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:24:06.923883    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-56gvc tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-2fdt8" podUID="8511e6e7-52b3-4328-9cd9-87a1e66bc3a4"
Sep 10 15:24:06.947163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.947128    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5974746f88-r8xq6"]
Sep 10 15:24:06.951203 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.951180    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5974746f88-r8xq6"
Sep 10 15:24:06.953575 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.953554    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8511e6e7-52b3-4328-9cd9-87a1e66bc3a4-tls-cert\") pod \"authorino-6d4dfc649f-2fdt8\" (UID: \"8511e6e7-52b3-4328-9cd9-87a1e66bc3a4\") " pod="kuadrant-system/authorino-6d4dfc649f-2fdt8"
Sep 10 15:24:06.953699 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.953586    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45c55\" (UniqueName: \"kubernetes.io/projected/1d9541ba-e4a0-452f-9844-e921d6d413b6-kube-api-access-45c55\") pod \"authorino-557f876f69-nm9np\" (UID: \"1d9541ba-e4a0-452f-9844-e921d6d413b6\") " pod="kuadrant-system/authorino-557f876f69-nm9np"
Sep 10 15:24:06.953699 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.953628    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56gvc\" (UniqueName: \"kubernetes.io/projected/8511e6e7-52b3-4328-9cd9-87a1e66bc3a4-kube-api-access-56gvc\") pod \"authorino-6d4dfc649f-2fdt8\" (UID: \"8511e6e7-52b3-4328-9cd9-87a1e66bc3a4\") " pod="kuadrant-system/authorino-6d4dfc649f-2fdt8"
Sep 10 15:24:06.955977 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.955907    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5974746f88-r8xq6"]
Sep 10 15:24:06.962400 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:06.962367    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45c55\" (UniqueName: \"kubernetes.io/projected/1d9541ba-e4a0-452f-9844-e921d6d413b6-kube-api-access-45c55\") pod \"authorino-557f876f69-nm9np\" (UID: \"1d9541ba-e4a0-452f-9844-e921d6d413b6\") " pod="kuadrant-system/authorino-557f876f69-nm9np"
Sep 10 15:24:07.054838 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.054779    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8511e6e7-52b3-4328-9cd9-87a1e66bc3a4-tls-cert\") pod \"authorino-6d4dfc649f-2fdt8\" (UID: \"8511e6e7-52b3-4328-9cd9-87a1e66bc3a4\") " pod="kuadrant-system/authorino-6d4dfc649f-2fdt8"
Sep 10 15:24:07.054997 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.054849    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f7f75252-3ece-4339-9744-628aec14958f-tls-cert\") pod \"authorino-5974746f88-r8xq6\" (UID: \"f7f75252-3ece-4339-9744-628aec14958f\") " pod="kuadrant-system/authorino-5974746f88-r8xq6"
Sep 10 15:24:07.054997 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.054898    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjr5\" (UniqueName: \"kubernetes.io/projected/f7f75252-3ece-4339-9744-628aec14958f-kube-api-access-5wjr5\") pod \"authorino-5974746f88-r8xq6\" (UID: \"f7f75252-3ece-4339-9744-628aec14958f\") " pod="kuadrant-system/authorino-5974746f88-r8xq6"
Sep 10 15:24:07.054997 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.054930    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56gvc\" (UniqueName: \"kubernetes.io/projected/8511e6e7-52b3-4328-9cd9-87a1e66bc3a4-kube-api-access-56gvc\") pod \"authorino-6d4dfc649f-2fdt8\" (UID: \"8511e6e7-52b3-4328-9cd9-87a1e66bc3a4\") " pod="kuadrant-system/authorino-6d4dfc649f-2fdt8"
Sep 10 15:24:07.057456 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.057426    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8511e6e7-52b3-4328-9cd9-87a1e66bc3a4-tls-cert\") pod \"authorino-6d4dfc649f-2fdt8\" (UID: \"8511e6e7-52b3-4328-9cd9-87a1e66bc3a4\") " pod="kuadrant-system/authorino-6d4dfc649f-2fdt8"
Sep 10 15:24:07.064772 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.064748    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56gvc\" (UniqueName: \"kubernetes.io/projected/8511e6e7-52b3-4328-9cd9-87a1e66bc3a4-kube-api-access-56gvc\") pod \"authorino-6d4dfc649f-2fdt8\" (UID: \"8511e6e7-52b3-4328-9cd9-87a1e66bc3a4\") " pod="kuadrant-system/authorino-6d4dfc649f-2fdt8"
Sep 10 15:24:07.155746 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.155696    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjr5\" (UniqueName: \"kubernetes.io/projected/f7f75252-3ece-4339-9744-628aec14958f-kube-api-access-5wjr5\") pod \"authorino-5974746f88-r8xq6\" (UID: \"f7f75252-3ece-4339-9744-628aec14958f\") " pod="kuadrant-system/authorino-5974746f88-r8xq6"
Sep 10 15:24:07.155949 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.155841    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f7f75252-3ece-4339-9744-628aec14958f-tls-cert\") pod \"authorino-5974746f88-r8xq6\" (UID: \"f7f75252-3ece-4339-9744-628aec14958f\") " pod="kuadrant-system/authorino-5974746f88-r8xq6"
Sep 10 15:24:07.158290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.158267    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f7f75252-3ece-4339-9744-628aec14958f-tls-cert\") pod \"authorino-5974746f88-r8xq6\" (UID: \"f7f75252-3ece-4339-9744-628aec14958f\") " pod="kuadrant-system/authorino-5974746f88-r8xq6"
Sep 10 15:24:07.165446 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.165418    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjr5\" (UniqueName: \"kubernetes.io/projected/f7f75252-3ece-4339-9744-628aec14958f-kube-api-access-5wjr5\") pod \"authorino-5974746f88-r8xq6\" (UID: \"f7f75252-3ece-4339-9744-628aec14958f\") " pod="kuadrant-system/authorino-5974746f88-r8xq6"
Sep 10 15:24:07.179919 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.179898    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2fdt8"
Sep 10 15:24:07.180006 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.179898    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nm9np"
Sep 10 15:24:07.187627 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.187607    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2fdt8"
Sep 10 15:24:07.190141 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.190125    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nm9np"
Sep 10 15:24:07.256922 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.256842    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56gvc\" (UniqueName: \"kubernetes.io/projected/8511e6e7-52b3-4328-9cd9-87a1e66bc3a4-kube-api-access-56gvc\") pod \"8511e6e7-52b3-4328-9cd9-87a1e66bc3a4\" (UID: \"8511e6e7-52b3-4328-9cd9-87a1e66bc3a4\") "
Sep 10 15:24:07.256922 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.256899    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8511e6e7-52b3-4328-9cd9-87a1e66bc3a4-tls-cert\") pod \"8511e6e7-52b3-4328-9cd9-87a1e66bc3a4\" (UID: \"8511e6e7-52b3-4328-9cd9-87a1e66bc3a4\") "
Sep 10 15:24:07.257137 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.256930    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45c55\" (UniqueName: \"kubernetes.io/projected/1d9541ba-e4a0-452f-9844-e921d6d413b6-kube-api-access-45c55\") pod \"1d9541ba-e4a0-452f-9844-e921d6d413b6\" (UID: \"1d9541ba-e4a0-452f-9844-e921d6d413b6\") "
Sep 10 15:24:07.259122 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.259094    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8511e6e7-52b3-4328-9cd9-87a1e66bc3a4-kube-api-access-56gvc" (OuterVolumeSpecName: "kube-api-access-56gvc") pod "8511e6e7-52b3-4328-9cd9-87a1e66bc3a4" (UID: "8511e6e7-52b3-4328-9cd9-87a1e66bc3a4"). InnerVolumeSpecName "kube-api-access-56gvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:24:07.259224 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.259204    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8511e6e7-52b3-4328-9cd9-87a1e66bc3a4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8511e6e7-52b3-4328-9cd9-87a1e66bc3a4" (UID: "8511e6e7-52b3-4328-9cd9-87a1e66bc3a4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:24:07.259273 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.259207    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9541ba-e4a0-452f-9844-e921d6d413b6-kube-api-access-45c55" (OuterVolumeSpecName: "kube-api-access-45c55") pod "1d9541ba-e4a0-452f-9844-e921d6d413b6" (UID: "1d9541ba-e4a0-452f-9844-e921d6d413b6"). InnerVolumeSpecName "kube-api-access-45c55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:24:07.261326 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.261309    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5974746f88-r8xq6"
Sep 10 15:24:07.357402 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.357357    2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8511e6e7-52b3-4328-9cd9-87a1e66bc3a4-tls-cert\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:24:07.357402 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.357398    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45c55\" (UniqueName: \"kubernetes.io/projected/1d9541ba-e4a0-452f-9844-e921d6d413b6-kube-api-access-45c55\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:24:07.357402 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.357411    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56gvc\" (UniqueName: \"kubernetes.io/projected/8511e6e7-52b3-4328-9cd9-87a1e66bc3a4-kube-api-access-56gvc\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:24:07.529961 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:07.529880    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5974746f88-r8xq6"]
Sep 10 15:24:07.533629 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:24:07.533603    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7f75252_3ece_4339_9744_628aec14958f.slice/crio-0b88cf20bae63b7a66a43ca7a4f059d3d3ec5aa95a47893dff7f1ac23d1eab7c WatchSource:0}: Error finding container 0b88cf20bae63b7a66a43ca7a4f059d3d3ec5aa95a47893dff7f1ac23d1eab7c: Status 404 returned error can't find the container with id 0b88cf20bae63b7a66a43ca7a4f059d3d3ec5aa95a47893dff7f1ac23d1eab7c
Sep 10 15:24:08.189992 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:08.189947    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5974746f88-r8xq6" event={"ID":"f7f75252-3ece-4339-9744-628aec14958f","Type":"ContainerStarted","Data":"0b88cf20bae63b7a66a43ca7a4f059d3d3ec5aa95a47893dff7f1ac23d1eab7c"}
Sep 10 15:24:08.189992 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:08.189976    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nm9np"
Sep 10 15:24:08.189992 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:08.189998    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-2fdt8"
Sep 10 15:24:08.216585 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:08.216557    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2fdt8"]
Sep 10 15:24:08.218037 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:08.218013    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-2fdt8"]
Sep 10 15:24:08.238766 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:08.238734    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-nm9np"]
Sep 10 15:24:08.242138 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:08.242113    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-nm9np"]
Sep 10 15:24:09.196358 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:09.196319    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5974746f88-r8xq6" event={"ID":"f7f75252-3ece-4339-9744-628aec14958f","Type":"ContainerStarted","Data":"b8eae67317142762eb73feeff743f1b5c6d9d0d771df3deb07cbc905a26e0ed0"}
Sep 10 15:24:09.211718 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:09.211666    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5974746f88-r8xq6" podStartSLOduration=2.547513119 podStartE2EDuration="3.211651069s" totalImagesPullingTime="664.13795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:24:09.210462349 +0000 UTC m=+511.626258167" watchObservedRunningTime="2026-09-10 15:24:09.211651069 +0000 UTC m=+511.627446891"
Sep 10 15:24:09.234191 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:09.234155    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qcl74"]
Sep 10 15:24:09.234374 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:09.234350    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-qcl74" podUID="ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37" containerName="authorino" containerID="cri-o://4ea2d2df0725b39bbfe48f30385e30ca10f42195070e9d719bd72def0f37513f" gracePeriod=30
Sep 10 15:24:09.473126 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:09.473102    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qcl74"
Sep 10 15:24:09.573819 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:09.573761    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc2l5\" (UniqueName: \"kubernetes.io/projected/ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37-kube-api-access-qc2l5\") pod \"ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37\" (UID: \"ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37\") "
Sep 10 15:24:09.576156 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:09.576130    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37-kube-api-access-qc2l5" (OuterVolumeSpecName: "kube-api-access-qc2l5") pod "ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37" (UID: "ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37"). InnerVolumeSpecName "kube-api-access-qc2l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:24:09.674339 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:09.674303    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc2l5\" (UniqueName: \"kubernetes.io/projected/ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37-kube-api-access-qc2l5\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:24:10.125342 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:10.125307    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d9541ba-e4a0-452f-9844-e921d6d413b6" path="/var/lib/kubelet/pods/1d9541ba-e4a0-452f-9844-e921d6d413b6/volumes"
Sep 10 15:24:10.125557 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:10.125544    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8511e6e7-52b3-4328-9cd9-87a1e66bc3a4" path="/var/lib/kubelet/pods/8511e6e7-52b3-4328-9cd9-87a1e66bc3a4/volumes"
Sep 10 15:24:10.202084 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:10.202049    2573 generic.go:356] "Generic (PLEG): container finished" podID="ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37" containerID="4ea2d2df0725b39bbfe48f30385e30ca10f42195070e9d719bd72def0f37513f" exitCode=0
Sep 10 15:24:10.202546 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:10.202099    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qcl74"
Sep 10 15:24:10.202546 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:10.202139    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qcl74" event={"ID":"ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37","Type":"ContainerDied","Data":"4ea2d2df0725b39bbfe48f30385e30ca10f42195070e9d719bd72def0f37513f"}
Sep 10 15:24:10.202546 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:10.202177    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qcl74" event={"ID":"ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37","Type":"ContainerDied","Data":"c6feaf7d511d376b7d9a5b452b4a59b4730db06d44c96bd4b94a16868713fba2"}
Sep 10 15:24:10.202546 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:10.202196    2573 scope.go:117] "RemoveContainer" containerID="4ea2d2df0725b39bbfe48f30385e30ca10f42195070e9d719bd72def0f37513f"
Sep 10 15:24:10.210188 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:10.210165    2573 scope.go:117] "RemoveContainer" containerID="4ea2d2df0725b39bbfe48f30385e30ca10f42195070e9d719bd72def0f37513f"
Sep 10 15:24:10.210463 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:24:10.210444    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea2d2df0725b39bbfe48f30385e30ca10f42195070e9d719bd72def0f37513f\": container with ID starting with 4ea2d2df0725b39bbfe48f30385e30ca10f42195070e9d719bd72def0f37513f not found: ID does not exist" containerID="4ea2d2df0725b39bbfe48f30385e30ca10f42195070e9d719bd72def0f37513f"
Sep 10 15:24:10.210542 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:10.210471    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea2d2df0725b39bbfe48f30385e30ca10f42195070e9d719bd72def0f37513f"} err="failed to get container status \"4ea2d2df0725b39bbfe48f30385e30ca10f42195070e9d719bd72def0f37513f\": rpc error: code = NotFound desc = could not find container \"4ea2d2df0725b39bbfe48f30385e30ca10f42195070e9d719bd72def0f37513f\": container with ID starting with 4ea2d2df0725b39bbfe48f30385e30ca10f42195070e9d719bd72def0f37513f not found: ID does not exist"
Sep 10 15:24:10.217638 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:10.217618    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qcl74"]
Sep 10 15:24:10.221354 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:10.221335    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qcl74"]
Sep 10 15:24:12.124590 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:12.124557    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37" path="/var/lib/kubelet/pods/ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37/volumes"
Sep 10 15:24:14.083199 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.083169    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57cc444cff-sjtz6"]
Sep 10 15:24:14.083644 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.083587    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37" containerName="authorino"
Sep 10 15:24:14.083644 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.083605    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37" containerName="authorino"
Sep 10 15:24:14.083735 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.083687    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecbb71d0-b5f8-4d4f-ae3e-9946f95d3f37" containerName="authorino"
Sep 10 15:24:14.088849 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.088829    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:14.091097 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.091072    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 15:24:14.091208 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.091097    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 15:24:14.093108 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.093079    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gqnvd\""
Sep 10 15:24:14.093108 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.093076    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 15:24:14.101908 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.101886    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57cc444cff-sjtz6"]
Sep 10 15:24:14.215706 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.215669    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-webhook-cert\") pod \"maas-controller-57cc444cff-sjtz6\" (UID: \"8de7ca77-478b-4bb1-a931-1382f4dd2796\") " pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:14.215706 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.215714    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd5bm\" (UniqueName: \"kubernetes.io/projected/8de7ca77-478b-4bb1-a931-1382f4dd2796-kube-api-access-bd5bm\") pod \"maas-controller-57cc444cff-sjtz6\" (UID: \"8de7ca77-478b-4bb1-a931-1382f4dd2796\") " pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:14.215959 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.215748    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-metrics-certs\") pod \"maas-controller-57cc444cff-sjtz6\" (UID: \"8de7ca77-478b-4bb1-a931-1382f4dd2796\") " pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:14.316729 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.316693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-metrics-certs\") pod \"maas-controller-57cc444cff-sjtz6\" (UID: \"8de7ca77-478b-4bb1-a931-1382f4dd2796\") " pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:14.316923 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.316776    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-webhook-cert\") pod \"maas-controller-57cc444cff-sjtz6\" (UID: \"8de7ca77-478b-4bb1-a931-1382f4dd2796\") " pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:14.316923 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.316831    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd5bm\" (UniqueName: \"kubernetes.io/projected/8de7ca77-478b-4bb1-a931-1382f4dd2796-kube-api-access-bd5bm\") pod \"maas-controller-57cc444cff-sjtz6\" (UID: \"8de7ca77-478b-4bb1-a931-1382f4dd2796\") " pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:14.316923 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:24:14.316881    2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 10 15:24:14.317043 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:24:14.316979    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-metrics-certs podName:8de7ca77-478b-4bb1-a931-1382f4dd2796 nodeName:}" failed. No retries permitted until 2026-09-10 15:24:14.816957298 +0000 UTC m=+517.232753095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-metrics-certs") pod "maas-controller-57cc444cff-sjtz6" (UID: "8de7ca77-478b-4bb1-a931-1382f4dd2796") : secret "maas-controller-metrics-tls" not found
Sep 10 15:24:14.319454 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.319422    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-webhook-cert\") pod \"maas-controller-57cc444cff-sjtz6\" (UID: \"8de7ca77-478b-4bb1-a931-1382f4dd2796\") " pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:14.326290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.326267    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd5bm\" (UniqueName: \"kubernetes.io/projected/8de7ca77-478b-4bb1-a931-1382f4dd2796-kube-api-access-bd5bm\") pod \"maas-controller-57cc444cff-sjtz6\" (UID: \"8de7ca77-478b-4bb1-a931-1382f4dd2796\") " pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:14.821380 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.821341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-metrics-certs\") pod \"maas-controller-57cc444cff-sjtz6\" (UID: \"8de7ca77-478b-4bb1-a931-1382f4dd2796\") " pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:14.824018 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.823992    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-metrics-certs\") pod \"maas-controller-57cc444cff-sjtz6\" (UID: \"8de7ca77-478b-4bb1-a931-1382f4dd2796\") " pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:14.999517 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:14.999475    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:15.273210 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:15.273175    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57cc444cff-sjtz6"]
Sep 10 15:24:15.276360 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:24:15.276332    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de7ca77_478b_4bb1_a931_1382f4dd2796.slice/crio-f75929116d6e1d2b5c20e29e8a738e5f6b1099877ec4ad750b850a39c7bcf5e3 WatchSource:0}: Error finding container f75929116d6e1d2b5c20e29e8a738e5f6b1099877ec4ad750b850a39c7bcf5e3: Status 404 returned error can't find the container with id f75929116d6e1d2b5c20e29e8a738e5f6b1099877ec4ad750b850a39c7bcf5e3
Sep 10 15:24:16.235926 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:16.235799    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57cc444cff-sjtz6" event={"ID":"8de7ca77-478b-4bb1-a931-1382f4dd2796","Type":"ContainerStarted","Data":"f75929116d6e1d2b5c20e29e8a738e5f6b1099877ec4ad750b850a39c7bcf5e3"}
Sep 10 15:24:18.246436 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:18.246400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57cc444cff-sjtz6" event={"ID":"8de7ca77-478b-4bb1-a931-1382f4dd2796","Type":"ContainerStarted","Data":"b5c09fb824211392adb34211dd99b7baae22e34942e05407a8d07957787575a5"}
Sep 10 15:24:18.246854 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:18.246465    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:18.274689 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:18.274620    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57cc444cff-sjtz6" podStartSLOduration=1.983745343 podStartE2EDuration="4.274598944s" totalImagesPullingTime="2.290853601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:24:18.273292834 +0000 UTC m=+520.689088661" watchObservedRunningTime="2026-09-10 15:24:18.274598944 +0000 UTC m=+520.690394777"
Sep 10 15:24:29.258118 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:29.258048    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:24:51.190812 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.190756    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"]
Sep 10 15:24:51.217580 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.217550    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"]
Sep 10 15:24:51.217726 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.217654    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.221141 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.221115    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 15:24:51.221301 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.221117    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gljcp\""
Sep 10 15:24:51.221301 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.221117    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 15:24:51.303847 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.303818    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-metrics-certs\") pod \"maas-api-788d559bf-sjxxm\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") " pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.304032 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.303863    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-maas-api-tls\") pod \"maas-api-788d559bf-sjxxm\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") " pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.304032 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.303879    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-openshift-service-ca\") pod \"maas-api-788d559bf-sjxxm\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") " pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.304032 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.303946    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgr4w\" (UniqueName: \"kubernetes.io/projected/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-kube-api-access-dgr4w\") pod \"maas-api-788d559bf-sjxxm\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") " pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.405024 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.404991    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-metrics-certs\") pod \"maas-api-788d559bf-sjxxm\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") " pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.405215 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.405052    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-maas-api-tls\") pod \"maas-api-788d559bf-sjxxm\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") " pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.405215 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.405077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-openshift-service-ca\") pod \"maas-api-788d559bf-sjxxm\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") " pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.405215 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.405106    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgr4w\" (UniqueName: \"kubernetes.io/projected/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-kube-api-access-dgr4w\") pod \"maas-api-788d559bf-sjxxm\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") " pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.405215 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:24:51.405141    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 10 15:24:51.405427 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:24:51.405219    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-metrics-certs podName:8bf98897-ed7a-4933-8a73-2b3a3bef2d46 nodeName:}" failed. No retries permitted until 2026-09-10 15:24:51.905190911 +0000 UTC m=+554.320986727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-metrics-certs") pod "maas-api-788d559bf-sjxxm" (UID: "8bf98897-ed7a-4933-8a73-2b3a3bef2d46") : secret "maas-api-metrics-tls" not found
Sep 10 15:24:51.405958 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.405885    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-openshift-service-ca\") pod \"maas-api-788d559bf-sjxxm\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") " pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.407736 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.407715    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-maas-api-tls\") pod \"maas-api-788d559bf-sjxxm\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") " pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.417774 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.417749    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgr4w\" (UniqueName: \"kubernetes.io/projected/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-kube-api-access-dgr4w\") pod \"maas-api-788d559bf-sjxxm\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") " pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.716105 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.716070    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-kxctv"]
Sep 10 15:24:51.719713 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.719694    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-kxctv"
Sep 10 15:24:51.723108 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.723089    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xl6p5\""
Sep 10 15:24:51.723246 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.723228    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 15:24:51.732428 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.732405    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-kxctv"]
Sep 10 15:24:51.808180 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.808138    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5c2x\" (UniqueName: \"kubernetes.io/projected/4e9b368b-9061-4e9d-be30-4b59ada419c8-kube-api-access-v5c2x\") pod \"payload-processing-5b54b7fbb7-kxctv\" (UID: \"4e9b368b-9061-4e9d-be30-4b59ada419c8\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kxctv"
Sep 10 15:24:51.808180 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.808174    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9b368b-9061-4e9d-be30-4b59ada419c8-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-kxctv\" (UID: \"4e9b368b-9061-4e9d-be30-4b59ada419c8\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kxctv"
Sep 10 15:24:51.909579 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.909533    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-metrics-certs\") pod \"maas-api-788d559bf-sjxxm\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") " pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.909780 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.909588    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5c2x\" (UniqueName: \"kubernetes.io/projected/4e9b368b-9061-4e9d-be30-4b59ada419c8-kube-api-access-v5c2x\") pod \"payload-processing-5b54b7fbb7-kxctv\" (UID: \"4e9b368b-9061-4e9d-be30-4b59ada419c8\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kxctv"
Sep 10 15:24:51.909780 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.909611    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9b368b-9061-4e9d-be30-4b59ada419c8-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-kxctv\" (UID: \"4e9b368b-9061-4e9d-be30-4b59ada419c8\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kxctv"
Sep 10 15:24:51.910366 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.910346    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9b368b-9061-4e9d-be30-4b59ada419c8-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-kxctv\" (UID: \"4e9b368b-9061-4e9d-be30-4b59ada419c8\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kxctv"
Sep 10 15:24:51.912141 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.912117    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-metrics-certs\") pod \"maas-api-788d559bf-sjxxm\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") " pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:51.926045 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:51.926014    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5c2x\" (UniqueName: \"kubernetes.io/projected/4e9b368b-9061-4e9d-be30-4b59ada419c8-kube-api-access-v5c2x\") pod \"payload-processing-5b54b7fbb7-kxctv\" (UID: \"4e9b368b-9061-4e9d-be30-4b59ada419c8\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-kxctv"
Sep 10 15:24:52.030497 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:52.030413    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-kxctv"
Sep 10 15:24:52.127956 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:52.127931    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:52.319825 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:52.319768    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-kxctv"]
Sep 10 15:24:52.321776 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:24:52.321750    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9b368b_9061_4e9d_be30_4b59ada419c8.slice/crio-808716ad534ccc5b0fc320dfa791b3ffde7bf68975589af9e9cb6a01767632a7 WatchSource:0}: Error finding container 808716ad534ccc5b0fc320dfa791b3ffde7bf68975589af9e9cb6a01767632a7: Status 404 returned error can't find the container with id 808716ad534ccc5b0fc320dfa791b3ffde7bf68975589af9e9cb6a01767632a7
Sep 10 15:24:52.400747 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:52.400709    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-kxctv" event={"ID":"4e9b368b-9061-4e9d-be30-4b59ada419c8","Type":"ContainerStarted","Data":"808716ad534ccc5b0fc320dfa791b3ffde7bf68975589af9e9cb6a01767632a7"}
Sep 10 15:24:52.412703 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:52.412671    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"]
Sep 10 15:24:52.415718 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:24:52.415684    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf98897_ed7a_4933_8a73_2b3a3bef2d46.slice/crio-ebb651c72de6801db4bda2f20b95ea7afeacc230ee84815b2a6c92643c71364d WatchSource:0}: Error finding container ebb651c72de6801db4bda2f20b95ea7afeacc230ee84815b2a6c92643c71364d: Status 404 returned error can't find the container with id ebb651c72de6801db4bda2f20b95ea7afeacc230ee84815b2a6c92643c71364d
Sep 10 15:24:53.409773 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:53.409730    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm" event={"ID":"8bf98897-ed7a-4933-8a73-2b3a3bef2d46","Type":"ContainerStarted","Data":"ebb651c72de6801db4bda2f20b95ea7afeacc230ee84815b2a6c92643c71364d"}
Sep 10 15:24:55.422354 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:55.422315    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-kxctv" event={"ID":"4e9b368b-9061-4e9d-be30-4b59ada419c8","Type":"ContainerStarted","Data":"947a3363ecb055b5ef313d7637e61ac1a6eb2ab805a7fa65528dc35b2c8a18bb"}
Sep 10 15:24:55.422846 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:55.422385    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-kxctv"
Sep 10 15:24:55.424145 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:55.424122    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm" event={"ID":"8bf98897-ed7a-4933-8a73-2b3a3bef2d46","Type":"ContainerStarted","Data":"3bac7b601772fdd4f60f99ff2ac62f10f79ee15e2001ba9256447b167e15e2e2"}
Sep 10 15:24:55.424278 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:55.424264    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:24:55.474156 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:55.474105    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm" podStartSLOduration=2.47385519 podStartE2EDuration="4.474090569s" totalImagesPullingTime="2.000235379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:24:55.471710121 +0000 UTC m=+557.887505941" watchObservedRunningTime="2026-09-10 15:24:55.474090569 +0000 UTC m=+557.889886387"
Sep 10 15:24:55.474315 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:24:55.474217    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-kxctv" podStartSLOduration=2.380396594 podStartE2EDuration="4.474211446s" totalImagesPullingTime="2.093814852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:24:55.447869439 +0000 UTC m=+557.863665257" watchObservedRunningTime="2026-09-10 15:24:55.474211446 +0000 UTC m=+557.890007265"
Sep 10 15:25:01.434834 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:01.434775    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:25:06.431109 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:06.431080    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-kxctv"
Sep 10 15:25:41.332663 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.332627    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"]
Sep 10 15:25:41.336089 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.336072    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.338428 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.338403    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 15:25:41.339124 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.339100    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 15:25:41.339231 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.339125    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 15:25:41.339231 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.339200    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8dhf4\""
Sep 10 15:25:41.349722 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.349698    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"]
Sep 10 15:25:41.519352 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.519315    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzr2l\" (UniqueName: \"kubernetes.io/projected/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-kube-api-access-wzr2l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.519551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.519388    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.519551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.519407    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.519551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.519423    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.519551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.519441    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.519551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.519502    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.620406 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.620321    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.620406 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.620362    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.620406 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.620388    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.620647 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.620411    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.620647 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.620448    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.620647 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.620491    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzr2l\" (UniqueName: \"kubernetes.io/projected/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-kube-api-access-wzr2l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.620846 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.620820    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.620946 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.620854    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.620946 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.620914    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.622980 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.622953    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.623158 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.623143    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.631701 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.631678    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzr2l\" (UniqueName: \"kubernetes.io/projected/a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4-kube-api-access-wzr2l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-clkjf\" (UID: \"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.646538 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.646513    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:41.924757 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:41.924731    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"]
Sep 10 15:25:41.927108 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:25:41.927083    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a4ddb4_eb74_4bc1_84c7_b57ea8a5a9d4.slice/crio-7ff67eedb42c13c352a4915318aca9e2de4e4c3690be135ed813b257aff3d247 WatchSource:0}: Error finding container 7ff67eedb42c13c352a4915318aca9e2de4e4c3690be135ed813b257aff3d247: Status 404 returned error can't find the container with id 7ff67eedb42c13c352a4915318aca9e2de4e4c3690be135ed813b257aff3d247
Sep 10 15:25:42.638491 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:42.638460    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf" event={"ID":"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4","Type":"ContainerStarted","Data":"7ff67eedb42c13c352a4915318aca9e2de4e4c3690be135ed813b257aff3d247"}
Sep 10 15:25:43.645296 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:43.645255    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf" event={"ID":"a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4","Type":"ContainerStarted","Data":"70586ec142fc6cb065db1ddf176b6bacc7499be2b400c0688a4adb83ee6b0446"}
Sep 10 15:25:43.665987 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:43.665931    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf" podStartSLOduration=1.662828621 podStartE2EDuration="2.665913389s" totalImagesPullingTime="1.003084768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:25:43.664533795 +0000 UTC m=+606.080329629" watchObservedRunningTime="2026-09-10 15:25:43.665913389 +0000 UTC m=+606.081709208"
Sep 10 15:25:51.646826 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:51.646774    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:51.647226 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:51.646857    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:51.659291 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:51.659258    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:25:51.695230 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:25:51.695203    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-clkjf"
Sep 10 15:27:09.160019 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:09.159984    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6846c95c-6w464"]
Sep 10 15:27:09.163321 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:09.163298    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6846c95c-6w464"
Sep 10 15:27:09.172397 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:09.172365    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6846c95c-6w464"]
Sep 10 15:27:09.264499 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:09.264466    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46084e47-e4cb-472d-af64-1de691f0eca0-tls-cert\") pod \"authorino-b6846c95c-6w464\" (UID: \"46084e47-e4cb-472d-af64-1de691f0eca0\") " pod="kuadrant-system/authorino-b6846c95c-6w464"
Sep 10 15:27:09.264658 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:09.264549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqzk2\" (UniqueName: \"kubernetes.io/projected/46084e47-e4cb-472d-af64-1de691f0eca0-kube-api-access-fqzk2\") pod \"authorino-b6846c95c-6w464\" (UID: \"46084e47-e4cb-472d-af64-1de691f0eca0\") " pod="kuadrant-system/authorino-b6846c95c-6w464"
Sep 10 15:27:09.365562 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:09.365528    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqzk2\" (UniqueName: \"kubernetes.io/projected/46084e47-e4cb-472d-af64-1de691f0eca0-kube-api-access-fqzk2\") pod \"authorino-b6846c95c-6w464\" (UID: \"46084e47-e4cb-472d-af64-1de691f0eca0\") " pod="kuadrant-system/authorino-b6846c95c-6w464"
Sep 10 15:27:09.365749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:09.365586    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46084e47-e4cb-472d-af64-1de691f0eca0-tls-cert\") pod \"authorino-b6846c95c-6w464\" (UID: \"46084e47-e4cb-472d-af64-1de691f0eca0\") " pod="kuadrant-system/authorino-b6846c95c-6w464"
Sep 10 15:27:09.368267 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:09.368239    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46084e47-e4cb-472d-af64-1de691f0eca0-tls-cert\") pod \"authorino-b6846c95c-6w464\" (UID: \"46084e47-e4cb-472d-af64-1de691f0eca0\") " pod="kuadrant-system/authorino-b6846c95c-6w464"
Sep 10 15:27:09.374290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:09.374258    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqzk2\" (UniqueName: \"kubernetes.io/projected/46084e47-e4cb-472d-af64-1de691f0eca0-kube-api-access-fqzk2\") pod \"authorino-b6846c95c-6w464\" (UID: \"46084e47-e4cb-472d-af64-1de691f0eca0\") " pod="kuadrant-system/authorino-b6846c95c-6w464"
Sep 10 15:27:09.473281 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:09.473196    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6846c95c-6w464"
Sep 10 15:27:09.743637 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:09.743610    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6846c95c-6w464"]
Sep 10 15:27:09.749003 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:09.748981    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:27:10.054498 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:10.054410    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6846c95c-6w464" event={"ID":"46084e47-e4cb-472d-af64-1de691f0eca0","Type":"ContainerStarted","Data":"954f2d211e85b3bccb2bfa399a1189866cc853cbd3651b2a938baef9b2da533e"}
Sep 10 15:27:11.062819 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:11.062760    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6846c95c-6w464" event={"ID":"46084e47-e4cb-472d-af64-1de691f0eca0","Type":"ContainerStarted","Data":"114bd0ae3f82f36f73fb044b0260fe11ce4361da09f25b7ba19cb234816ae7f6"}
Sep 10 15:27:11.099270 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:11.099199    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6846c95c-6w464" podStartSLOduration=1.316368924 podStartE2EDuration="2.09917439s" totalImagesPullingTime="782.805466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:11.097561649 +0000 UTC m=+693.513357468" watchObservedRunningTime="2026-09-10 15:27:11.09917439 +0000 UTC m=+693.514970210"
Sep 10 15:27:11.159342 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:11.159297    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5974746f88-r8xq6"]
Sep 10 15:27:11.160213 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:11.160160    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5974746f88-r8xq6" podUID="f7f75252-3ece-4339-9744-628aec14958f" containerName="authorino" containerID="cri-o://b8eae67317142762eb73feeff743f1b5c6d9d0d771df3deb07cbc905a26e0ed0" gracePeriod=30
Sep 10 15:27:11.424155 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:11.424118    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5974746f88-r8xq6"
Sep 10 15:27:11.482986 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:11.482951    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wjr5\" (UniqueName: \"kubernetes.io/projected/f7f75252-3ece-4339-9744-628aec14958f-kube-api-access-5wjr5\") pod \"f7f75252-3ece-4339-9744-628aec14958f\" (UID: \"f7f75252-3ece-4339-9744-628aec14958f\") "
Sep 10 15:27:11.482986 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:11.482993    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f7f75252-3ece-4339-9744-628aec14958f-tls-cert\") pod \"f7f75252-3ece-4339-9744-628aec14958f\" (UID: \"f7f75252-3ece-4339-9744-628aec14958f\") "
Sep 10 15:27:11.485279 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:11.485246    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f75252-3ece-4339-9744-628aec14958f-kube-api-access-5wjr5" (OuterVolumeSpecName: "kube-api-access-5wjr5") pod "f7f75252-3ece-4339-9744-628aec14958f" (UID: "f7f75252-3ece-4339-9744-628aec14958f"). InnerVolumeSpecName "kube-api-access-5wjr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:27:11.493501 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:11.493473    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f75252-3ece-4339-9744-628aec14958f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f7f75252-3ece-4339-9744-628aec14958f" (UID: "f7f75252-3ece-4339-9744-628aec14958f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:27:11.584077 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:11.584041    2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f7f75252-3ece-4339-9744-628aec14958f-tls-cert\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:27:11.584077 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:11.584070    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wjr5\" (UniqueName: \"kubernetes.io/projected/f7f75252-3ece-4339-9744-628aec14958f-kube-api-access-5wjr5\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:27:12.070244 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:12.070207    2573 generic.go:356] "Generic (PLEG): container finished" podID="f7f75252-3ece-4339-9744-628aec14958f" containerID="b8eae67317142762eb73feeff743f1b5c6d9d0d771df3deb07cbc905a26e0ed0" exitCode=0
Sep 10 15:27:12.070721 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:12.070251    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5974746f88-r8xq6"
Sep 10 15:27:12.070721 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:12.070302    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5974746f88-r8xq6" event={"ID":"f7f75252-3ece-4339-9744-628aec14958f","Type":"ContainerDied","Data":"b8eae67317142762eb73feeff743f1b5c6d9d0d771df3deb07cbc905a26e0ed0"}
Sep 10 15:27:12.070721 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:12.070341    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5974746f88-r8xq6" event={"ID":"f7f75252-3ece-4339-9744-628aec14958f","Type":"ContainerDied","Data":"0b88cf20bae63b7a66a43ca7a4f059d3d3ec5aa95a47893dff7f1ac23d1eab7c"}
Sep 10 15:27:12.070721 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:12.070365    2573 scope.go:117] "RemoveContainer" containerID="b8eae67317142762eb73feeff743f1b5c6d9d0d771df3deb07cbc905a26e0ed0"
Sep 10 15:27:12.080203 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:12.080173    2573 scope.go:117] "RemoveContainer" containerID="b8eae67317142762eb73feeff743f1b5c6d9d0d771df3deb07cbc905a26e0ed0"
Sep 10 15:27:12.080574 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:12.080550    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8eae67317142762eb73feeff743f1b5c6d9d0d771df3deb07cbc905a26e0ed0\": container with ID starting with b8eae67317142762eb73feeff743f1b5c6d9d0d771df3deb07cbc905a26e0ed0 not found: ID does not exist" containerID="b8eae67317142762eb73feeff743f1b5c6d9d0d771df3deb07cbc905a26e0ed0"
Sep 10 15:27:12.080661 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:12.080580    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8eae67317142762eb73feeff743f1b5c6d9d0d771df3deb07cbc905a26e0ed0"} err="failed to get container status \"b8eae67317142762eb73feeff743f1b5c6d9d0d771df3deb07cbc905a26e0ed0\": rpc error: code = NotFound desc = could not find container \"b8eae67317142762eb73feeff743f1b5c6d9d0d771df3deb07cbc905a26e0ed0\": container with ID starting with b8eae67317142762eb73feeff743f1b5c6d9d0d771df3deb07cbc905a26e0ed0 not found: ID does not exist"
Sep 10 15:27:12.093018 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:12.092989    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5974746f88-r8xq6"]
Sep 10 15:27:12.096245 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:12.096216    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5974746f88-r8xq6"]
Sep 10 15:27:12.145191 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:12.144836    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7f75252-3ece-4339-9744-628aec14958f" path="/var/lib/kubelet/pods/f7f75252-3ece-4339-9744-628aec14958f/volumes"
Sep 10 15:27:29.948424 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:29.948342    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7833-04b372"]
Sep 10 15:27:29.948774 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:29.948667    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f75252-3ece-4339-9744-628aec14958f" containerName="authorino"
Sep 10 15:27:29.948774 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:29.948679    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f75252-3ece-4339-9744-628aec14958f" containerName="authorino"
Sep 10 15:27:29.948774 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:29.948736    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f75252-3ece-4339-9744-628aec14958f" containerName="authorino"
Sep 10 15:27:29.952871 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:29.952854    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7833-04b372"
Sep 10 15:27:29.955545 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:29.955520    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h7f4v\""
Sep 10 15:27:29.959563 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:29.959529    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7833-04b372"]
Sep 10 15:27:30.044373 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:30.044329    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp2xm\" (UniqueName: \"kubernetes.io/projected/40d44972-7a24-4f89-9709-b193bef1163c-kube-api-access-rp2xm\") pod \"test-curl-7833-04b372\" (UID: \"40d44972-7a24-4f89-9709-b193bef1163c\") " pod="opendatahub/test-curl-7833-04b372"
Sep 10 15:27:30.144899 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:30.144865    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp2xm\" (UniqueName: \"kubernetes.io/projected/40d44972-7a24-4f89-9709-b193bef1163c-kube-api-access-rp2xm\") pod \"test-curl-7833-04b372\" (UID: \"40d44972-7a24-4f89-9709-b193bef1163c\") " pod="opendatahub/test-curl-7833-04b372"
Sep 10 15:27:30.153705 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:30.153681    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp2xm\" (UniqueName: \"kubernetes.io/projected/40d44972-7a24-4f89-9709-b193bef1163c-kube-api-access-rp2xm\") pod \"test-curl-7833-04b372\" (UID: \"40d44972-7a24-4f89-9709-b193bef1163c\") " pod="opendatahub/test-curl-7833-04b372"
Sep 10 15:27:30.262419 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:30.262324    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7833-04b372"
Sep 10 15:27:30.534313 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:30.534283    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7833-04b372"]
Sep 10 15:27:30.536498 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:27:30.536453    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d44972_7a24_4f89_9709_b193bef1163c.slice/crio-0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d WatchSource:0}: Error finding container 0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d: Status 404 returned error can't find the container with id 0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d
Sep 10 15:27:31.161265 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:31.161234    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7833-04b372" event={"ID":"40d44972-7a24-4f89-9709-b193bef1163c","Type":"ContainerStarted","Data":"0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d"}
Sep 10 15:27:33.174039 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.174005    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7833-04b372" event={"ID":"40d44972-7a24-4f89-9709-b193bef1163c","Type":"ContainerStarted","Data":"21c53241eaccc4460855c265abf4c65e3be641aac097d0dc3acdedc863f32a2e"}
Sep 10 15:27:33.193780 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.193438    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"]
Sep 10 15:27:33.193993 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.193845    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7833-04b372" podStartSLOduration=1.93194751 podStartE2EDuration="4.193824927s" totalImagesPullingTime="2.261877417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:33.192745554 +0000 UTC m=+715.608541402" watchObservedRunningTime="2026-09-10 15:27:33.193824927 +0000 UTC m=+715.609620746"
Sep 10 15:27:33.202847 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.202815    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:33.210991 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.210963    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-229c42-serving-cert\""
Sep 10 15:27:33.230407 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.230326    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"]
Sep 10 15:27:33.372433 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.372394    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:33.372652 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.372460    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-metrics-certs\") pod \"maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:33.372652 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.372500    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-maas-api-tls\") pod \"maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:33.372652 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.372574    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk2zg\" (UniqueName: \"kubernetes.io/projected/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-kube-api-access-pk2zg\") pod \"maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:33.473927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.473848    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-maas-api-tls\") pod \"maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:33.473927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.473905    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk2zg\" (UniqueName: \"kubernetes.io/projected/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-kube-api-access-pk2zg\") pod \"maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:33.474116 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.473946    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:33.474116 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.473978    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-metrics-certs\") pod \"maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:33.474116 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:33.474029    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-serving-cert: secret "maas-api-e2e-inf-a-229c42-serving-cert" not found
Sep 10 15:27:33.474116 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:33.474111    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-maas-api-tls podName:0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d nodeName:}" failed. No retries permitted until 2026-09-10 15:27:33.974087888 +0000 UTC m=+716.389883688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-maas-api-tls") pod "maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m" (UID: "0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d") : secret "maas-api-e2e-inf-a-229c42-serving-cert" not found
Sep 10 15:27:33.474768 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.474694    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:33.476563 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.476543    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-metrics-certs\") pod \"maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:33.487562 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.487534    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk2zg\" (UniqueName: \"kubernetes.io/projected/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-kube-api-access-pk2zg\") pod \"maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:33.876441 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.876401    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7833-04b372"]
Sep 10 15:27:33.882339 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.882298    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7833-04b372"]
Sep 10 15:27:33.978624 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.978586    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-maas-api-tls\") pod \"maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:33.981148 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:33.981114    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-maas-api-tls\") pod \"maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:34.025369 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.025327    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7833-c42cb1"]
Sep 10 15:27:34.025740 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.025722    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d44972-7a24-4f89-9709-b193bef1163c" containerName="test-curl-7833-04b372"
Sep 10 15:27:34.025847 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.025743    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d44972-7a24-4f89-9709-b193bef1163c" containerName="test-curl-7833-04b372"
Sep 10 15:27:34.025914 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.025856    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d44972-7a24-4f89-9709-b193bef1163c" containerName="test-curl-7833-04b372"
Sep 10 15:27:34.029048 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.029026    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7833-c42cb1"
Sep 10 15:27:34.037163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.037132    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7833-c42cb1"]
Sep 10 15:27:34.108228 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.108194    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"]
Sep 10 15:27:34.111482 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.111466    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"
Sep 10 15:27:34.113752 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.113724    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:34.114064 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.114049    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-229c42-dockercfg-6vk2q\""
Sep 10 15:27:34.114525 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.114502    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-229c42\""
Sep 10 15:27:34.129551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.129520    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"]
Sep 10 15:27:34.180297 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.180264    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkw5z\" (UniqueName: \"kubernetes.io/projected/7770b7ba-7a05-4104-ae36-6efbab053454-kube-api-access-mkw5z\") pod \"test-curl-7833-c42cb1\" (UID: \"7770b7ba-7a05-4104-ae36-6efbab053454\") " pod="opendatahub/test-curl-7833-c42cb1"
Sep 10 15:27:34.181171 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:34.181149    2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7833-04b372" container="test-curl-7833-04b372"
Sep 10 15:27:34.280912 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.280877    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-682md\" (UniqueName: \"kubernetes.io/projected/237414a6-4bdf-4185-985d-c66709c67d27-kube-api-access-682md\") pod \"payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck\" (UID: \"237414a6-4bdf-4185-985d-c66709c67d27\") " pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"
Sep 10 15:27:34.280912 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.280921    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkw5z\" (UniqueName: \"kubernetes.io/projected/7770b7ba-7a05-4104-ae36-6efbab053454-kube-api-access-mkw5z\") pod \"test-curl-7833-c42cb1\" (UID: \"7770b7ba-7a05-4104-ae36-6efbab053454\") " pod="opendatahub/test-curl-7833-c42cb1"
Sep 10 15:27:34.281176 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.281058    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237414a6-4bdf-4185-985d-c66709c67d27-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck\" (UID: \"237414a6-4bdf-4185-985d-c66709c67d27\") " pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"
Sep 10 15:27:34.293565 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.293524    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkw5z\" (UniqueName: \"kubernetes.io/projected/7770b7ba-7a05-4104-ae36-6efbab053454-kube-api-access-mkw5z\") pod \"test-curl-7833-c42cb1\" (UID: \"7770b7ba-7a05-4104-ae36-6efbab053454\") " pod="opendatahub/test-curl-7833-c42cb1"
Sep 10 15:27:34.339355 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.339320    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7833-c42cb1"
Sep 10 15:27:34.382323 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.382164    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-682md\" (UniqueName: \"kubernetes.io/projected/237414a6-4bdf-4185-985d-c66709c67d27-kube-api-access-682md\") pod \"payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck\" (UID: \"237414a6-4bdf-4185-985d-c66709c67d27\") " pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"
Sep 10 15:27:34.382323 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.382288    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237414a6-4bdf-4185-985d-c66709c67d27-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck\" (UID: \"237414a6-4bdf-4185-985d-c66709c67d27\") " pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"
Sep 10 15:27:34.383614 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.383229    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237414a6-4bdf-4185-985d-c66709c67d27-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck\" (UID: \"237414a6-4bdf-4185-985d-c66709c67d27\") " pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"
Sep 10 15:27:34.397755 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.397089    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-682md\" (UniqueName: \"kubernetes.io/projected/237414a6-4bdf-4185-985d-c66709c67d27-kube-api-access-682md\") pod \"payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck\" (UID: \"237414a6-4bdf-4185-985d-c66709c67d27\") " pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"
Sep 10 15:27:34.399688 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.399656    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"]
Sep 10 15:27:34.405829 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:27:34.404766    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb6fcc5_2830_4be4_90f3_003f3d4c4e3d.slice/crio-44561dd2e3f5d391cbe196860157ec7f822bfe5b8bf71e61de0ea43faf8a90f9 WatchSource:0}: Error finding container 44561dd2e3f5d391cbe196860157ec7f822bfe5b8bf71e61de0ea43faf8a90f9: Status 404 returned error can't find the container with id 44561dd2e3f5d391cbe196860157ec7f822bfe5b8bf71e61de0ea43faf8a90f9
Sep 10 15:27:34.423425 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.423344    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"
Sep 10 15:27:34.620304 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.620236    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7833-c42cb1"]
Sep 10 15:27:34.622895 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:27:34.622869    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7770b7ba_7a05_4104_ae36_6efbab053454.slice/crio-a153c0ae61a3d2e396851d7cbae48456cc7be2aa4fe75f66dfc31fd549ef9311 WatchSource:0}: Error finding container a153c0ae61a3d2e396851d7cbae48456cc7be2aa4fe75f66dfc31fd549ef9311: Status 404 returned error can't find the container with id a153c0ae61a3d2e396851d7cbae48456cc7be2aa4fe75f66dfc31fd549ef9311
Sep 10 15:27:34.700638 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:34.700609    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"]
Sep 10 15:27:34.703640 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:27:34.703615    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237414a6_4bdf_4185_985d_c66709c67d27.slice/crio-a98949f77eb19375cb73580f22d25cda3489501fdfb3fb8004166e09b93d258d WatchSource:0}: Error finding container a98949f77eb19375cb73580f22d25cda3489501fdfb3fb8004166e09b93d258d: Status 404 returned error can't find the container with id a98949f77eb19375cb73580f22d25cda3489501fdfb3fb8004166e09b93d258d
Sep 10 15:27:35.187574 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.187479    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck" event={"ID":"237414a6-4bdf-4185-985d-c66709c67d27","Type":"ContainerStarted","Data":"73512bbecfa1c4832487502c2b33cbe13ea541af8399e814b30f61e922c90954"}
Sep 10 15:27:35.187574 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.187549    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"
Sep 10 15:27:35.187574 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.187572    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck" event={"ID":"237414a6-4bdf-4185-985d-c66709c67d27","Type":"ContainerStarted","Data":"a98949f77eb19375cb73580f22d25cda3489501fdfb3fb8004166e09b93d258d"}
Sep 10 15:27:35.189358 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.189329    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7833-c42cb1" event={"ID":"7770b7ba-7a05-4104-ae36-6efbab053454","Type":"ContainerStarted","Data":"9adc56f27adcd623d7cafef3c53f1e434ebeffb68ad7a082ad50757c3085341c"}
Sep 10 15:27:35.189470 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.189365    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7833-c42cb1" event={"ID":"7770b7ba-7a05-4104-ae36-6efbab053454","Type":"ContainerStarted","Data":"a153c0ae61a3d2e396851d7cbae48456cc7be2aa4fe75f66dfc31fd549ef9311"}
Sep 10 15:27:35.191217 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.191172    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m" event={"ID":"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d","Type":"ContainerStarted","Data":"1ccfc56e2b9cfb9656db58889bae9fd39391ba5e1df89f7bb0b83912c6681da4"}
Sep 10 15:27:35.191370 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.191180    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7833-04b372" podUID="40d44972-7a24-4f89-9709-b193bef1163c" containerName="test-curl-7833-04b372" containerID="cri-o://21c53241eaccc4460855c265abf4c65e3be641aac097d0dc3acdedc863f32a2e" gracePeriod=2
Sep 10 15:27:35.191370 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.191236    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m" event={"ID":"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d","Type":"ContainerStarted","Data":"44561dd2e3f5d391cbe196860157ec7f822bfe5b8bf71e61de0ea43faf8a90f9"}
Sep 10 15:27:35.191370 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.191273    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:35.208902 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.208846    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck" podStartSLOduration=1.208827576 podStartE2EDuration="1.208827576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:35.207154125 +0000 UTC m=+717.622949943" watchObservedRunningTime="2026-09-10 15:27:35.208827576 +0000 UTC m=+717.624623398"
Sep 10 15:27:35.227254 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.227203    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m" podStartSLOduration=1.808935817 podStartE2EDuration="2.227189247s" totalImagesPullingTime="418.25343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:35.225301022 +0000 UTC m=+717.641096842" watchObservedRunningTime="2026-09-10 15:27:35.227189247 +0000 UTC m=+717.642985065"
Sep 10 15:27:35.240703 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.240657    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7833-c42cb1" podStartSLOduration=1.240642248 podStartE2EDuration="1.240642248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:35.238929278 +0000 UTC m=+717.654725091" watchObservedRunningTime="2026-09-10 15:27:35.240642248 +0000 UTC m=+717.656438068"
Sep 10 15:27:35.874812 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.868290    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7833-c42cb1"]
Sep 10 15:27:35.878593 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:35.878513    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7833-c42cb1"]
Sep 10 15:27:36.195995 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:36.195914    2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7833-c42cb1" container="test-curl-7833-c42cb1"
Sep 10 15:27:37.201535 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:37.201492    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7833-c42cb1" podUID="7770b7ba-7a05-4104-ae36-6efbab053454" containerName="test-curl-7833-c42cb1" containerID="cri-o://9adc56f27adcd623d7cafef3c53f1e434ebeffb68ad7a082ad50757c3085341c" gracePeriod=2
Sep 10 15:27:37.337557 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:37.337535    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7833-04b372"
Sep 10 15:27:37.406638 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:37.406606    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp2xm\" (UniqueName: \"kubernetes.io/projected/40d44972-7a24-4f89-9709-b193bef1163c-kube-api-access-rp2xm\") pod \"40d44972-7a24-4f89-9709-b193bef1163c\" (UID: \"40d44972-7a24-4f89-9709-b193bef1163c\") "
Sep 10 15:27:37.408930 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:37.408898    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d44972-7a24-4f89-9709-b193bef1163c-kube-api-access-rp2xm" (OuterVolumeSpecName: "kube-api-access-rp2xm") pod "40d44972-7a24-4f89-9709-b193bef1163c" (UID: "40d44972-7a24-4f89-9709-b193bef1163c"). InnerVolumeSpecName "kube-api-access-rp2xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:27:37.507568 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:37.507484    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp2xm\" (UniqueName: \"kubernetes.io/projected/40d44972-7a24-4f89-9709-b193bef1163c-kube-api-access-rp2xm\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:27:38.125643 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:38.125614    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d44972-7a24-4f89-9709-b193bef1163c" path="/var/lib/kubelet/pods/40d44972-7a24-4f89-9709-b193bef1163c/volumes"
Sep 10 15:27:38.206998 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:38.206964    2573 generic.go:356] "Generic (PLEG): container finished" podID="40d44972-7a24-4f89-9709-b193bef1163c" containerID="21c53241eaccc4460855c265abf4c65e3be641aac097d0dc3acdedc863f32a2e" exitCode=137
Sep 10 15:27:38.207410 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:38.207007    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7833-04b372"
Sep 10 15:27:38.207410 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:38.207019    2573 scope.go:117] "RemoveContainer" containerID="21c53241eaccc4460855c265abf4c65e3be641aac097d0dc3acdedc863f32a2e"
Sep 10 15:27:38.215096 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:38.215068    2573 scope.go:117] "RemoveContainer" containerID="21c53241eaccc4460855c265abf4c65e3be641aac097d0dc3acdedc863f32a2e"
Sep 10 15:27:38.215370 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:38.215354    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c53241eaccc4460855c265abf4c65e3be641aac097d0dc3acdedc863f32a2e\": container with ID starting with 21c53241eaccc4460855c265abf4c65e3be641aac097d0dc3acdedc863f32a2e not found: ID does not exist" containerID="21c53241eaccc4460855c265abf4c65e3be641aac097d0dc3acdedc863f32a2e"
Sep 10 15:27:38.215414 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:38.215380    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c53241eaccc4460855c265abf4c65e3be641aac097d0dc3acdedc863f32a2e"} err="failed to get container status \"21c53241eaccc4460855c265abf4c65e3be641aac097d0dc3acdedc863f32a2e\": rpc error: code = NotFound desc = could not find container \"21c53241eaccc4460855c265abf4c65e3be641aac097d0dc3acdedc863f32a2e\": container with ID starting with 21c53241eaccc4460855c265abf4c65e3be641aac097d0dc3acdedc863f32a2e not found: ID does not exist"
Sep 10 15:27:39.383639 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:39.383611    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7833-c42cb1"
Sep 10 15:27:39.523291 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:39.523200    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkw5z\" (UniqueName: \"kubernetes.io/projected/7770b7ba-7a05-4104-ae36-6efbab053454-kube-api-access-mkw5z\") pod \"7770b7ba-7a05-4104-ae36-6efbab053454\" (UID: \"7770b7ba-7a05-4104-ae36-6efbab053454\") "
Sep 10 15:27:39.525470 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:39.525436    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7770b7ba-7a05-4104-ae36-6efbab053454-kube-api-access-mkw5z" (OuterVolumeSpecName: "kube-api-access-mkw5z") pod "7770b7ba-7a05-4104-ae36-6efbab053454" (UID: "7770b7ba-7a05-4104-ae36-6efbab053454"). InnerVolumeSpecName "kube-api-access-mkw5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:27:39.624219 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:39.624184    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkw5z\" (UniqueName: \"kubernetes.io/projected/7770b7ba-7a05-4104-ae36-6efbab053454-kube-api-access-mkw5z\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:27:39.862323 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:39.862291    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d44972_7a24_4f89_9709_b193bef1163c.slice/crio-0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d\": RecentStats: unable to find data in memory cache]"
Sep 10 15:27:40.126055 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.125979    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7770b7ba-7a05-4104-ae36-6efbab053454" path="/var/lib/kubelet/pods/7770b7ba-7a05-4104-ae36-6efbab053454/volumes"
Sep 10 15:27:40.221315 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.221283    2573 generic.go:356] "Generic (PLEG): container finished" podID="7770b7ba-7a05-4104-ae36-6efbab053454" containerID="9adc56f27adcd623d7cafef3c53f1e434ebeffb68ad7a082ad50757c3085341c" exitCode=137
Sep 10 15:27:40.221599 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.221341    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7833-c42cb1"
Sep 10 15:27:40.221599 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.221379    2573 scope.go:117] "RemoveContainer" containerID="9adc56f27adcd623d7cafef3c53f1e434ebeffb68ad7a082ad50757c3085341c"
Sep 10 15:27:40.229622 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.229601    2573 scope.go:117] "RemoveContainer" containerID="9adc56f27adcd623d7cafef3c53f1e434ebeffb68ad7a082ad50757c3085341c"
Sep 10 15:27:40.229931 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:40.229910    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9adc56f27adcd623d7cafef3c53f1e434ebeffb68ad7a082ad50757c3085341c\": container with ID starting with 9adc56f27adcd623d7cafef3c53f1e434ebeffb68ad7a082ad50757c3085341c not found: ID does not exist" containerID="9adc56f27adcd623d7cafef3c53f1e434ebeffb68ad7a082ad50757c3085341c"
Sep 10 15:27:40.229978 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.229942    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9adc56f27adcd623d7cafef3c53f1e434ebeffb68ad7a082ad50757c3085341c"} err="failed to get container status \"9adc56f27adcd623d7cafef3c53f1e434ebeffb68ad7a082ad50757c3085341c\": rpc error: code = NotFound desc = could not find container \"9adc56f27adcd623d7cafef3c53f1e434ebeffb68ad7a082ad50757c3085341c\": container with ID starting with 9adc56f27adcd623d7cafef3c53f1e434ebeffb68ad7a082ad50757c3085341c not found: ID does not exist"
Sep 10 15:27:40.749061 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.749023    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"]
Sep 10 15:27:40.749611 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.749579    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7770b7ba-7a05-4104-ae36-6efbab053454" containerName="test-curl-7833-c42cb1"
Sep 10 15:27:40.749705 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.749616    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7770b7ba-7a05-4104-ae36-6efbab053454" containerName="test-curl-7833-c42cb1"
Sep 10 15:27:40.749770 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.749724    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7770b7ba-7a05-4104-ae36-6efbab053454" containerName="test-curl-7833-c42cb1"
Sep 10 15:27:40.754178 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.754154    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:40.756478 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.756456    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8b74e71b-serving-cert\""
Sep 10 15:27:40.764823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.764779    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"]
Sep 10 15:27:40.833743 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.833701    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b38a64dd-a0c9-4247-a591-6e969faad94c-openshift-service-ca\") pod \"maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:40.833956 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.833761    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-maas-api-tls\") pod \"maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:40.833956 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.833875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmwlz\" (UniqueName: \"kubernetes.io/projected/b38a64dd-a0c9-4247-a591-6e969faad94c-kube-api-access-pmwlz\") pod \"maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:40.833956 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.833915    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-metrics-certs\") pod \"maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:40.934584 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.934547    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwlz\" (UniqueName: \"kubernetes.io/projected/b38a64dd-a0c9-4247-a591-6e969faad94c-kube-api-access-pmwlz\") pod \"maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:40.934825 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.934592    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-metrics-certs\") pod \"maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:40.934825 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.934619    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b38a64dd-a0c9-4247-a591-6e969faad94c-openshift-service-ca\") pod \"maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:40.934825 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.934647    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-maas-api-tls\") pod \"maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:40.934825 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:40.934757    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-serving-cert: secret "maas-api-e2e-ait-8b74e71b-serving-cert" not found
Sep 10 15:27:40.935054 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:40.934835    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-maas-api-tls podName:b38a64dd-a0c9-4247-a591-6e969faad94c nodeName:}" failed. No retries permitted until 2026-09-10 15:27:41.434815838 +0000 UTC m=+723.850611648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-maas-api-tls") pod "maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5" (UID: "b38a64dd-a0c9-4247-a591-6e969faad94c") : secret "maas-api-e2e-ait-8b74e71b-serving-cert" not found
Sep 10 15:27:40.935429 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.935402    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b38a64dd-a0c9-4247-a591-6e969faad94c-openshift-service-ca\") pod \"maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:40.937374 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.937354    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-metrics-certs\") pod \"maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:40.943769 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:40.943700    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmwlz\" (UniqueName: \"kubernetes.io/projected/b38a64dd-a0c9-4247-a591-6e969faad94c-kube-api-access-pmwlz\") pod \"maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:41.201753 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.201718    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:27:41.438661 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.438629    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-maas-api-tls\") pod \"maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:41.441063 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.441044    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-maas-api-tls\") pod \"maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:41.666645 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.666603    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:41.675375 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.675336    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"]
Sep 10 15:27:41.679437 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.679290    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"
Sep 10 15:27:41.683154 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.683128    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8b74e71b-dockercfg-c9jgw\""
Sep 10 15:27:41.683292 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.683183    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8b74e71b\""
Sep 10 15:27:41.706036 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.705997    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"]
Sep 10 15:27:41.741108 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.741071    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbshp\" (UniqueName: \"kubernetes.io/projected/bd991ca2-2d30-427c-8ea5-9c44cbd85543-kube-api-access-tbshp\") pod \"payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6\" (UID: \"bd991ca2-2d30-427c-8ea5-9c44cbd85543\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"
Sep 10 15:27:41.741251 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.741111    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd991ca2-2d30-427c-8ea5-9c44cbd85543-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6\" (UID: \"bd991ca2-2d30-427c-8ea5-9c44cbd85543\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"
Sep 10 15:27:41.841676 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.841643    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbshp\" (UniqueName: \"kubernetes.io/projected/bd991ca2-2d30-427c-8ea5-9c44cbd85543-kube-api-access-tbshp\") pod \"payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6\" (UID: \"bd991ca2-2d30-427c-8ea5-9c44cbd85543\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"
Sep 10 15:27:41.842056 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.841686    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd991ca2-2d30-427c-8ea5-9c44cbd85543-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6\" (UID: \"bd991ca2-2d30-427c-8ea5-9c44cbd85543\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"
Sep 10 15:27:41.842388 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.842369    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd991ca2-2d30-427c-8ea5-9c44cbd85543-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6\" (UID: \"bd991ca2-2d30-427c-8ea5-9c44cbd85543\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"
Sep 10 15:27:41.857849 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.857821    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbshp\" (UniqueName: \"kubernetes.io/projected/bd991ca2-2d30-427c-8ea5-9c44cbd85543-kube-api-access-tbshp\") pod \"payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6\" (UID: \"bd991ca2-2d30-427c-8ea5-9c44cbd85543\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"
Sep 10 15:27:41.961583 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:41.961555    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"]
Sep 10 15:27:41.963821 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:27:41.963777    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38a64dd_a0c9_4247_a591_6e969faad94c.slice/crio-e6839514dd4cf26ef4651c08a03577330bfe50e8598b85368db9ae2cf7342d28 WatchSource:0}: Error finding container e6839514dd4cf26ef4651c08a03577330bfe50e8598b85368db9ae2cf7342d28: Status 404 returned error can't find the container with id e6839514dd4cf26ef4651c08a03577330bfe50e8598b85368db9ae2cf7342d28
Sep 10 15:27:42.002778 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.002742    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"
Sep 10 15:27:42.075198 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.075160    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"]
Sep 10 15:27:42.081351 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.081326    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.084408 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.084384    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-2e2b69-serving-cert\""
Sep 10 15:27:42.096005 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.095818    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"]
Sep 10 15:27:42.143934 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.143900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5ef197-45a1-45bf-9f96-356f16904af1-openshift-service-ca\") pod \"maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.144202 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.143976    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-859dg\" (UniqueName: \"kubernetes.io/projected/4a5ef197-45a1-45bf-9f96-356f16904af1-kube-api-access-859dg\") pod \"maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.144202 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.144094    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-metrics-certs\") pod \"maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.144202 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.144138    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-maas-api-tls\") pod \"maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.233294 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.233214    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5" event={"ID":"b38a64dd-a0c9-4247-a591-6e969faad94c","Type":"ContainerStarted","Data":"e6839514dd4cf26ef4651c08a03577330bfe50e8598b85368db9ae2cf7342d28"}
Sep 10 15:27:42.244746 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.244717    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-metrics-certs\") pod \"maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.244893 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.244753    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-maas-api-tls\") pod \"maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.244893 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.244831    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5ef197-45a1-45bf-9f96-356f16904af1-openshift-service-ca\") pod \"maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.244893 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.244868    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859dg\" (UniqueName: \"kubernetes.io/projected/4a5ef197-45a1-45bf-9f96-356f16904af1-kube-api-access-859dg\") pod \"maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.245015 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:42.244959    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-serving-cert: secret "maas-api-e2e-autores-2e2b69-serving-cert" not found
Sep 10 15:27:42.245051 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:42.245028    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-maas-api-tls podName:4a5ef197-45a1-45bf-9f96-356f16904af1 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:42.74501116 +0000 UTC m=+725.160806960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-maas-api-tls") pod "maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st" (UID: "4a5ef197-45a1-45bf-9f96-356f16904af1") : secret "maas-api-e2e-autores-2e2b69-serving-cert" not found
Sep 10 15:27:42.245522 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.245501    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5ef197-45a1-45bf-9f96-356f16904af1-openshift-service-ca\") pod \"maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.247388 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.247367    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-metrics-certs\") pod \"maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.258955 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.258935    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-859dg\" (UniqueName: \"kubernetes.io/projected/4a5ef197-45a1-45bf-9f96-356f16904af1-kube-api-access-859dg\") pod \"maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.290304 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.290280    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"]
Sep 10 15:27:42.293109 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:27:42.293083    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd991ca2_2d30_427c_8ea5_9c44cbd85543.slice/crio-8a7a7a57ca770021a8913ac083e2a414a9d1f72c1795f8b1f2b008805b4c0745 WatchSource:0}: Error finding container 8a7a7a57ca770021a8913ac083e2a414a9d1f72c1795f8b1f2b008805b4c0745: Status 404 returned error can't find the container with id 8a7a7a57ca770021a8913ac083e2a414a9d1f72c1795f8b1f2b008805b4c0745
Sep 10 15:27:42.753779 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.753739    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-maas-api-tls\") pod \"maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.757902 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.757857    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-maas-api-tls\") pod \"maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:42.858414 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.858377    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"]
Sep 10 15:27:42.861892 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.861874    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"
Sep 10 15:27:42.864171 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.864146    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-2e2b69-dockercfg-dx885\""
Sep 10 15:27:42.864426 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.864409    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-2e2b69\""
Sep 10 15:27:42.875399 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.875318    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"]
Sep 10 15:27:42.956741 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.956701    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk87n\" (UniqueName: \"kubernetes.io/projected/56a611a3-834d-47c2-859b-c95fe6ba59b4-kube-api-access-bk87n\") pod \"payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl\" (UID: \"56a611a3-834d-47c2-859b-c95fe6ba59b4\") " pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"
Sep 10 15:27:42.956921 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.956753    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a611a3-834d-47c2-859b-c95fe6ba59b4-plugins-config-volume\") pod \"payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl\" (UID: \"56a611a3-834d-47c2-859b-c95fe6ba59b4\") " pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"
Sep 10 15:27:42.992828 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:42.992773    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:43.057454 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.057420    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk87n\" (UniqueName: \"kubernetes.io/projected/56a611a3-834d-47c2-859b-c95fe6ba59b4-kube-api-access-bk87n\") pod \"payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl\" (UID: \"56a611a3-834d-47c2-859b-c95fe6ba59b4\") " pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"
Sep 10 15:27:43.057594 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.057479    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a611a3-834d-47c2-859b-c95fe6ba59b4-plugins-config-volume\") pod \"payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl\" (UID: \"56a611a3-834d-47c2-859b-c95fe6ba59b4\") " pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"
Sep 10 15:27:43.058304 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.058283    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a611a3-834d-47c2-859b-c95fe6ba59b4-plugins-config-volume\") pod \"payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl\" (UID: \"56a611a3-834d-47c2-859b-c95fe6ba59b4\") " pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"
Sep 10 15:27:43.066330 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.066302    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk87n\" (UniqueName: \"kubernetes.io/projected/56a611a3-834d-47c2-859b-c95fe6ba59b4-kube-api-access-bk87n\") pod \"payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl\" (UID: \"56a611a3-834d-47c2-859b-c95fe6ba59b4\") " pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"
Sep 10 15:27:43.175489 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.175452    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"
Sep 10 15:27:43.243126 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.243088    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5" event={"ID":"b38a64dd-a0c9-4247-a591-6e969faad94c","Type":"ContainerStarted","Data":"5d000fc8a1aa09c01761c498bcab7d820073e81aa8303adc0b1d8f651eee2971"}
Sep 10 15:27:43.243382 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.243364    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:43.245195 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.245165    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6" event={"ID":"bd991ca2-2d30-427c-8ea5-9c44cbd85543","Type":"ContainerStarted","Data":"b418d9889d90c1e81445e6477ccca33819deed85d2efb5d70da729843db82f61"}
Sep 10 15:27:43.245326 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.245220    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6" event={"ID":"bd991ca2-2d30-427c-8ea5-9c44cbd85543","Type":"ContainerStarted","Data":"8a7a7a57ca770021a8913ac083e2a414a9d1f72c1795f8b1f2b008805b4c0745"}
Sep 10 15:27:43.245416 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.245400    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"
Sep 10 15:27:43.264078 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.264028    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5" podStartSLOduration=2.976534527 podStartE2EDuration="3.264010046s" totalImagesPullingTime="287.475519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:43.262571431 +0000 UTC m=+725.678367251" watchObservedRunningTime="2026-09-10 15:27:43.264010046 +0000 UTC m=+725.679805899"
Sep 10 15:27:43.270653 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.270435    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"]
Sep 10 15:27:43.282139 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.282085    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6" podStartSLOduration=2.28207426 podStartE2EDuration="2.28207426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:43.281337728 +0000 UTC m=+725.697133542" watchObservedRunningTime="2026-09-10 15:27:43.28207426 +0000 UTC m=+725.697870080"
Sep 10 15:27:43.453536 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:43.453509    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"]
Sep 10 15:27:43.455877 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:27:43.455839    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a611a3_834d_47c2_859b_c95fe6ba59b4.slice/crio-41a33af606199c9a182f7d2faeae45fd68d531a4112c9a5e998b5aaf649a0f10 WatchSource:0}: Error finding container 41a33af606199c9a182f7d2faeae45fd68d531a4112c9a5e998b5aaf649a0f10: Status 404 returned error can't find the container with id 41a33af606199c9a182f7d2faeae45fd68d531a4112c9a5e998b5aaf649a0f10
Sep 10 15:27:44.256069 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:44.256030    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl" event={"ID":"56a611a3-834d-47c2-859b-c95fe6ba59b4","Type":"ContainerStarted","Data":"351850e6d78eaf00d14d74099f6842923b5b6acd621657c1b44d939526341b51"}
Sep 10 15:27:44.256543 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:44.256096    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl" event={"ID":"56a611a3-834d-47c2-859b-c95fe6ba59b4","Type":"ContainerStarted","Data":"41a33af606199c9a182f7d2faeae45fd68d531a4112c9a5e998b5aaf649a0f10"}
Sep 10 15:27:44.256543 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:44.256194    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"
Sep 10 15:27:44.258069 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:44.258042    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st" event={"ID":"4a5ef197-45a1-45bf-9f96-356f16904af1","Type":"ContainerStarted","Data":"be4d1988c34eeb91ec09e52375f45981d10f2720c591e0dbe1a1b8a9fc20a073"}
Sep 10 15:27:44.258185 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:44.258090    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st" event={"ID":"4a5ef197-45a1-45bf-9f96-356f16904af1","Type":"ContainerStarted","Data":"88623889b60499dc4b8a943324f706cf8b1691bc13fbd278369d75cbf2dd8c4f"}
Sep 10 15:27:44.258185 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:44.258163    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:44.276224 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:44.276167    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl" podStartSLOduration=2.276148833 podStartE2EDuration="2.276148833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:44.274505757 +0000 UTC m=+726.690301578" watchObservedRunningTime="2026-09-10 15:27:44.276148833 +0000 UTC m=+726.691944653"
Sep 10 15:27:44.293300 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:44.293242    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st" podStartSLOduration=1.9774996360000001 podStartE2EDuration="2.293215429s" totalImagesPullingTime="315.715793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:44.29176884 +0000 UTC m=+726.707564660" watchObservedRunningTime="2026-09-10 15:27:44.293215429 +0000 UTC m=+726.709011249"
Sep 10 15:27:46.198451 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:46.198421    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"
Sep 10 15:27:49.263747 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:49.263719    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:27:49.904806 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:49.904764    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d44972_7a24_4f89_9709_b193bef1163c.slice/crio-0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d\": RecentStats: unable to find data in memory cache]"
Sep 10 15:27:50.269388 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:50.269299    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:27:52.955441 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:52.955399    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d44972_7a24_4f89_9709_b193bef1163c.slice/crio-0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d\": RecentStats: unable to find data in memory cache]"
Sep 10 15:27:54.260809 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:54.260768    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"
Sep 10 15:27:55.265980 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:27:55.265950    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"
Sep 10 15:27:59.952068 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:27:59.952021    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d44972_7a24_4f89_9709_b193bef1163c.slice/crio-0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d\": RecentStats: unable to find data in memory cache]"
Sep 10 15:28:02.957336 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:02.957297    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"]
Sep 10 15:28:02.960745 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:02.960726    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:02.963619 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:02.963594    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-50fe56-serving-cert\""
Sep 10 15:28:02.974764 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:02.974728    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"]
Sep 10 15:28:03.035605 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.035567    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:03.035605 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.035608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds4mg\" (UniqueName: \"kubernetes.io/projected/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-kube-api-access-ds4mg\") pod \"maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:03.035831 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.035704    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-metrics-certs\") pod \"maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:03.035831 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.035725    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-maas-api-tls\") pod \"maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:03.136726 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.136682    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds4mg\" (UniqueName: \"kubernetes.io/projected/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-kube-api-access-ds4mg\") pod \"maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:03.136964 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.136767    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-metrics-certs\") pod \"maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:03.136964 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.136824    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-maas-api-tls\") pod \"maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:03.136964 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.136903    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:03.137137 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:03.137024    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-serving-cert: secret "maas-api-e2e-inf-b-50fe56-serving-cert" not found
Sep 10 15:28:03.137137 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:03.137116    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-maas-api-tls podName:d91c8248-5db1-4ce5-8930-ca9b1e7a84c9 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:03.637099516 +0000 UTC m=+746.052895313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-maas-api-tls") pod "maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln" (UID: "d91c8248-5db1-4ce5-8930-ca9b1e7a84c9") : secret "maas-api-e2e-inf-b-50fe56-serving-cert" not found
Sep 10 15:28:03.137574 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.137551    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:03.139337 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.139316    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-metrics-certs\") pod \"maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:03.148046 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.148024    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds4mg\" (UniqueName: \"kubernetes.io/projected/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-kube-api-access-ds4mg\") pod \"maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:03.642158 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.642122    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-maas-api-tls\") pod \"maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:03.644741 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.644710    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-maas-api-tls\") pod \"maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:03.846998 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:03.846961    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d44972_7a24_4f89_9709_b193bef1163c.slice/crio-0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d\": RecentStats: unable to find data in memory cache]"
Sep 10 15:28:03.847282 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:03.847258    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d44972_7a24_4f89_9709_b193bef1163c.slice/crio-0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d\": RecentStats: unable to find data in memory cache]"
Sep 10 15:28:03.875785 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:03.875753    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:04.157229 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:04.157201    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"]
Sep 10 15:28:04.159902 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:28:04.159871    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91c8248_5db1_4ce5_8930_ca9b1e7a84c9.slice/crio-efa7d829825074a1ea6a368ee5fa1baa5ced76a9c307adf55aaf82e866860dde WatchSource:0}: Error finding container efa7d829825074a1ea6a368ee5fa1baa5ced76a9c307adf55aaf82e866860dde: Status 404 returned error can't find the container with id efa7d829825074a1ea6a368ee5fa1baa5ced76a9c307adf55aaf82e866860dde
Sep 10 15:28:04.362710 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:04.362675    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln" event={"ID":"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9","Type":"ContainerStarted","Data":"efa7d829825074a1ea6a368ee5fa1baa5ced76a9c307adf55aaf82e866860dde"}
Sep 10 15:28:05.370853 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:05.370815    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln" event={"ID":"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9","Type":"ContainerStarted","Data":"dbf374aa88399ae1c162eb44697f60968379a8052374e3f71311b7c184aa6d7f"}
Sep 10 15:28:05.371343 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:05.370986    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:05.390865 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:05.390812    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln" podStartSLOduration=3.031161801 podStartE2EDuration="3.390767994s" totalImagesPullingTime="359.606193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:05.389836832 +0000 UTC m=+747.805632652" watchObservedRunningTime="2026-09-10 15:28:05.390767994 +0000 UTC m=+747.806563812"
Sep 10 15:28:08.006151 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:08.006115    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d44972_7a24_4f89_9709_b193bef1163c.slice/crio-0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d\": RecentStats: unable to find data in memory cache]"
Sep 10 15:28:09.964211 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:09.964170    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d44972_7a24_4f89_9709_b193bef1163c.slice/crio-0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d\": RecentStats: unable to find data in memory cache]"
Sep 10 15:28:11.382505 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:11.382476    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:28:20.011471 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:20.011432    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d44972_7a24_4f89_9709_b193bef1163c.slice/crio-0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d\": RecentStats: unable to find data in memory cache]"
Sep 10 15:28:21.596721 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:21.596679    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"]
Sep 10 15:28:21.597324 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:21.597009    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5" podUID="b38a64dd-a0c9-4247-a591-6e969faad94c" containerName="maas-api" containerID="cri-o://5d000fc8a1aa09c01761c498bcab7d820073e81aa8303adc0b1d8f651eee2971" gracePeriod=30
Sep 10 15:28:21.688040 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:21.688004    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"]
Sep 10 15:28:21.688276 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:21.688251    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6" podUID="bd991ca2-2d30-427c-8ea5-9c44cbd85543" containerName="payload-pre-processing" containerID="cri-o://b418d9889d90c1e81445e6477ccca33819deed85d2efb5d70da729843db82f61" gracePeriod=30
Sep 10 15:28:21.951130 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:21.951100    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:28:21.980758 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:21.980726    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"
Sep 10 15:28:22.116476 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.116392    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmwlz\" (UniqueName: \"kubernetes.io/projected/b38a64dd-a0c9-4247-a591-6e969faad94c-kube-api-access-pmwlz\") pod \"b38a64dd-a0c9-4247-a591-6e969faad94c\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") "
Sep 10 15:28:22.116476 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.116449    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-maas-api-tls\") pod \"b38a64dd-a0c9-4247-a591-6e969faad94c\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") "
Sep 10 15:28:22.116689 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.116569    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd991ca2-2d30-427c-8ea5-9c44cbd85543-plugins-config-volume\") pod \"bd991ca2-2d30-427c-8ea5-9c44cbd85543\" (UID: \"bd991ca2-2d30-427c-8ea5-9c44cbd85543\") "
Sep 10 15:28:22.116689 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.116613    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b38a64dd-a0c9-4247-a591-6e969faad94c-openshift-service-ca\") pod \"b38a64dd-a0c9-4247-a591-6e969faad94c\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") "
Sep 10 15:28:22.116689 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.116636    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbshp\" (UniqueName: \"kubernetes.io/projected/bd991ca2-2d30-427c-8ea5-9c44cbd85543-kube-api-access-tbshp\") pod \"bd991ca2-2d30-427c-8ea5-9c44cbd85543\" (UID: \"bd991ca2-2d30-427c-8ea5-9c44cbd85543\") "
Sep 10 15:28:22.116889 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.116817    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-metrics-certs\") pod \"b38a64dd-a0c9-4247-a591-6e969faad94c\" (UID: \"b38a64dd-a0c9-4247-a591-6e969faad94c\") "
Sep 10 15:28:22.117007 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.116983    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38a64dd-a0c9-4247-a591-6e969faad94c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b38a64dd-a0c9-4247-a591-6e969faad94c" (UID: "b38a64dd-a0c9-4247-a591-6e969faad94c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:28:22.117083 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.117000    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd991ca2-2d30-427c-8ea5-9c44cbd85543-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd991ca2-2d30-427c-8ea5-9c44cbd85543" (UID: "bd991ca2-2d30-427c-8ea5-9c44cbd85543"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:28:22.117159 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.117127    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd991ca2-2d30-427c-8ea5-9c44cbd85543-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:28:22.117159 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.117149    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b38a64dd-a0c9-4247-a591-6e969faad94c-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:28:22.119067 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.119036    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b38a64dd-a0c9-4247-a591-6e969faad94c" (UID: "b38a64dd-a0c9-4247-a591-6e969faad94c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:28:22.119168 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.119077    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38a64dd-a0c9-4247-a591-6e969faad94c-kube-api-access-pmwlz" (OuterVolumeSpecName: "kube-api-access-pmwlz") pod "b38a64dd-a0c9-4247-a591-6e969faad94c" (UID: "b38a64dd-a0c9-4247-a591-6e969faad94c"). InnerVolumeSpecName "kube-api-access-pmwlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:28:22.119248 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.119223    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b38a64dd-a0c9-4247-a591-6e969faad94c" (UID: "b38a64dd-a0c9-4247-a591-6e969faad94c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:28:22.119296 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.119251    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd991ca2-2d30-427c-8ea5-9c44cbd85543-kube-api-access-tbshp" (OuterVolumeSpecName: "kube-api-access-tbshp") pod "bd991ca2-2d30-427c-8ea5-9c44cbd85543" (UID: "bd991ca2-2d30-427c-8ea5-9c44cbd85543"). InnerVolumeSpecName "kube-api-access-tbshp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:28:22.218414 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.218372    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:28:22.218596 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.218426    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbshp\" (UniqueName: \"kubernetes.io/projected/bd991ca2-2d30-427c-8ea5-9c44cbd85543-kube-api-access-tbshp\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:28:22.218596 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.218438    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b38a64dd-a0c9-4247-a591-6e969faad94c-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:28:22.218596 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.218447    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmwlz\" (UniqueName: \"kubernetes.io/projected/b38a64dd-a0c9-4247-a591-6e969faad94c-kube-api-access-pmwlz\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:28:22.461210 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.461115    2573 generic.go:356] "Generic (PLEG): container finished" podID="bd991ca2-2d30-427c-8ea5-9c44cbd85543" containerID="b418d9889d90c1e81445e6477ccca33819deed85d2efb5d70da729843db82f61" exitCode=0
Sep 10 15:28:22.461210 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.461190    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"
Sep 10 15:28:22.461424 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.461192    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6" event={"ID":"bd991ca2-2d30-427c-8ea5-9c44cbd85543","Type":"ContainerDied","Data":"b418d9889d90c1e81445e6477ccca33819deed85d2efb5d70da729843db82f61"}
Sep 10 15:28:22.461424 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.461350    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6" event={"ID":"bd991ca2-2d30-427c-8ea5-9c44cbd85543","Type":"ContainerDied","Data":"8a7a7a57ca770021a8913ac083e2a414a9d1f72c1795f8b1f2b008805b4c0745"}
Sep 10 15:28:22.461424 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.461383    2573 scope.go:117] "RemoveContainer" containerID="b418d9889d90c1e81445e6477ccca33819deed85d2efb5d70da729843db82f61"
Sep 10 15:28:22.462879 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.462851    2573 generic.go:356] "Generic (PLEG): container finished" podID="b38a64dd-a0c9-4247-a591-6e969faad94c" containerID="5d000fc8a1aa09c01761c498bcab7d820073e81aa8303adc0b1d8f651eee2971" exitCode=0
Sep 10 15:28:22.463002 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.462900    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"
Sep 10 15:28:22.463002 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.462908    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5" event={"ID":"b38a64dd-a0c9-4247-a591-6e969faad94c","Type":"ContainerDied","Data":"5d000fc8a1aa09c01761c498bcab7d820073e81aa8303adc0b1d8f651eee2971"}
Sep 10 15:28:22.463139 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.463056    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5" event={"ID":"b38a64dd-a0c9-4247-a591-6e969faad94c","Type":"ContainerDied","Data":"e6839514dd4cf26ef4651c08a03577330bfe50e8598b85368db9ae2cf7342d28"}
Sep 10 15:28:22.471763 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.471741    2573 scope.go:117] "RemoveContainer" containerID="b418d9889d90c1e81445e6477ccca33819deed85d2efb5d70da729843db82f61"
Sep 10 15:28:22.472120 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:22.472100    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b418d9889d90c1e81445e6477ccca33819deed85d2efb5d70da729843db82f61\": container with ID starting with b418d9889d90c1e81445e6477ccca33819deed85d2efb5d70da729843db82f61 not found: ID does not exist" containerID="b418d9889d90c1e81445e6477ccca33819deed85d2efb5d70da729843db82f61"
Sep 10 15:28:22.472212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.472132    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b418d9889d90c1e81445e6477ccca33819deed85d2efb5d70da729843db82f61"} err="failed to get container status \"b418d9889d90c1e81445e6477ccca33819deed85d2efb5d70da729843db82f61\": rpc error: code = NotFound desc = could not find container \"b418d9889d90c1e81445e6477ccca33819deed85d2efb5d70da729843db82f61\": container with ID starting with b418d9889d90c1e81445e6477ccca33819deed85d2efb5d70da729843db82f61 not found: ID does not exist"
Sep 10 15:28:22.472212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.472156    2573 scope.go:117] "RemoveContainer" containerID="5d000fc8a1aa09c01761c498bcab7d820073e81aa8303adc0b1d8f651eee2971"
Sep 10 15:28:22.481088 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.481068    2573 scope.go:117] "RemoveContainer" containerID="5d000fc8a1aa09c01761c498bcab7d820073e81aa8303adc0b1d8f651eee2971"
Sep 10 15:28:22.481332 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:22.481313    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d000fc8a1aa09c01761c498bcab7d820073e81aa8303adc0b1d8f651eee2971\": container with ID starting with 5d000fc8a1aa09c01761c498bcab7d820073e81aa8303adc0b1d8f651eee2971 not found: ID does not exist" containerID="5d000fc8a1aa09c01761c498bcab7d820073e81aa8303adc0b1d8f651eee2971"
Sep 10 15:28:22.481385 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.481339    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d000fc8a1aa09c01761c498bcab7d820073e81aa8303adc0b1d8f651eee2971"} err="failed to get container status \"5d000fc8a1aa09c01761c498bcab7d820073e81aa8303adc0b1d8f651eee2971\": rpc error: code = NotFound desc = could not find container \"5d000fc8a1aa09c01761c498bcab7d820073e81aa8303adc0b1d8f651eee2971\": container with ID starting with 5d000fc8a1aa09c01761c498bcab7d820073e81aa8303adc0b1d8f651eee2971 not found: ID does not exist"
Sep 10 15:28:22.511915 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.511869    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"]
Sep 10 15:28:22.521619 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.521590    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b74e71b-56db44b594-chzl6"]
Sep 10 15:28:22.541915 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.541880    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"]
Sep 10 15:28:22.545855 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:22.545831    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b74e71b-fc57f5f94-2dxd5"]
Sep 10 15:28:22.962011 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:22.961976    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d44972_7a24_4f89_9709_b193bef1163c.slice/crio-0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d\": RecentStats: unable to find data in memory cache]"
Sep 10 15:28:24.126077 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:24.126046    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b38a64dd-a0c9-4247-a591-6e969faad94c" path="/var/lib/kubelet/pods/b38a64dd-a0c9-4247-a591-6e969faad94c/volumes"
Sep 10 15:28:24.126425 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:24.126376    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd991ca2-2d30-427c-8ea5-9c44cbd85543" path="/var/lib/kubelet/pods/bd991ca2-2d30-427c-8ea5-9c44cbd85543/volumes"
Sep 10 15:28:26.170487 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.170454    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"]
Sep 10 15:28:26.171590 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.171565    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd991ca2-2d30-427c-8ea5-9c44cbd85543" containerName="payload-pre-processing"
Sep 10 15:28:26.171732 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.171719    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd991ca2-2d30-427c-8ea5-9c44cbd85543" containerName="payload-pre-processing"
Sep 10 15:28:26.171853 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.171840    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b38a64dd-a0c9-4247-a591-6e969faad94c" containerName="maas-api"
Sep 10 15:28:26.171948 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.171938    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b38a64dd-a0c9-4247-a591-6e969faad94c" containerName="maas-api"
Sep 10 15:28:26.172166 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.172151    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b38a64dd-a0c9-4247-a591-6e969faad94c" containerName="maas-api"
Sep 10 15:28:26.172271 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.172260    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd991ca2-2d30-427c-8ea5-9c44cbd85543" containerName="payload-pre-processing"
Sep 10 15:28:26.177695 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.177673    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.180515 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.180488    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8d054c9d-serving-cert\""
Sep 10 15:28:26.189271 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.188621    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"]
Sep 10 15:28:26.249744 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.249706    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rvmm\" (UniqueName: \"kubernetes.io/projected/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-kube-api-access-8rvmm\") pod \"maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.249929 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.249751    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-maas-api-tls\") pod \"maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.249929 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.249846    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-openshift-service-ca\") pod \"maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.249929 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.249916    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-metrics-certs\") pod \"maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.350953 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.350912    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-metrics-certs\") pod \"maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.351122 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.350969    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvmm\" (UniqueName: \"kubernetes.io/projected/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-kube-api-access-8rvmm\") pod \"maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.351122 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.350992    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-maas-api-tls\") pod \"maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.351122 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.351025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-openshift-service-ca\") pod \"maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.351227 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:26.351146    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-serving-cert: secret "maas-api-e2e-migrate-8d054c9d-serving-cert" not found
Sep 10 15:28:26.351227 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:26.351219    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-maas-api-tls podName:5db3de3a-569a-46ad-8f25-4c9ead89ac3c nodeName:}" failed. No retries permitted until 2026-09-10 15:28:26.851202279 +0000 UTC m=+769.266998076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-maas-api-tls") pod "maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759" (UID: "5db3de3a-569a-46ad-8f25-4c9ead89ac3c") : secret "maas-api-e2e-migrate-8d054c9d-serving-cert" not found
Sep 10 15:28:26.351813 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.351762    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-openshift-service-ca\") pod \"maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.353576 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.353550    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-metrics-certs\") pod \"maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.360331 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.360310    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvmm\" (UniqueName: \"kubernetes.io/projected/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-kube-api-access-8rvmm\") pod \"maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.854897 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.854860    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-maas-api-tls\") pod \"maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.857462 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.857431    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-maas-api-tls\") pod \"maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:26.908001 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.907958    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"]
Sep 10 15:28:26.911747 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.911724    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"
Sep 10 15:28:26.913970 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.913948    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8d054c9d\""
Sep 10 15:28:26.914399 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.914385    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8d054c9d-dockercfg-95tlk\""
Sep 10 15:28:26.924940 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.924916    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"]
Sep 10 15:28:26.955239 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.955207    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7src8\" (UniqueName: \"kubernetes.io/projected/432f246a-4726-4e25-945d-d660dfab3c23-kube-api-access-7src8\") pod \"payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg\" (UID: \"432f246a-4726-4e25-945d-d660dfab3c23\") " pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"
Sep 10 15:28:26.955239 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:26.955239    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432f246a-4726-4e25-945d-d660dfab3c23-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg\" (UID: \"432f246a-4726-4e25-945d-d660dfab3c23\") " pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"
Sep 10 15:28:27.056257 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:27.056203    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7src8\" (UniqueName: \"kubernetes.io/projected/432f246a-4726-4e25-945d-d660dfab3c23-kube-api-access-7src8\") pod \"payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg\" (UID: \"432f246a-4726-4e25-945d-d660dfab3c23\") " pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"
Sep 10 15:28:27.056414 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:27.056262    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432f246a-4726-4e25-945d-d660dfab3c23-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg\" (UID: \"432f246a-4726-4e25-945d-d660dfab3c23\") " pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"
Sep 10 15:28:27.057000 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:27.056983    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432f246a-4726-4e25-945d-d660dfab3c23-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg\" (UID: \"432f246a-4726-4e25-945d-d660dfab3c23\") " pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"
Sep 10 15:28:27.065167 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:27.065130    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7src8\" (UniqueName: \"kubernetes.io/projected/432f246a-4726-4e25-945d-d660dfab3c23-kube-api-access-7src8\") pod \"payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg\" (UID: \"432f246a-4726-4e25-945d-d660dfab3c23\") " pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"
Sep 10 15:28:27.091952 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:27.091915    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:27.222331 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:27.222293    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"
Sep 10 15:28:27.365977 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:27.365889    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"]
Sep 10 15:28:27.492617 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:27.492539    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759" event={"ID":"5db3de3a-569a-46ad-8f25-4c9ead89ac3c","Type":"ContainerStarted","Data":"b628086c5f00ce8ae284b423e754553c28542a5f0b32f6450245fde4da9a36a8"}
Sep 10 15:28:27.501924 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:27.501896    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"]
Sep 10 15:28:27.504659 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:28:27.504636    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432f246a_4726_4e25_945d_d660dfab3c23.slice/crio-f68bb2830a0e4e6c3159ee0599ecba3d6cb46e682e49b0a0314686331753b032 WatchSource:0}: Error finding container f68bb2830a0e4e6c3159ee0599ecba3d6cb46e682e49b0a0314686331753b032: Status 404 returned error can't find the container with id f68bb2830a0e4e6c3159ee0599ecba3d6cb46e682e49b0a0314686331753b032
Sep 10 15:28:28.499755 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:28.499715    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg" event={"ID":"432f246a-4726-4e25-945d-d660dfab3c23","Type":"ContainerStarted","Data":"fd51451e02c6e12516882eea3f06ac7ac480187741dc337db84131637896f924"}
Sep 10 15:28:28.500232 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:28.499772    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg" event={"ID":"432f246a-4726-4e25-945d-d660dfab3c23","Type":"ContainerStarted","Data":"f68bb2830a0e4e6c3159ee0599ecba3d6cb46e682e49b0a0314686331753b032"}
Sep 10 15:28:28.500232 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:28.499846    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"
Sep 10 15:28:28.501421 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:28.501392    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759" event={"ID":"5db3de3a-569a-46ad-8f25-4c9ead89ac3c","Type":"ContainerStarted","Data":"dc059264ea2623eb5a5de2fd67472091c46d12e09ce7420ff1b9a9c7c9d958ba"}
Sep 10 15:28:28.501560 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:28.501515    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:28.521339 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:28.521286    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg" podStartSLOduration=2.521266451 podStartE2EDuration="2.521266451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:28.519313979 +0000 UTC m=+770.935109798" watchObservedRunningTime="2026-09-10 15:28:28.521266451 +0000 UTC m=+770.937062274"
Sep 10 15:28:28.540423 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:28.540371    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759" podStartSLOduration=2.238288783 podStartE2EDuration="2.540356954s" totalImagesPullingTime="302.068171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:28.538731826 +0000 UTC m=+770.954527682" watchObservedRunningTime="2026-09-10 15:28:28.540356954 +0000 UTC m=+770.956152773"
Sep 10 15:28:30.059524 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:30.059487    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d44972_7a24_4f89_9709_b193bef1163c.slice/crio-0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d\": RecentStats: unable to find data in memory cache]"
Sep 10 15:28:34.512453 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:34.512419    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:37.990102 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:37.990069    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d44972_7a24_4f89_9709_b193bef1163c.slice/crio-0400805b58a87652a8e633bb0100ec74c2f66a36b01744b70a0283b7ba40865d\": RecentStats: unable to find data in memory cache]"
Sep 10 15:28:39.509559 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:39.509525    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"
Sep 10 15:28:42.452727 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.452696    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"]
Sep 10 15:28:42.457835 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.457815    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.460164 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.460139    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2e2b69\"/\"kube-root-ca.crt\""
Sep 10 15:28:42.460280 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.460147    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2e2b69\"/\"openshift-service-ca.crt\""
Sep 10 15:28:42.460828 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.460812    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2e2b69\"/\"autores-model-2e2b69-kserve-self-signed-certs\""
Sep 10 15:28:42.460881 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.460814    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2e2b69\"/\"default-dockercfg-n42m6\""
Sep 10 15:28:42.466577 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.466554    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"]
Sep 10 15:28:42.495115 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.495081    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw8g5\" (UniqueName: \"kubernetes.io/projected/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-kube-api-access-xw8g5\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.495339 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.495159    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-dshm\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.495339 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.495186    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-home\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.495339 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.495207    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-tmp-dir\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.495339 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.495263    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-model-cache\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.495339 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.495318    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-tls-certs\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.595767 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.595729    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-model-cache\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.595966 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.595831    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-tls-certs\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.595966 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.595870    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw8g5\" (UniqueName: \"kubernetes.io/projected/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-kube-api-access-xw8g5\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.595966 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.595941    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-dshm\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.596112 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.595965    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-home\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.596112 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.595988    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-tmp-dir\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.596265 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.596242    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-model-cache\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.596366 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.596349    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-tmp-dir\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.596421 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.596354    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-home\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.598395 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.598368    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-dshm\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.598522 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.598507    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-tls-certs\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.604772 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.604746    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw8g5\" (UniqueName: \"kubernetes.io/projected/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-kube-api-access-xw8g5\") pod \"autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") " pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:42.766954 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:42.766861    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:43.047022 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:43.046996    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"]
Sep 10 15:28:43.049597 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:28:43.049567    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda639e0ee_af61_4b30_9c7c_03e8b3226a8f.slice/crio-397f7919b9b8700f28432ae77ad9090b0907c27ff81bc93f771b1fb6ed181c30 WatchSource:0}: Error finding container 397f7919b9b8700f28432ae77ad9090b0907c27ff81bc93f771b1fb6ed181c30: Status 404 returned error can't find the container with id 397f7919b9b8700f28432ae77ad9090b0907c27ff81bc93f771b1fb6ed181c30
Sep 10 15:28:43.591437 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:43.591398    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4" event={"ID":"a639e0ee-af61-4b30-9c7c-03e8b3226a8f","Type":"ContainerStarted","Data":"6da1c2eff02489ec9a079010ab1303fa2db0f1c65c15d94d1c0fc2da931cb0de"}
Sep 10 15:28:43.591437 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:43.591439    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4" event={"ID":"a639e0ee-af61-4b30-9c7c-03e8b3226a8f","Type":"ContainerStarted","Data":"397f7919b9b8700f28432ae77ad9090b0907c27ff81bc93f771b1fb6ed181c30"}
Sep 10 15:28:43.610509 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:43.610458    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4" podStartSLOduration=1.610442581 podStartE2EDuration="1.610442581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:43.609547917 +0000 UTC m=+786.025343737" watchObservedRunningTime="2026-09-10 15:28:43.610442581 +0000 UTC m=+786.026238400"
Sep 10 15:28:52.767450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:52.767416    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:52.767450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:52.767460    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:52.781086 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:52.781062    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:53.658142 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:53.658109    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:28:57.775757 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:57.775720    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"]
Sep 10 15:28:57.776176 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:57.775997    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759" podUID="5db3de3a-569a-46ad-8f25-4c9ead89ac3c" containerName="maas-api" containerID="cri-o://dc059264ea2623eb5a5de2fd67472091c46d12e09ce7420ff1b9a9c7c9d958ba" gracePeriod=30
Sep 10 15:28:57.845410 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:57.845373    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"]
Sep 10 15:28:57.845616 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:57.845586    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg" podUID="432f246a-4726-4e25-945d-d660dfab3c23" containerName="payload-processing" containerID="cri-o://fd51451e02c6e12516882eea3f06ac7ac480187741dc337db84131637896f924" gracePeriod=30
Sep 10 15:28:58.110600 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.110575    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:58.142778 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.142753    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"
Sep 10 15:28:58.235580 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.235542    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-metrics-certs\") pod \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") "
Sep 10 15:28:58.235774 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.235592    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432f246a-4726-4e25-945d-d660dfab3c23-plugins-config-volume\") pod \"432f246a-4726-4e25-945d-d660dfab3c23\" (UID: \"432f246a-4726-4e25-945d-d660dfab3c23\") "
Sep 10 15:28:58.235774 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.235650    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-maas-api-tls\") pod \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") "
Sep 10 15:28:58.235774 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.235690    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-openshift-service-ca\") pod \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") "
Sep 10 15:28:58.235774 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.235744    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rvmm\" (UniqueName: \"kubernetes.io/projected/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-kube-api-access-8rvmm\") pod \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\" (UID: \"5db3de3a-569a-46ad-8f25-4c9ead89ac3c\") "
Sep 10 15:28:58.236032 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.235780    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7src8\" (UniqueName: \"kubernetes.io/projected/432f246a-4726-4e25-945d-d660dfab3c23-kube-api-access-7src8\") pod \"432f246a-4726-4e25-945d-d660dfab3c23\" (UID: \"432f246a-4726-4e25-945d-d660dfab3c23\") "
Sep 10 15:28:58.236032 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.236009    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432f246a-4726-4e25-945d-d660dfab3c23-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "432f246a-4726-4e25-945d-d660dfab3c23" (UID: "432f246a-4726-4e25-945d-d660dfab3c23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:28:58.236212 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.236126    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5db3de3a-569a-46ad-8f25-4c9ead89ac3c" (UID: "5db3de3a-569a-46ad-8f25-4c9ead89ac3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:28:58.236620 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.236288    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432f246a-4726-4e25-945d-d660dfab3c23-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:28:58.237975 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.237955    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5db3de3a-569a-46ad-8f25-4c9ead89ac3c" (UID: "5db3de3a-569a-46ad-8f25-4c9ead89ac3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:28:58.238041 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.237989    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432f246a-4726-4e25-945d-d660dfab3c23-kube-api-access-7src8" (OuterVolumeSpecName: "kube-api-access-7src8") pod "432f246a-4726-4e25-945d-d660dfab3c23" (UID: "432f246a-4726-4e25-945d-d660dfab3c23"). InnerVolumeSpecName "kube-api-access-7src8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:28:58.238399 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.238374    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5db3de3a-569a-46ad-8f25-4c9ead89ac3c" (UID: "5db3de3a-569a-46ad-8f25-4c9ead89ac3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:28:58.238399 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.238386    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-kube-api-access-8rvmm" (OuterVolumeSpecName: "kube-api-access-8rvmm") pod "5db3de3a-569a-46ad-8f25-4c9ead89ac3c" (UID: "5db3de3a-569a-46ad-8f25-4c9ead89ac3c"). InnerVolumeSpecName "kube-api-access-8rvmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:28:58.337494 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.337459    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:28:58.337494 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.337493    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:28:58.337678 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.337502    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:28:58.337678 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.337511    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rvmm\" (UniqueName: \"kubernetes.io/projected/5db3de3a-569a-46ad-8f25-4c9ead89ac3c-kube-api-access-8rvmm\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:28:58.337678 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.337520    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7src8\" (UniqueName: \"kubernetes.io/projected/432f246a-4726-4e25-945d-d660dfab3c23-kube-api-access-7src8\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:28:58.674244 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.674156    2573 generic.go:356] "Generic (PLEG): container finished" podID="432f246a-4726-4e25-945d-d660dfab3c23" containerID="fd51451e02c6e12516882eea3f06ac7ac480187741dc337db84131637896f924" exitCode=0
Sep 10 15:28:58.674244 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.674206    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"
Sep 10 15:28:58.674478 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.674247    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg" event={"ID":"432f246a-4726-4e25-945d-d660dfab3c23","Type":"ContainerDied","Data":"fd51451e02c6e12516882eea3f06ac7ac480187741dc337db84131637896f924"}
Sep 10 15:28:58.674478 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.674314    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg" event={"ID":"432f246a-4726-4e25-945d-d660dfab3c23","Type":"ContainerDied","Data":"f68bb2830a0e4e6c3159ee0599ecba3d6cb46e682e49b0a0314686331753b032"}
Sep 10 15:28:58.674478 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.674345    2573 scope.go:117] "RemoveContainer" containerID="fd51451e02c6e12516882eea3f06ac7ac480187741dc337db84131637896f924"
Sep 10 15:28:58.676568 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.676536    2573 generic.go:356] "Generic (PLEG): container finished" podID="5db3de3a-569a-46ad-8f25-4c9ead89ac3c" containerID="dc059264ea2623eb5a5de2fd67472091c46d12e09ce7420ff1b9a9c7c9d958ba" exitCode=0
Sep 10 15:28:58.676675 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.676587    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"
Sep 10 15:28:58.676675 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.676628    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759" event={"ID":"5db3de3a-569a-46ad-8f25-4c9ead89ac3c","Type":"ContainerDied","Data":"dc059264ea2623eb5a5de2fd67472091c46d12e09ce7420ff1b9a9c7c9d958ba"}
Sep 10 15:28:58.676675 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.676671    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759" event={"ID":"5db3de3a-569a-46ad-8f25-4c9ead89ac3c","Type":"ContainerDied","Data":"b628086c5f00ce8ae284b423e754553c28542a5f0b32f6450245fde4da9a36a8"}
Sep 10 15:28:58.684695 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.684677    2573 scope.go:117] "RemoveContainer" containerID="fd51451e02c6e12516882eea3f06ac7ac480187741dc337db84131637896f924"
Sep 10 15:28:58.685038 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:58.685012    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd51451e02c6e12516882eea3f06ac7ac480187741dc337db84131637896f924\": container with ID starting with fd51451e02c6e12516882eea3f06ac7ac480187741dc337db84131637896f924 not found: ID does not exist" containerID="fd51451e02c6e12516882eea3f06ac7ac480187741dc337db84131637896f924"
Sep 10 15:28:58.685287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.685050    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd51451e02c6e12516882eea3f06ac7ac480187741dc337db84131637896f924"} err="failed to get container status \"fd51451e02c6e12516882eea3f06ac7ac480187741dc337db84131637896f924\": rpc error: code = NotFound desc = could not find container \"fd51451e02c6e12516882eea3f06ac7ac480187741dc337db84131637896f924\": container with ID starting with fd51451e02c6e12516882eea3f06ac7ac480187741dc337db84131637896f924 not found: ID does not exist"
Sep 10 15:28:58.685287 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.685075    2573 scope.go:117] "RemoveContainer" containerID="dc059264ea2623eb5a5de2fd67472091c46d12e09ce7420ff1b9a9c7c9d958ba"
Sep 10 15:28:58.694119 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.694101    2573 scope.go:117] "RemoveContainer" containerID="dc059264ea2623eb5a5de2fd67472091c46d12e09ce7420ff1b9a9c7c9d958ba"
Sep 10 15:28:58.694362 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:28:58.694344    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc059264ea2623eb5a5de2fd67472091c46d12e09ce7420ff1b9a9c7c9d958ba\": container with ID starting with dc059264ea2623eb5a5de2fd67472091c46d12e09ce7420ff1b9a9c7c9d958ba not found: ID does not exist" containerID="dc059264ea2623eb5a5de2fd67472091c46d12e09ce7420ff1b9a9c7c9d958ba"
Sep 10 15:28:58.694422 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.694371    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc059264ea2623eb5a5de2fd67472091c46d12e09ce7420ff1b9a9c7c9d958ba"} err="failed to get container status \"dc059264ea2623eb5a5de2fd67472091c46d12e09ce7420ff1b9a9c7c9d958ba\": rpc error: code = NotFound desc = could not find container \"dc059264ea2623eb5a5de2fd67472091c46d12e09ce7420ff1b9a9c7c9d958ba\": container with ID starting with dc059264ea2623eb5a5de2fd67472091c46d12e09ce7420ff1b9a9c7c9d958ba not found: ID does not exist"
Sep 10 15:28:58.702310 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.702289    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"]
Sep 10 15:28:58.706097 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.706077    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8d054c9d-cf5fdc8b6-gnkqg"]
Sep 10 15:28:58.717818 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.717776    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"]
Sep 10 15:28:58.720259 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:28:58.720241    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8d054c9d-589bc7bbd8-v4759"]
Sep 10 15:29:00.128163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:00.128124    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="432f246a-4726-4e25-945d-d660dfab3c23" path="/var/lib/kubelet/pods/432f246a-4726-4e25-945d-d660dfab3c23/volumes"
Sep 10 15:29:00.128568 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:00.128552    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5db3de3a-569a-46ad-8f25-4c9ead89ac3c" path="/var/lib/kubelet/pods/5db3de3a-569a-46ad-8f25-4c9ead89ac3c/volumes"
Sep 10 15:29:02.245581 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.245526    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"]
Sep 10 15:29:02.246100 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.246080    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432f246a-4726-4e25-945d-d660dfab3c23" containerName="payload-processing"
Sep 10 15:29:02.246100 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.246099    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="432f246a-4726-4e25-945d-d660dfab3c23" containerName="payload-processing"
Sep 10 15:29:02.246227 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.246118    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db3de3a-569a-46ad-8f25-4c9ead89ac3c" containerName="maas-api"
Sep 10 15:29:02.246227 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.246127    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db3de3a-569a-46ad-8f25-4c9ead89ac3c" containerName="maas-api"
Sep 10 15:29:02.246320 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.246234    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db3de3a-569a-46ad-8f25-4c9ead89ac3c" containerName="maas-api"
Sep 10 15:29:02.246320 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.246249    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="432f246a-4726-4e25-945d-d660dfab3c23" containerName="payload-processing"
Sep 10 15:29:02.249261 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.249237    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.251520 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.251499    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9fd15233-serving-cert\""
Sep 10 15:29:02.259503 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.259481    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"]
Sep 10 15:29:02.377860 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.377812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-schmx\" (UniqueName: \"kubernetes.io/projected/103011a9-c7ae-4a16-b078-8b8f3b926b5c-kube-api-access-schmx\") pod \"maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.378055 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.377937    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-maas-api-tls\") pod \"maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.378055 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.377993    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-metrics-certs\") pod \"maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.378154 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.378090    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103011a9-c7ae-4a16-b078-8b8f3b926b5c-openshift-service-ca\") pod \"maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.479273 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.479235    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-maas-api-tls\") pod \"maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.479448 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.479286    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-metrics-certs\") pod \"maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.479448 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.479330    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103011a9-c7ae-4a16-b078-8b8f3b926b5c-openshift-service-ca\") pod \"maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.479448 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.479385    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-schmx\" (UniqueName: \"kubernetes.io/projected/103011a9-c7ae-4a16-b078-8b8f3b926b5c-kube-api-access-schmx\") pod \"maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.479448 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:29:02.479393    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-serving-cert: secret "maas-api-e2e-ait-9fd15233-serving-cert" not found
Sep 10 15:29:02.479614 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:29:02.479472    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-maas-api-tls podName:103011a9-c7ae-4a16-b078-8b8f3b926b5c nodeName:}" failed. No retries permitted until 2026-09-10 15:29:02.979451244 +0000 UTC m=+805.395247058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-maas-api-tls") pod "maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9" (UID: "103011a9-c7ae-4a16-b078-8b8f3b926b5c") : secret "maas-api-e2e-ait-9fd15233-serving-cert" not found
Sep 10 15:29:02.480221 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.480143    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103011a9-c7ae-4a16-b078-8b8f3b926b5c-openshift-service-ca\") pod \"maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.482193 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.482164    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-metrics-certs\") pod \"maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.490774 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.490744    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-schmx\" (UniqueName: \"kubernetes.io/projected/103011a9-c7ae-4a16-b078-8b8f3b926b5c-kube-api-access-schmx\") pod \"maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.983467 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.983433    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-maas-api-tls\") pod \"maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.986195 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.986169    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-maas-api-tls\") pod \"maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:02.998551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:02.998524    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"]
Sep 10 15:29:03.002203 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.002183    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"
Sep 10 15:29:03.004474 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.004452    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9fd15233\""
Sep 10 15:29:03.004594 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.004506    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9fd15233-dockercfg-xd877\""
Sep 10 15:29:03.012858 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.012838    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"]
Sep 10 15:29:03.084156 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.084120    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sjlh\" (UniqueName: \"kubernetes.io/projected/27225269-7251-4c93-aa24-1b53e482a17e-kube-api-access-7sjlh\") pod \"payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t\" (UID: \"27225269-7251-4c93-aa24-1b53e482a17e\") " pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"
Sep 10 15:29:03.084156 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.084167    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27225269-7251-4c93-aa24-1b53e482a17e-plugins-config-volume\") pod \"payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t\" (UID: \"27225269-7251-4c93-aa24-1b53e482a17e\") " pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"
Sep 10 15:29:03.160592 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.160553    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:03.185431 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.185401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sjlh\" (UniqueName: \"kubernetes.io/projected/27225269-7251-4c93-aa24-1b53e482a17e-kube-api-access-7sjlh\") pod \"payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t\" (UID: \"27225269-7251-4c93-aa24-1b53e482a17e\") " pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"
Sep 10 15:29:03.185568 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.185474    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27225269-7251-4c93-aa24-1b53e482a17e-plugins-config-volume\") pod \"payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t\" (UID: \"27225269-7251-4c93-aa24-1b53e482a17e\") " pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"
Sep 10 15:29:03.186315 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.186269    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27225269-7251-4c93-aa24-1b53e482a17e-plugins-config-volume\") pod \"payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t\" (UID: \"27225269-7251-4c93-aa24-1b53e482a17e\") " pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"
Sep 10 15:29:03.194634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.194608    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sjlh\" (UniqueName: \"kubernetes.io/projected/27225269-7251-4c93-aa24-1b53e482a17e-kube-api-access-7sjlh\") pod \"payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t\" (UID: \"27225269-7251-4c93-aa24-1b53e482a17e\") " pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"
Sep 10 15:29:03.313561 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.313473    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"
Sep 10 15:29:03.444514 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.444489    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"]
Sep 10 15:29:03.446504 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:29:03.446476    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103011a9_c7ae_4a16_b078_8b8f3b926b5c.slice/crio-5f0d52c99383043531956f5f655a31376e9072be59704ebaf409a2226245bc09 WatchSource:0}: Error finding container 5f0d52c99383043531956f5f655a31376e9072be59704ebaf409a2226245bc09: Status 404 returned error can't find the container with id 5f0d52c99383043531956f5f655a31376e9072be59704ebaf409a2226245bc09
Sep 10 15:29:03.591205 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.591177    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"]
Sep 10 15:29:03.593673 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:29:03.593640    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27225269_7251_4c93_aa24_1b53e482a17e.slice/crio-14bb185c1b4013d3720f6e304a1c80c54c295e9fea503fb6108c9408e000cdbe WatchSource:0}: Error finding container 14bb185c1b4013d3720f6e304a1c80c54c295e9fea503fb6108c9408e000cdbe: Status 404 returned error can't find the container with id 14bb185c1b4013d3720f6e304a1c80c54c295e9fea503fb6108c9408e000cdbe
Sep 10 15:29:03.704295 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.704249    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9" event={"ID":"103011a9-c7ae-4a16-b078-8b8f3b926b5c","Type":"ContainerStarted","Data":"5f0d52c99383043531956f5f655a31376e9072be59704ebaf409a2226245bc09"}
Sep 10 15:29:03.705995 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.705966    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t" event={"ID":"27225269-7251-4c93-aa24-1b53e482a17e","Type":"ContainerStarted","Data":"dd77e8baad7789fa28386be18a18e452557b155d9f1a46110bdc4f60d07db77c"}
Sep 10 15:29:03.706148 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.706016    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t" event={"ID":"27225269-7251-4c93-aa24-1b53e482a17e","Type":"ContainerStarted","Data":"14bb185c1b4013d3720f6e304a1c80c54c295e9fea503fb6108c9408e000cdbe"}
Sep 10 15:29:03.706148 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.706098    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"
Sep 10 15:29:03.729205 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:03.729143    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t" podStartSLOduration=1.729127306 podStartE2EDuration="1.729127306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:29:03.727615171 +0000 UTC m=+806.143410990" watchObservedRunningTime="2026-09-10 15:29:03.729127306 +0000 UTC m=+806.144923124"
Sep 10 15:29:04.712844 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:04.712782    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9" event={"ID":"103011a9-c7ae-4a16-b078-8b8f3b926b5c","Type":"ContainerStarted","Data":"f45a2d6c41383664ca7007cda23c023b16558502fdb82075c0edb9e12a629cbc"}
Sep 10 15:29:04.713226 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:04.712951    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:04.746174 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:04.746119    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9" podStartSLOduration=2.414802492 podStartE2EDuration="2.746104481s" totalImagesPullingTime="331.301989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:29:04.744755604 +0000 UTC m=+807.160551434" watchObservedRunningTime="2026-09-10 15:29:04.746104481 +0000 UTC m=+807.161900323"
Sep 10 15:29:10.722560 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:10.722530    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:14.715538 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:14.715501    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"
Sep 10 15:29:28.684683 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:28.684643    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"]
Sep 10 15:29:28.685157 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:28.684914    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st" podUID="4a5ef197-45a1-45bf-9f96-356f16904af1" containerName="maas-api" containerID="cri-o://be4d1988c34eeb91ec09e52375f45981d10f2720c591e0dbe1a1b8a9fc20a073" gracePeriod=30
Sep 10 15:29:28.772369 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:28.771008    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"]
Sep 10 15:29:28.772369 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:28.771265    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl" podUID="56a611a3-834d-47c2-859b-c95fe6ba59b4" containerName="payload-processing" containerID="cri-o://351850e6d78eaf00d14d74099f6842923b5b6acd621657c1b44d939526341b51" gracePeriod=30
Sep 10 15:29:28.841856 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:28.841811    2573 generic.go:356] "Generic (PLEG): container finished" podID="4a5ef197-45a1-45bf-9f96-356f16904af1" containerID="be4d1988c34eeb91ec09e52375f45981d10f2720c591e0dbe1a1b8a9fc20a073" exitCode=0
Sep 10 15:29:28.841856 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:28.841828    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st" event={"ID":"4a5ef197-45a1-45bf-9f96-356f16904af1","Type":"ContainerDied","Data":"be4d1988c34eeb91ec09e52375f45981d10f2720c591e0dbe1a1b8a9fc20a073"}
Sep 10 15:29:29.036091 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.036064    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:29:29.075636 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.075607    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"
Sep 10 15:29:29.093187 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.093152    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-metrics-certs\") pod \"4a5ef197-45a1-45bf-9f96-356f16904af1\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") "
Sep 10 15:29:29.093349 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.093193    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a611a3-834d-47c2-859b-c95fe6ba59b4-plugins-config-volume\") pod \"56a611a3-834d-47c2-859b-c95fe6ba59b4\" (UID: \"56a611a3-834d-47c2-859b-c95fe6ba59b4\") "
Sep 10 15:29:29.093349 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.093218    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bk87n\" (UniqueName: \"kubernetes.io/projected/56a611a3-834d-47c2-859b-c95fe6ba59b4-kube-api-access-bk87n\") pod \"56a611a3-834d-47c2-859b-c95fe6ba59b4\" (UID: \"56a611a3-834d-47c2-859b-c95fe6ba59b4\") "
Sep 10 15:29:29.093349 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.093240    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5ef197-45a1-45bf-9f96-356f16904af1-openshift-service-ca\") pod \"4a5ef197-45a1-45bf-9f96-356f16904af1\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") "
Sep 10 15:29:29.093518 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.093312    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-859dg\" (UniqueName: \"kubernetes.io/projected/4a5ef197-45a1-45bf-9f96-356f16904af1-kube-api-access-859dg\") pod \"4a5ef197-45a1-45bf-9f96-356f16904af1\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") "
Sep 10 15:29:29.093574 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.093529    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-maas-api-tls\") pod \"4a5ef197-45a1-45bf-9f96-356f16904af1\" (UID: \"4a5ef197-45a1-45bf-9f96-356f16904af1\") "
Sep 10 15:29:29.093714 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.093663    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5ef197-45a1-45bf-9f96-356f16904af1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a5ef197-45a1-45bf-9f96-356f16904af1" (UID: "4a5ef197-45a1-45bf-9f96-356f16904af1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:29:29.093938 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.093916    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5ef197-45a1-45bf-9f96-356f16904af1-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:29.094277 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.094254    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a611a3-834d-47c2-859b-c95fe6ba59b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56a611a3-834d-47c2-859b-c95fe6ba59b4" (UID: "56a611a3-834d-47c2-859b-c95fe6ba59b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:29:29.096203 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.096176    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5ef197-45a1-45bf-9f96-356f16904af1-kube-api-access-859dg" (OuterVolumeSpecName: "kube-api-access-859dg") pod "4a5ef197-45a1-45bf-9f96-356f16904af1" (UID: "4a5ef197-45a1-45bf-9f96-356f16904af1"). InnerVolumeSpecName "kube-api-access-859dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:29:29.096371 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.096350    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a5ef197-45a1-45bf-9f96-356f16904af1" (UID: "4a5ef197-45a1-45bf-9f96-356f16904af1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:29:29.096437 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.096378    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4a5ef197-45a1-45bf-9f96-356f16904af1" (UID: "4a5ef197-45a1-45bf-9f96-356f16904af1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:29:29.096700 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.096671    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a611a3-834d-47c2-859b-c95fe6ba59b4-kube-api-access-bk87n" (OuterVolumeSpecName: "kube-api-access-bk87n") pod "56a611a3-834d-47c2-859b-c95fe6ba59b4" (UID: "56a611a3-834d-47c2-859b-c95fe6ba59b4"). InnerVolumeSpecName "kube-api-access-bk87n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:29:29.194927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.194843    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-859dg\" (UniqueName: \"kubernetes.io/projected/4a5ef197-45a1-45bf-9f96-356f16904af1-kube-api-access-859dg\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:29.194927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.194872    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:29.194927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.194884    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5ef197-45a1-45bf-9f96-356f16904af1-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:29.194927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.194892    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a611a3-834d-47c2-859b-c95fe6ba59b4-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:29.194927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.194901    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk87n\" (UniqueName: \"kubernetes.io/projected/56a611a3-834d-47c2-859b-c95fe6ba59b4-kube-api-access-bk87n\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:29.848845 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.848802    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"
Sep 10 15:29:29.848845 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.848813    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st" event={"ID":"4a5ef197-45a1-45bf-9f96-356f16904af1","Type":"ContainerDied","Data":"88623889b60499dc4b8a943324f706cf8b1691bc13fbd278369d75cbf2dd8c4f"}
Sep 10 15:29:29.849383 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.848869    2573 scope.go:117] "RemoveContainer" containerID="be4d1988c34eeb91ec09e52375f45981d10f2720c591e0dbe1a1b8a9fc20a073"
Sep 10 15:29:29.850267 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.850200    2573 generic.go:356] "Generic (PLEG): container finished" podID="56a611a3-834d-47c2-859b-c95fe6ba59b4" containerID="351850e6d78eaf00d14d74099f6842923b5b6acd621657c1b44d939526341b51" exitCode=0
Sep 10 15:29:29.850267 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.850249    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"
Sep 10 15:29:29.850402 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.850254    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl" event={"ID":"56a611a3-834d-47c2-859b-c95fe6ba59b4","Type":"ContainerDied","Data":"351850e6d78eaf00d14d74099f6842923b5b6acd621657c1b44d939526341b51"}
Sep 10 15:29:29.850402 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.850365    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl" event={"ID":"56a611a3-834d-47c2-859b-c95fe6ba59b4","Type":"ContainerDied","Data":"41a33af606199c9a182f7d2faeae45fd68d531a4112c9a5e998b5aaf649a0f10"}
Sep 10 15:29:29.858337 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.858318    2573 scope.go:117] "RemoveContainer" containerID="351850e6d78eaf00d14d74099f6842923b5b6acd621657c1b44d939526341b51"
Sep 10 15:29:29.866639 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.866622    2573 scope.go:117] "RemoveContainer" containerID="351850e6d78eaf00d14d74099f6842923b5b6acd621657c1b44d939526341b51"
Sep 10 15:29:29.866925 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:29:29.866905    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"351850e6d78eaf00d14d74099f6842923b5b6acd621657c1b44d939526341b51\": container with ID starting with 351850e6d78eaf00d14d74099f6842923b5b6acd621657c1b44d939526341b51 not found: ID does not exist" containerID="351850e6d78eaf00d14d74099f6842923b5b6acd621657c1b44d939526341b51"
Sep 10 15:29:29.866988 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.866933    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"351850e6d78eaf00d14d74099f6842923b5b6acd621657c1b44d939526341b51"} err="failed to get container status \"351850e6d78eaf00d14d74099f6842923b5b6acd621657c1b44d939526341b51\": rpc error: code = NotFound desc = could not find container \"351850e6d78eaf00d14d74099f6842923b5b6acd621657c1b44d939526341b51\": container with ID starting with 351850e6d78eaf00d14d74099f6842923b5b6acd621657c1b44d939526341b51 not found: ID does not exist"
Sep 10 15:29:29.871368 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.871333    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"]
Sep 10 15:29:29.877672 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.877653    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2e2b69-6d58f4665c-lv8st"]
Sep 10 15:29:29.888072 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.888053    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"]
Sep 10 15:29:29.892074 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:29.892054    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2e2b69-6b889cdb45-9psdl"]
Sep 10 15:29:30.126384 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:30.126352    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a5ef197-45a1-45bf-9f96-356f16904af1" path="/var/lib/kubelet/pods/4a5ef197-45a1-45bf-9f96-356f16904af1/volumes"
Sep 10 15:29:30.126744 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:30.126730    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56a611a3-834d-47c2-859b-c95fe6ba59b4" path="/var/lib/kubelet/pods/56a611a3-834d-47c2-859b-c95fe6ba59b4/volumes"
Sep 10 15:29:32.943558 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:32.943525    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"]
Sep 10 15:29:32.944034 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:32.943846    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9" podUID="103011a9-c7ae-4a16-b078-8b8f3b926b5c" containerName="maas-api" containerID="cri-o://f45a2d6c41383664ca7007cda23c023b16558502fdb82075c0edb9e12a629cbc" gracePeriod=30
Sep 10 15:29:33.006081 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.006046    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"]
Sep 10 15:29:33.006327 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.006296    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t" podUID="27225269-7251-4c93-aa24-1b53e482a17e" containerName="payload-processing" containerID="cri-o://dd77e8baad7789fa28386be18a18e452557b155d9f1a46110bdc4f60d07db77c" gracePeriod=30
Sep 10 15:29:33.256019 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.255994    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:33.294995 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.294970    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"
Sep 10 15:29:33.331112 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.331074    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-maas-api-tls\") pod \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") "
Sep 10 15:29:33.331277 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.331135    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-schmx\" (UniqueName: \"kubernetes.io/projected/103011a9-c7ae-4a16-b078-8b8f3b926b5c-kube-api-access-schmx\") pod \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") "
Sep 10 15:29:33.331277 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.331178    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-metrics-certs\") pod \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") "
Sep 10 15:29:33.331277 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.331211    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103011a9-c7ae-4a16-b078-8b8f3b926b5c-openshift-service-ca\") pod \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\" (UID: \"103011a9-c7ae-4a16-b078-8b8f3b926b5c\") "
Sep 10 15:29:33.331277 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.331267    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sjlh\" (UniqueName: \"kubernetes.io/projected/27225269-7251-4c93-aa24-1b53e482a17e-kube-api-access-7sjlh\") pod \"27225269-7251-4c93-aa24-1b53e482a17e\" (UID: \"27225269-7251-4c93-aa24-1b53e482a17e\") "
Sep 10 15:29:33.331505 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.331294    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27225269-7251-4c93-aa24-1b53e482a17e-plugins-config-volume\") pod \"27225269-7251-4c93-aa24-1b53e482a17e\" (UID: \"27225269-7251-4c93-aa24-1b53e482a17e\") "
Sep 10 15:29:33.331690 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.331660    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103011a9-c7ae-4a16-b078-8b8f3b926b5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "103011a9-c7ae-4a16-b078-8b8f3b926b5c" (UID: "103011a9-c7ae-4a16-b078-8b8f3b926b5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:29:33.331926 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.331746    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27225269-7251-4c93-aa24-1b53e482a17e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27225269-7251-4c93-aa24-1b53e482a17e" (UID: "27225269-7251-4c93-aa24-1b53e482a17e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:29:33.333570 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.333542    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27225269-7251-4c93-aa24-1b53e482a17e-kube-api-access-7sjlh" (OuterVolumeSpecName: "kube-api-access-7sjlh") pod "27225269-7251-4c93-aa24-1b53e482a17e" (UID: "27225269-7251-4c93-aa24-1b53e482a17e"). InnerVolumeSpecName "kube-api-access-7sjlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:29:33.333664 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.333567    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "103011a9-c7ae-4a16-b078-8b8f3b926b5c" (UID: "103011a9-c7ae-4a16-b078-8b8f3b926b5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:29:33.333664 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.333573    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103011a9-c7ae-4a16-b078-8b8f3b926b5c-kube-api-access-schmx" (OuterVolumeSpecName: "kube-api-access-schmx") pod "103011a9-c7ae-4a16-b078-8b8f3b926b5c" (UID: "103011a9-c7ae-4a16-b078-8b8f3b926b5c"). InnerVolumeSpecName "kube-api-access-schmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:29:33.333765 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.333746    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "103011a9-c7ae-4a16-b078-8b8f3b926b5c" (UID: "103011a9-c7ae-4a16-b078-8b8f3b926b5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:29:33.432330 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.432295    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103011a9-c7ae-4a16-b078-8b8f3b926b5c-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:33.432330 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.432326    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sjlh\" (UniqueName: \"kubernetes.io/projected/27225269-7251-4c93-aa24-1b53e482a17e-kube-api-access-7sjlh\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:33.432330 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.432336    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27225269-7251-4c93-aa24-1b53e482a17e-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:33.432574 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.432345    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:33.432574 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.432355    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-schmx\" (UniqueName: \"kubernetes.io/projected/103011a9-c7ae-4a16-b078-8b8f3b926b5c-kube-api-access-schmx\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:33.432574 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.432365    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103011a9-c7ae-4a16-b078-8b8f3b926b5c-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:33.880827 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.878320    2573 generic.go:356] "Generic (PLEG): container finished" podID="27225269-7251-4c93-aa24-1b53e482a17e" containerID="dd77e8baad7789fa28386be18a18e452557b155d9f1a46110bdc4f60d07db77c" exitCode=0
Sep 10 15:29:33.880827 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.878463    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t" event={"ID":"27225269-7251-4c93-aa24-1b53e482a17e","Type":"ContainerDied","Data":"dd77e8baad7789fa28386be18a18e452557b155d9f1a46110bdc4f60d07db77c"}
Sep 10 15:29:33.880827 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.878544    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t" event={"ID":"27225269-7251-4c93-aa24-1b53e482a17e","Type":"ContainerDied","Data":"14bb185c1b4013d3720f6e304a1c80c54c295e9fea503fb6108c9408e000cdbe"}
Sep 10 15:29:33.880827 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.878585    2573 scope.go:117] "RemoveContainer" containerID="dd77e8baad7789fa28386be18a18e452557b155d9f1a46110bdc4f60d07db77c"
Sep 10 15:29:33.880827 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.879214    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"
Sep 10 15:29:33.883190 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.883157    2573 generic.go:356] "Generic (PLEG): container finished" podID="103011a9-c7ae-4a16-b078-8b8f3b926b5c" containerID="f45a2d6c41383664ca7007cda23c023b16558502fdb82075c0edb9e12a629cbc" exitCode=0
Sep 10 15:29:33.883335 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.883201    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"
Sep 10 15:29:33.883335 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.883233    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9" event={"ID":"103011a9-c7ae-4a16-b078-8b8f3b926b5c","Type":"ContainerDied","Data":"f45a2d6c41383664ca7007cda23c023b16558502fdb82075c0edb9e12a629cbc"}
Sep 10 15:29:33.883335 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.883272    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9" event={"ID":"103011a9-c7ae-4a16-b078-8b8f3b926b5c","Type":"ContainerDied","Data":"5f0d52c99383043531956f5f655a31376e9072be59704ebaf409a2226245bc09"}
Sep 10 15:29:33.890591 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.890569    2573 scope.go:117] "RemoveContainer" containerID="dd77e8baad7789fa28386be18a18e452557b155d9f1a46110bdc4f60d07db77c"
Sep 10 15:29:33.890921 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:29:33.890897    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd77e8baad7789fa28386be18a18e452557b155d9f1a46110bdc4f60d07db77c\": container with ID starting with dd77e8baad7789fa28386be18a18e452557b155d9f1a46110bdc4f60d07db77c not found: ID does not exist" containerID="dd77e8baad7789fa28386be18a18e452557b155d9f1a46110bdc4f60d07db77c"
Sep 10 15:29:33.891010 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.890928    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd77e8baad7789fa28386be18a18e452557b155d9f1a46110bdc4f60d07db77c"} err="failed to get container status \"dd77e8baad7789fa28386be18a18e452557b155d9f1a46110bdc4f60d07db77c\": rpc error: code = NotFound desc = could not find container \"dd77e8baad7789fa28386be18a18e452557b155d9f1a46110bdc4f60d07db77c\": container with ID starting with dd77e8baad7789fa28386be18a18e452557b155d9f1a46110bdc4f60d07db77c not found: ID does not exist"
Sep 10 15:29:33.891010 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.890945    2573 scope.go:117] "RemoveContainer" containerID="f45a2d6c41383664ca7007cda23c023b16558502fdb82075c0edb9e12a629cbc"
Sep 10 15:29:33.899647 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.899629    2573 scope.go:117] "RemoveContainer" containerID="f45a2d6c41383664ca7007cda23c023b16558502fdb82075c0edb9e12a629cbc"
Sep 10 15:29:33.899937 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:29:33.899918    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45a2d6c41383664ca7007cda23c023b16558502fdb82075c0edb9e12a629cbc\": container with ID starting with f45a2d6c41383664ca7007cda23c023b16558502fdb82075c0edb9e12a629cbc not found: ID does not exist" containerID="f45a2d6c41383664ca7007cda23c023b16558502fdb82075c0edb9e12a629cbc"
Sep 10 15:29:33.899995 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.899944    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45a2d6c41383664ca7007cda23c023b16558502fdb82075c0edb9e12a629cbc"} err="failed to get container status \"f45a2d6c41383664ca7007cda23c023b16558502fdb82075c0edb9e12a629cbc\": rpc error: code = NotFound desc = could not find container \"f45a2d6c41383664ca7007cda23c023b16558502fdb82075c0edb9e12a629cbc\": container with ID starting with f45a2d6c41383664ca7007cda23c023b16558502fdb82075c0edb9e12a629cbc not found: ID does not exist"
Sep 10 15:29:33.906438 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.906414    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"]
Sep 10 15:29:33.910483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.910462    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9fd15233-fb9cc7c65-d5d9t"]
Sep 10 15:29:33.923408 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.923388    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"]
Sep 10 15:29:33.927321 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:33.927298    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9fd15233-588bf9dbb7-jzhh9"]
Sep 10 15:29:34.126744 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:34.126713    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103011a9-c7ae-4a16-b078-8b8f3b926b5c" path="/var/lib/kubelet/pods/103011a9-c7ae-4a16-b078-8b8f3b926b5c/volumes"
Sep 10 15:29:34.127111 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:34.127100    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27225269-7251-4c93-aa24-1b53e482a17e" path="/var/lib/kubelet/pods/27225269-7251-4c93-aa24-1b53e482a17e/volumes"
Sep 10 15:29:34.272032 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:34.271945    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"]
Sep 10 15:29:38.431866 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.431739    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"]
Sep 10 15:29:38.432452 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.432431    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56a611a3-834d-47c2-859b-c95fe6ba59b4" containerName="payload-processing"
Sep 10 15:29:38.432498 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.432456    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="56a611a3-834d-47c2-859b-c95fe6ba59b4" containerName="payload-processing"
Sep 10 15:29:38.432498 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.432479    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a5ef197-45a1-45bf-9f96-356f16904af1" containerName="maas-api"
Sep 10 15:29:38.432498 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.432489    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a5ef197-45a1-45bf-9f96-356f16904af1" containerName="maas-api"
Sep 10 15:29:38.432588 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.432498    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103011a9-c7ae-4a16-b078-8b8f3b926b5c" containerName="maas-api"
Sep 10 15:29:38.432588 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.432509    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="103011a9-c7ae-4a16-b078-8b8f3b926b5c" containerName="maas-api"
Sep 10 15:29:38.432588 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.432524    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27225269-7251-4c93-aa24-1b53e482a17e" containerName="payload-processing"
Sep 10 15:29:38.432588 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.432532    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="27225269-7251-4c93-aa24-1b53e482a17e" containerName="payload-processing"
Sep 10 15:29:38.432734 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.432620    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="27225269-7251-4c93-aa24-1b53e482a17e" containerName="payload-processing"
Sep 10 15:29:38.432734 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.432634    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="56a611a3-834d-47c2-859b-c95fe6ba59b4" containerName="payload-processing"
Sep 10 15:29:38.432734 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.432643    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a5ef197-45a1-45bf-9f96-356f16904af1" containerName="maas-api"
Sep 10 15:29:38.432734 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.432653    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="103011a9-c7ae-4a16-b078-8b8f3b926b5c" containerName="maas-api"
Sep 10 15:29:38.437977 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.437954    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:38.440057 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.440034    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f5d25c39-serving-cert\""
Sep 10 15:29:38.445727 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.445643    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"]
Sep 10 15:29:38.476066 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.476028    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-metrics-certs\") pod \"maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:38.476229 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.476089    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwxvn\" (UniqueName: \"kubernetes.io/projected/c49e3938-3b5c-46b6-aae9-d63251674e08-kube-api-access-bwxvn\") pod \"maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:38.476229 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.476150    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49e3938-3b5c-46b6-aae9-d63251674e08-openshift-service-ca\") pod \"maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:38.476338 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.476227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-maas-api-tls\") pod \"maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:38.577588 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.577551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49e3938-3b5c-46b6-aae9-d63251674e08-openshift-service-ca\") pod \"maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:38.577755 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.577602    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-maas-api-tls\") pod \"maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:38.577755 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.577670    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-metrics-certs\") pod \"maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:38.577923 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:29:38.577817    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-serving-cert: secret "maas-api-e2e-derive-f5d25c39-serving-cert" not found
Sep 10 15:29:38.577923 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.577871    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxvn\" (UniqueName: \"kubernetes.io/projected/c49e3938-3b5c-46b6-aae9-d63251674e08-kube-api-access-bwxvn\") pod \"maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:38.577923 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:29:38.577884    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-maas-api-tls podName:c49e3938-3b5c-46b6-aae9-d63251674e08 nodeName:}" failed. No retries permitted until 2026-09-10 15:29:39.077863491 +0000 UTC m=+841.493659288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-maas-api-tls") pod "maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv" (UID: "c49e3938-3b5c-46b6-aae9-d63251674e08") : secret "maas-api-e2e-derive-f5d25c39-serving-cert" not found
Sep 10 15:29:38.578562 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.578529    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49e3938-3b5c-46b6-aae9-d63251674e08-openshift-service-ca\") pod \"maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:38.580236 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.580216    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-metrics-certs\") pod \"maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:38.588496 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:38.588474    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwxvn\" (UniqueName: \"kubernetes.io/projected/c49e3938-3b5c-46b6-aae9-d63251674e08-kube-api-access-bwxvn\") pod \"maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:39.082604 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.082555    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-maas-api-tls\") pod \"maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:39.085368 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.085337    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-maas-api-tls\") pod \"maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:39.099772 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.099748    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"]
Sep 10 15:29:39.103255 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.103241    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"
Sep 10 15:29:39.105713 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.105690    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f5d25c39-dockercfg-mg6j5\""
Sep 10 15:29:39.105853 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.105740    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f5d25c39\""
Sep 10 15:29:39.114026 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.114003    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"]
Sep 10 15:29:39.183302 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.183271    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa185323-bead-4fb3-94a5-5e3fc6db0ae9-plugins-config-volume\") pod \"payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw\" (UID: \"aa185323-bead-4fb3-94a5-5e3fc6db0ae9\") " pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"
Sep 10 15:29:39.183502 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.183326    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6wv6\" (UniqueName: \"kubernetes.io/projected/aa185323-bead-4fb3-94a5-5e3fc6db0ae9-kube-api-access-g6wv6\") pod \"payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw\" (UID: \"aa185323-bead-4fb3-94a5-5e3fc6db0ae9\") " pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"
Sep 10 15:29:39.284245 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.284203    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6wv6\" (UniqueName: \"kubernetes.io/projected/aa185323-bead-4fb3-94a5-5e3fc6db0ae9-kube-api-access-g6wv6\") pod \"payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw\" (UID: \"aa185323-bead-4fb3-94a5-5e3fc6db0ae9\") " pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"
Sep 10 15:29:39.284416 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.284285    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa185323-bead-4fb3-94a5-5e3fc6db0ae9-plugins-config-volume\") pod \"payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw\" (UID: \"aa185323-bead-4fb3-94a5-5e3fc6db0ae9\") " pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"
Sep 10 15:29:39.284940 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.284920    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa185323-bead-4fb3-94a5-5e3fc6db0ae9-plugins-config-volume\") pod \"payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw\" (UID: \"aa185323-bead-4fb3-94a5-5e3fc6db0ae9\") " pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"
Sep 10 15:29:39.297599 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.297566    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6wv6\" (UniqueName: \"kubernetes.io/projected/aa185323-bead-4fb3-94a5-5e3fc6db0ae9-kube-api-access-g6wv6\") pod \"payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw\" (UID: \"aa185323-bead-4fb3-94a5-5e3fc6db0ae9\") " pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"
Sep 10 15:29:39.349684 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.349591    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:39.414206 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.414104    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"
Sep 10 15:29:39.627208 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.627185    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"]
Sep 10 15:29:39.629220 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:29:39.629191    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49e3938_3b5c_46b6_aae9_d63251674e08.slice/crio-fd74e11ce9c9f02da1d322026736df135c579c28cf53a1f6c3d70db0c9159064 WatchSource:0}: Error finding container fd74e11ce9c9f02da1d322026736df135c579c28cf53a1f6c3d70db0c9159064: Status 404 returned error can't find the container with id fd74e11ce9c9f02da1d322026736df135c579c28cf53a1f6c3d70db0c9159064
Sep 10 15:29:39.692022 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.691995    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"]
Sep 10 15:29:39.694447 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:29:39.694420    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa185323_bead_4fb3_94a5_5e3fc6db0ae9.slice/crio-30dfd4da2044c8ed5e0728eabaf49c8a6ce592981c9aa54bbef1d8802d39c44e WatchSource:0}: Error finding container 30dfd4da2044c8ed5e0728eabaf49c8a6ce592981c9aa54bbef1d8802d39c44e: Status 404 returned error can't find the container with id 30dfd4da2044c8ed5e0728eabaf49c8a6ce592981c9aa54bbef1d8802d39c44e
Sep 10 15:29:39.919385 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.919303    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv" event={"ID":"c49e3938-3b5c-46b6-aae9-d63251674e08","Type":"ContainerStarted","Data":"fd74e11ce9c9f02da1d322026736df135c579c28cf53a1f6c3d70db0c9159064"}
Sep 10 15:29:39.921107 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.921078    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw" event={"ID":"aa185323-bead-4fb3-94a5-5e3fc6db0ae9","Type":"ContainerStarted","Data":"15d36c54674a2b5697de7e4bfac838b35c4efde1b2f3977dd325251c2012568f"}
Sep 10 15:29:39.921264 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.921122    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw" event={"ID":"aa185323-bead-4fb3-94a5-5e3fc6db0ae9","Type":"ContainerStarted","Data":"30dfd4da2044c8ed5e0728eabaf49c8a6ce592981c9aa54bbef1d8802d39c44e"}
Sep 10 15:29:39.921264 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.921225    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"
Sep 10 15:29:39.940460 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:39.940396    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw" podStartSLOduration=0.940378422 podStartE2EDuration="940.378422ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:29:39.939148224 +0000 UTC m=+842.354944044" watchObservedRunningTime="2026-09-10 15:29:39.940378422 +0000 UTC m=+842.356174246"
Sep 10 15:29:40.927559 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:40.927516    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv" event={"ID":"c49e3938-3b5c-46b6-aae9-d63251674e08","Type":"ContainerStarted","Data":"beb09914759cd08bfaccfeaa74609274d1095c1606892a35e1e48ae486ad0fab"}
Sep 10 15:29:40.927996 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:40.927774    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:40.947213 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:40.947170    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv" podStartSLOduration=2.5292755639999998 podStartE2EDuration="2.947155489s" totalImagesPullingTime="417.879925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:29:40.94510745 +0000 UTC m=+843.360903281" watchObservedRunningTime="2026-09-10 15:29:40.947155489 +0000 UTC m=+843.362951309"
Sep 10 15:29:46.937469 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:46.937440    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:29:49.292894 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.292857    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4" podUID="a639e0ee-af61-4b30-9c7c-03e8b3226a8f" containerName="main" containerID="cri-o://6da1c2eff02489ec9a079010ab1303fa2db0f1c65c15d94d1c0fc2da931cb0de" gracePeriod=45
Sep 10 15:29:49.538081 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.538054    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:29:49.679810 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.679764    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-tmp-dir\") pod \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") "
Sep 10 15:29:49.680004 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.679825    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw8g5\" (UniqueName: \"kubernetes.io/projected/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-kube-api-access-xw8g5\") pod \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") "
Sep 10 15:29:49.680004 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.679842    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-dshm\") pod \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") "
Sep 10 15:29:49.680004 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.679872    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-model-cache\") pod \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") "
Sep 10 15:29:49.680004 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.679896    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-home\") pod \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") "
Sep 10 15:29:49.680004 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.679936    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-tls-certs\") pod \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\" (UID: \"a639e0ee-af61-4b30-9c7c-03e8b3226a8f\") "
Sep 10 15:29:49.680337 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.680096    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a639e0ee-af61-4b30-9c7c-03e8b3226a8f" (UID: "a639e0ee-af61-4b30-9c7c-03e8b3226a8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:29:49.680337 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.680137    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a639e0ee-af61-4b30-9c7c-03e8b3226a8f" (UID: "a639e0ee-af61-4b30-9c7c-03e8b3226a8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:29:49.680337 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.680187    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-home" (OuterVolumeSpecName: "home") pod "a639e0ee-af61-4b30-9c7c-03e8b3226a8f" (UID: "a639e0ee-af61-4b30-9c7c-03e8b3226a8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:29:49.682219 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.682192    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-dshm" (OuterVolumeSpecName: "dshm") pod "a639e0ee-af61-4b30-9c7c-03e8b3226a8f" (UID: "a639e0ee-af61-4b30-9c7c-03e8b3226a8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:29:49.682335 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.682257    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a639e0ee-af61-4b30-9c7c-03e8b3226a8f" (UID: "a639e0ee-af61-4b30-9c7c-03e8b3226a8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:29:49.682335 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.682286    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-kube-api-access-xw8g5" (OuterVolumeSpecName: "kube-api-access-xw8g5") pod "a639e0ee-af61-4b30-9c7c-03e8b3226a8f" (UID: "a639e0ee-af61-4b30-9c7c-03e8b3226a8f"). InnerVolumeSpecName "kube-api-access-xw8g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:29:49.780764 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.780730    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-home\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:49.780764 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.780758    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-tls-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:49.780764 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.780770    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-tmp-dir\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:49.781010 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.780778    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw8g5\" (UniqueName: \"kubernetes.io/projected/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-kube-api-access-xw8g5\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:49.781010 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.780804    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-dshm\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:49.781010 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.780813    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a639e0ee-af61-4b30-9c7c-03e8b3226a8f-model-cache\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:29:49.979620 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.979533    2573 generic.go:356] "Generic (PLEG): container finished" podID="a639e0ee-af61-4b30-9c7c-03e8b3226a8f" containerID="6da1c2eff02489ec9a079010ab1303fa2db0f1c65c15d94d1c0fc2da931cb0de" exitCode=0
Sep 10 15:29:49.979620 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.979568    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4" event={"ID":"a639e0ee-af61-4b30-9c7c-03e8b3226a8f","Type":"ContainerDied","Data":"6da1c2eff02489ec9a079010ab1303fa2db0f1c65c15d94d1c0fc2da931cb0de"}
Sep 10 15:29:49.979620 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.979587    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"
Sep 10 15:29:49.979620 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.979617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4" event={"ID":"a639e0ee-af61-4b30-9c7c-03e8b3226a8f","Type":"ContainerDied","Data":"397f7919b9b8700f28432ae77ad9090b0907c27ff81bc93f771b1fb6ed181c30"}
Sep 10 15:29:49.979936 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.979635    2573 scope.go:117] "RemoveContainer" containerID="6da1c2eff02489ec9a079010ab1303fa2db0f1c65c15d94d1c0fc2da931cb0de"
Sep 10 15:29:49.989100 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.989084    2573 scope.go:117] "RemoveContainer" containerID="6da1c2eff02489ec9a079010ab1303fa2db0f1c65c15d94d1c0fc2da931cb0de"
Sep 10 15:29:49.989392 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:29:49.989374    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da1c2eff02489ec9a079010ab1303fa2db0f1c65c15d94d1c0fc2da931cb0de\": container with ID starting with 6da1c2eff02489ec9a079010ab1303fa2db0f1c65c15d94d1c0fc2da931cb0de not found: ID does not exist" containerID="6da1c2eff02489ec9a079010ab1303fa2db0f1c65c15d94d1c0fc2da931cb0de"
Sep 10 15:29:49.989449 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:49.989403    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da1c2eff02489ec9a079010ab1303fa2db0f1c65c15d94d1c0fc2da931cb0de"} err="failed to get container status \"6da1c2eff02489ec9a079010ab1303fa2db0f1c65c15d94d1c0fc2da931cb0de\": rpc error: code = NotFound desc = could not find container \"6da1c2eff02489ec9a079010ab1303fa2db0f1c65c15d94d1c0fc2da931cb0de\": container with ID starting with 6da1c2eff02489ec9a079010ab1303fa2db0f1c65c15d94d1c0fc2da931cb0de not found: ID does not exist"
Sep 10 15:29:50.002042 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:50.002020    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"]
Sep 10 15:29:50.006639 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:50.006621    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-2e2b69/autores-model-2e2b69-kserve-5f5cdcf55f-c9hf4"]
Sep 10 15:29:50.132172 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:50.132139    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a639e0ee-af61-4b30-9c7c-03e8b3226a8f" path="/var/lib/kubelet/pods/a639e0ee-af61-4b30-9c7c-03e8b3226a8f/volumes"
Sep 10 15:29:50.930214 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:29:50.930179    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"
Sep 10 15:30:01.823296 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.823264    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"]
Sep 10 15:30:01.823879 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.823732    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a639e0ee-af61-4b30-9c7c-03e8b3226a8f" containerName="main"
Sep 10 15:30:01.823879 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.823751    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a639e0ee-af61-4b30-9c7c-03e8b3226a8f" containerName="main"
Sep 10 15:30:01.824031 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.823919    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a639e0ee-af61-4b30-9c7c-03e8b3226a8f" containerName="main"
Sep 10 15:30:01.828518 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.828497    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:01.831688 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.831661    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-229c42\"/\"openshift-service-ca.crt\""
Sep 10 15:30:01.831855 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.831691    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-229c42\"/\"default-dockercfg-gg2jc\""
Sep 10 15:30:01.831855 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.831715    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-229c42\"/\"test-model-229c42-kserve-self-signed-certs\""
Sep 10 15:30:01.831855 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.831662    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-229c42\"/\"kube-root-ca.crt\""
Sep 10 15:30:01.837920 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.837894    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"]
Sep 10 15:30:01.985319 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.985277    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb7wp\" (UniqueName: \"kubernetes.io/projected/54ddddad-710f-4251-a0a4-fd3f0b702f57-kube-api-access-lb7wp\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:01.985319 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.985323    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-home\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:01.985522 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.985341    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-dshm\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:01.985522 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.985458    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-model-cache\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:01.985522 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.985505    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-tmp-dir\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:01.985626 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:01.985544    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ddddad-710f-4251-a0a4-fd3f0b702f57-tls-certs\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.086324 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.086222    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-home\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.086324 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.086269    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-dshm\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.086551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.086333    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-model-cache\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.086551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.086379    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-tmp-dir\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.086551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.086425    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ddddad-710f-4251-a0a4-fd3f0b702f57-tls-certs\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.086551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.086507    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb7wp\" (UniqueName: \"kubernetes.io/projected/54ddddad-710f-4251-a0a4-fd3f0b702f57-kube-api-access-lb7wp\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.086768 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.086685    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-home\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.086853 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.086785    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-model-cache\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.086904 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.086855    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-tmp-dir\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.088777 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.088748    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-dshm\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.089079 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.089063    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ddddad-710f-4251-a0a4-fd3f0b702f57-tls-certs\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.095104 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.095079    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb7wp\" (UniqueName: \"kubernetes.io/projected/54ddddad-710f-4251-a0a4-fd3f0b702f57-kube-api-access-lb7wp\") pod \"test-model-229c42-kserve-5d9f98c985-kcp89\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") " pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.139575 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.139545    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:02.414615 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:02.414587    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"]
Sep 10 15:30:02.417112 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:30:02.417082    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54ddddad_710f_4251_a0a4_fd3f0b702f57.slice/crio-8ee7a01fa296c5487c2a5601a5c6859a0ca928e1bf8f175c65c961696c48eff2 WatchSource:0}: Error finding container 8ee7a01fa296c5487c2a5601a5c6859a0ca928e1bf8f175c65c961696c48eff2: Status 404 returned error can't find the container with id 8ee7a01fa296c5487c2a5601a5c6859a0ca928e1bf8f175c65c961696c48eff2
Sep 10 15:30:03.050376 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:03.050331    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89" event={"ID":"54ddddad-710f-4251-a0a4-fd3f0b702f57","Type":"ContainerStarted","Data":"2c902eff69f59874fff4b4090e8f8ace13250294d9f0a64880a73d74e7b56a38"}
Sep 10 15:30:03.050376 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:03.050383    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89" event={"ID":"54ddddad-710f-4251-a0a4-fd3f0b702f57","Type":"ContainerStarted","Data":"8ee7a01fa296c5487c2a5601a5c6859a0ca928e1bf8f175c65c961696c48eff2"}
Sep 10 15:30:03.077915 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:03.077863    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89" podStartSLOduration=2.077843313 podStartE2EDuration="2.077843313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:30:03.076067533 +0000 UTC m=+865.491863353" watchObservedRunningTime="2026-09-10 15:30:03.077843313 +0000 UTC m=+865.493639133"
Sep 10 15:30:08.949368 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:08.949334    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"]
Sep 10 15:30:08.949923 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:08.949551    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv" podUID="c49e3938-3b5c-46b6-aae9-d63251674e08" containerName="maas-api" containerID="cri-o://beb09914759cd08bfaccfeaa74609274d1095c1606892a35e1e48ae486ad0fab" gracePeriod=30
Sep 10 15:30:09.014768 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.014726    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"]
Sep 10 15:30:09.015031 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.015004    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw" podUID="aa185323-bead-4fb3-94a5-5e3fc6db0ae9" containerName="payload-processing" containerID="cri-o://15d36c54674a2b5697de7e4bfac838b35c4efde1b2f3977dd325251c2012568f" gracePeriod=30
Sep 10 15:30:09.086874 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.086838    2573 generic.go:356] "Generic (PLEG): container finished" podID="c49e3938-3b5c-46b6-aae9-d63251674e08" containerID="beb09914759cd08bfaccfeaa74609274d1095c1606892a35e1e48ae486ad0fab" exitCode=0
Sep 10 15:30:09.087042 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.086920    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv" event={"ID":"c49e3938-3b5c-46b6-aae9-d63251674e08","Type":"ContainerDied","Data":"beb09914759cd08bfaccfeaa74609274d1095c1606892a35e1e48ae486ad0fab"}
Sep 10 15:30:09.274952 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.274906    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:30:09.329271 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.329247    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"
Sep 10 15:30:09.456982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.456946    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa185323-bead-4fb3-94a5-5e3fc6db0ae9-plugins-config-volume\") pod \"aa185323-bead-4fb3-94a5-5e3fc6db0ae9\" (UID: \"aa185323-bead-4fb3-94a5-5e3fc6db0ae9\") "
Sep 10 15:30:09.457185 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.457012    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-maas-api-tls\") pod \"c49e3938-3b5c-46b6-aae9-d63251674e08\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") "
Sep 10 15:30:09.457185 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.457072    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-metrics-certs\") pod \"c49e3938-3b5c-46b6-aae9-d63251674e08\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") "
Sep 10 15:30:09.457185 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.457149    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwxvn\" (UniqueName: \"kubernetes.io/projected/c49e3938-3b5c-46b6-aae9-d63251674e08-kube-api-access-bwxvn\") pod \"c49e3938-3b5c-46b6-aae9-d63251674e08\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") "
Sep 10 15:30:09.457185 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.457175    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6wv6\" (UniqueName: \"kubernetes.io/projected/aa185323-bead-4fb3-94a5-5e3fc6db0ae9-kube-api-access-g6wv6\") pod \"aa185323-bead-4fb3-94a5-5e3fc6db0ae9\" (UID: \"aa185323-bead-4fb3-94a5-5e3fc6db0ae9\") "
Sep 10 15:30:09.457409 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.457215    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49e3938-3b5c-46b6-aae9-d63251674e08-openshift-service-ca\") pod \"c49e3938-3b5c-46b6-aae9-d63251674e08\" (UID: \"c49e3938-3b5c-46b6-aae9-d63251674e08\") "
Sep 10 15:30:09.457409 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.457359    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa185323-bead-4fb3-94a5-5e3fc6db0ae9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa185323-bead-4fb3-94a5-5e3fc6db0ae9" (UID: "aa185323-bead-4fb3-94a5-5e3fc6db0ae9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:30:09.457615 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.457512    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa185323-bead-4fb3-94a5-5e3fc6db0ae9-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:30:09.457694 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.457671    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49e3938-3b5c-46b6-aae9-d63251674e08-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c49e3938-3b5c-46b6-aae9-d63251674e08" (UID: "c49e3938-3b5c-46b6-aae9-d63251674e08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:30:09.459443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.459419    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c49e3938-3b5c-46b6-aae9-d63251674e08" (UID: "c49e3938-3b5c-46b6-aae9-d63251674e08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:30:09.459443 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.459424    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa185323-bead-4fb3-94a5-5e3fc6db0ae9-kube-api-access-g6wv6" (OuterVolumeSpecName: "kube-api-access-g6wv6") pod "aa185323-bead-4fb3-94a5-5e3fc6db0ae9" (UID: "aa185323-bead-4fb3-94a5-5e3fc6db0ae9"). InnerVolumeSpecName "kube-api-access-g6wv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:30:09.459805 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.459772    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c49e3938-3b5c-46b6-aae9-d63251674e08" (UID: "c49e3938-3b5c-46b6-aae9-d63251674e08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:30:09.459864 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.459779    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49e3938-3b5c-46b6-aae9-d63251674e08-kube-api-access-bwxvn" (OuterVolumeSpecName: "kube-api-access-bwxvn") pod "c49e3938-3b5c-46b6-aae9-d63251674e08" (UID: "c49e3938-3b5c-46b6-aae9-d63251674e08"). InnerVolumeSpecName "kube-api-access-bwxvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:30:09.558688 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.558604    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwxvn\" (UniqueName: \"kubernetes.io/projected/c49e3938-3b5c-46b6-aae9-d63251674e08-kube-api-access-bwxvn\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:30:09.558688 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.558635    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6wv6\" (UniqueName: \"kubernetes.io/projected/aa185323-bead-4fb3-94a5-5e3fc6db0ae9-kube-api-access-g6wv6\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:30:09.558688 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.558646    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c49e3938-3b5c-46b6-aae9-d63251674e08-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:30:09.558688 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.558655    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:30:09.558688 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:09.558664    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c49e3938-3b5c-46b6-aae9-d63251674e08-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:30:10.093179 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.093143    2573 generic.go:356] "Generic (PLEG): container finished" podID="aa185323-bead-4fb3-94a5-5e3fc6db0ae9" containerID="15d36c54674a2b5697de7e4bfac838b35c4efde1b2f3977dd325251c2012568f" exitCode=0
Sep 10 15:30:10.093649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.093205    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"
Sep 10 15:30:10.093649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.093238    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw" event={"ID":"aa185323-bead-4fb3-94a5-5e3fc6db0ae9","Type":"ContainerDied","Data":"15d36c54674a2b5697de7e4bfac838b35c4efde1b2f3977dd325251c2012568f"}
Sep 10 15:30:10.093649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.093293    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw" event={"ID":"aa185323-bead-4fb3-94a5-5e3fc6db0ae9","Type":"ContainerDied","Data":"30dfd4da2044c8ed5e0728eabaf49c8a6ce592981c9aa54bbef1d8802d39c44e"}
Sep 10 15:30:10.093649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.093316    2573 scope.go:117] "RemoveContainer" containerID="15d36c54674a2b5697de7e4bfac838b35c4efde1b2f3977dd325251c2012568f"
Sep 10 15:30:10.094735 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.094710    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"
Sep 10 15:30:10.094735 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.094715    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv" event={"ID":"c49e3938-3b5c-46b6-aae9-d63251674e08","Type":"ContainerDied","Data":"fd74e11ce9c9f02da1d322026736df135c579c28cf53a1f6c3d70db0c9159064"}
Sep 10 15:30:10.102705 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.102686    2573 scope.go:117] "RemoveContainer" containerID="15d36c54674a2b5697de7e4bfac838b35c4efde1b2f3977dd325251c2012568f"
Sep 10 15:30:10.103048 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:30:10.103029    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d36c54674a2b5697de7e4bfac838b35c4efde1b2f3977dd325251c2012568f\": container with ID starting with 15d36c54674a2b5697de7e4bfac838b35c4efde1b2f3977dd325251c2012568f not found: ID does not exist" containerID="15d36c54674a2b5697de7e4bfac838b35c4efde1b2f3977dd325251c2012568f"
Sep 10 15:30:10.103113 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.103058    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d36c54674a2b5697de7e4bfac838b35c4efde1b2f3977dd325251c2012568f"} err="failed to get container status \"15d36c54674a2b5697de7e4bfac838b35c4efde1b2f3977dd325251c2012568f\": rpc error: code = NotFound desc = could not find container \"15d36c54674a2b5697de7e4bfac838b35c4efde1b2f3977dd325251c2012568f\": container with ID starting with 15d36c54674a2b5697de7e4bfac838b35c4efde1b2f3977dd325251c2012568f not found: ID does not exist"
Sep 10 15:30:10.103113 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.103074    2573 scope.go:117] "RemoveContainer" containerID="beb09914759cd08bfaccfeaa74609274d1095c1606892a35e1e48ae486ad0fab"
Sep 10 15:30:10.117813 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.117769    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"]
Sep 10 15:30:10.125130 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.125111    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f5d25c39-74d9d895c4-9t2bv"]
Sep 10 15:30:10.133432 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.133409    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"]
Sep 10 15:30:10.135581 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:10.135560    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f5d25c39-6484ccc56f-d68pw"]
Sep 10 15:30:12.125480 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:12.125445    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa185323-bead-4fb3-94a5-5e3fc6db0ae9" path="/var/lib/kubelet/pods/aa185323-bead-4fb3-94a5-5e3fc6db0ae9/volumes"
Sep 10 15:30:12.125878 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:12.125838    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c49e3938-3b5c-46b6-aae9-d63251674e08" path="/var/lib/kubelet/pods/c49e3938-3b5c-46b6-aae9-d63251674e08/volumes"
Sep 10 15:30:12.140445 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:12.140422    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:12.140619 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:12.140480    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:12.153516 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:12.153490    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:13.122965 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:13.122931    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:30:17.416966 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.416930    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"]
Sep 10 15:30:17.417434 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.417286    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c49e3938-3b5c-46b6-aae9-d63251674e08" containerName="maas-api"
Sep 10 15:30:17.417434 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.417299    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c49e3938-3b5c-46b6-aae9-d63251674e08" containerName="maas-api"
Sep 10 15:30:17.417434 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.417306    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa185323-bead-4fb3-94a5-5e3fc6db0ae9" containerName="payload-processing"
Sep 10 15:30:17.417434 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.417312    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa185323-bead-4fb3-94a5-5e3fc6db0ae9" containerName="payload-processing"
Sep 10 15:30:17.417434 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.417375    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa185323-bead-4fb3-94a5-5e3fc6db0ae9" containerName="payload-processing"
Sep 10 15:30:17.417434 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.417387    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c49e3938-3b5c-46b6-aae9-d63251674e08" containerName="maas-api"
Sep 10 15:30:17.421949 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.421925    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:17.425143 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.425114    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-57c78a-serving-cert\""
Sep 10 15:30:17.427951 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.427928    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-maas-api-tls\") pod \"maas-api-e2e-explicit-57c78a-57b6556d95-c54ct\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:17.428057 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.428007    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0874b59-f95e-4e86-a7b5-dc1085c53222-openshift-service-ca\") pod \"maas-api-e2e-explicit-57c78a-57b6556d95-c54ct\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:17.428099 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.428050    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4ftz\" (UniqueName: \"kubernetes.io/projected/b0874b59-f95e-4e86-a7b5-dc1085c53222-kube-api-access-z4ftz\") pod \"maas-api-e2e-explicit-57c78a-57b6556d95-c54ct\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:17.428134 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.428098    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-metrics-certs\") pod \"maas-api-e2e-explicit-57c78a-57b6556d95-c54ct\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:17.432874 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.432849    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"]
Sep 10 15:30:17.528562 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.528516    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0874b59-f95e-4e86-a7b5-dc1085c53222-openshift-service-ca\") pod \"maas-api-e2e-explicit-57c78a-57b6556d95-c54ct\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:17.528562 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.528566    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4ftz\" (UniqueName: \"kubernetes.io/projected/b0874b59-f95e-4e86-a7b5-dc1085c53222-kube-api-access-z4ftz\") pod \"maas-api-e2e-explicit-57c78a-57b6556d95-c54ct\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:17.528774 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.528695    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-metrics-certs\") pod \"maas-api-e2e-explicit-57c78a-57b6556d95-c54ct\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:17.528845 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.528776    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-maas-api-tls\") pod \"maas-api-e2e-explicit-57c78a-57b6556d95-c54ct\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:17.528941 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:30:17.528920    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-serving-cert: secret "maas-api-e2e-explicit-57c78a-serving-cert" not found
Sep 10 15:30:17.529013 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:30:17.529001    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-maas-api-tls podName:b0874b59-f95e-4e86-a7b5-dc1085c53222 nodeName:}" failed. No retries permitted until 2026-09-10 15:30:18.028980893 +0000 UTC m=+880.444776699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-maas-api-tls") pod "maas-api-e2e-explicit-57c78a-57b6556d95-c54ct" (UID: "b0874b59-f95e-4e86-a7b5-dc1085c53222") : secret "maas-api-e2e-explicit-57c78a-serving-cert" not found
Sep 10 15:30:17.529306 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.529291    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0874b59-f95e-4e86-a7b5-dc1085c53222-openshift-service-ca\") pod \"maas-api-e2e-explicit-57c78a-57b6556d95-c54ct\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:17.531274 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.531251    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-metrics-certs\") pod \"maas-api-e2e-explicit-57c78a-57b6556d95-c54ct\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:17.541862 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:17.541841    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4ftz\" (UniqueName: \"kubernetes.io/projected/b0874b59-f95e-4e86-a7b5-dc1085c53222-kube-api-access-z4ftz\") pod \"maas-api-e2e-explicit-57c78a-57b6556d95-c54ct\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:18.033141 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.033094    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-maas-api-tls\") pod \"maas-api-e2e-explicit-57c78a-57b6556d95-c54ct\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:18.035766 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.035729    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-maas-api-tls\") pod \"maas-api-e2e-explicit-57c78a-57b6556d95-c54ct\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:18.184606 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.184565    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"]
Sep 10 15:30:18.188139 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.188117    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"
Sep 10 15:30:18.190493 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.190464    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-57c78a-dockercfg-stdpt\""
Sep 10 15:30:18.190632 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.190603    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-57c78a\""
Sep 10 15:30:18.206021 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.205985    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"]
Sep 10 15:30:18.208002 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.207975    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"]
Sep 10 15:30:18.212104 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.212083    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"
Sep 10 15:30:18.228547 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.228526    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"]
Sep 10 15:30:18.234292 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.234264    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxx8s\" (UniqueName: \"kubernetes.io/projected/e0730bed-86f7-4307-9111-4671acf0da34-kube-api-access-wxx8s\") pod \"payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm\" (UID: \"e0730bed-86f7-4307-9111-4671acf0da34\") " pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"
Sep 10 15:30:18.234392 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.234307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw9fs\" (UniqueName: \"kubernetes.io/projected/519bcbaa-86eb-4ead-97cf-3d300039723b-kube-api-access-mw9fs\") pod \"payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6\" (UID: \"519bcbaa-86eb-4ead-97cf-3d300039723b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"
Sep 10 15:30:18.234392 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.234351    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0730bed-86f7-4307-9111-4671acf0da34-plugins-config-volume\") pod \"payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm\" (UID: \"e0730bed-86f7-4307-9111-4671acf0da34\") " pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"
Sep 10 15:30:18.234473 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.234427    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519bcbaa-86eb-4ead-97cf-3d300039723b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6\" (UID: \"519bcbaa-86eb-4ead-97cf-3d300039723b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"
Sep 10 15:30:18.333119 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.333085    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:18.335440 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.335401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxx8s\" (UniqueName: \"kubernetes.io/projected/e0730bed-86f7-4307-9111-4671acf0da34-kube-api-access-wxx8s\") pod \"payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm\" (UID: \"e0730bed-86f7-4307-9111-4671acf0da34\") " pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"
Sep 10 15:30:18.335440 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.335444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw9fs\" (UniqueName: \"kubernetes.io/projected/519bcbaa-86eb-4ead-97cf-3d300039723b-kube-api-access-mw9fs\") pod \"payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6\" (UID: \"519bcbaa-86eb-4ead-97cf-3d300039723b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"
Sep 10 15:30:18.335628 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.335473    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0730bed-86f7-4307-9111-4671acf0da34-plugins-config-volume\") pod \"payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm\" (UID: \"e0730bed-86f7-4307-9111-4671acf0da34\") " pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"
Sep 10 15:30:18.335628 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.335536    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519bcbaa-86eb-4ead-97cf-3d300039723b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6\" (UID: \"519bcbaa-86eb-4ead-97cf-3d300039723b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"
Sep 10 15:30:18.336424 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.336297    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519bcbaa-86eb-4ead-97cf-3d300039723b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6\" (UID: \"519bcbaa-86eb-4ead-97cf-3d300039723b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"
Sep 10 15:30:18.336492 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.336465    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0730bed-86f7-4307-9111-4671acf0da34-plugins-config-volume\") pod \"payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm\" (UID: \"e0730bed-86f7-4307-9111-4671acf0da34\") " pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"
Sep 10 15:30:18.345540 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.345485    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxx8s\" (UniqueName: \"kubernetes.io/projected/e0730bed-86f7-4307-9111-4671acf0da34-kube-api-access-wxx8s\") pod \"payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm\" (UID: \"e0730bed-86f7-4307-9111-4671acf0da34\") " pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"
Sep 10 15:30:18.345540 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.345482    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw9fs\" (UniqueName: \"kubernetes.io/projected/519bcbaa-86eb-4ead-97cf-3d300039723b-kube-api-access-mw9fs\") pod \"payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6\" (UID: \"519bcbaa-86eb-4ead-97cf-3d300039723b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"
Sep 10 15:30:18.499480 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.499445    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"
Sep 10 15:30:18.523074 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.522865    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"
Sep 10 15:30:18.618304 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.618238    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"]
Sep 10 15:30:18.622671 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:30:18.622625    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0874b59_f95e_4e86_a7b5_dc1085c53222.slice/crio-3650721ce31bc21e9fd6686b849de548496ade4ce2d085554902c43e351fbaaa WatchSource:0}: Error finding container 3650721ce31bc21e9fd6686b849de548496ade4ce2d085554902c43e351fbaaa: Status 404 returned error can't find the container with id 3650721ce31bc21e9fd6686b849de548496ade4ce2d085554902c43e351fbaaa
Sep 10 15:30:18.803106 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.803074    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"]
Sep 10 15:30:18.806683 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:30:18.806657    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519bcbaa_86eb_4ead_97cf_3d300039723b.slice/crio-dab0363a677c879609b7a63dde30d702754664c6186de73796c9f3f6510a27c9 WatchSource:0}: Error finding container dab0363a677c879609b7a63dde30d702754664c6186de73796c9f3f6510a27c9: Status 404 returned error can't find the container with id dab0363a677c879609b7a63dde30d702754664c6186de73796c9f3f6510a27c9
Sep 10 15:30:18.827399 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:18.827181    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"]
Sep 10 15:30:18.830292 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:30:18.830264    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0730bed_86f7_4307_9111_4671acf0da34.slice/crio-864a1c3fb16c3efe417dc7c31325e975af871307c20fb7f9388432f380ea7712 WatchSource:0}: Error finding container 864a1c3fb16c3efe417dc7c31325e975af871307c20fb7f9388432f380ea7712: Status 404 returned error can't find the container with id 864a1c3fb16c3efe417dc7c31325e975af871307c20fb7f9388432f380ea7712
Sep 10 15:30:19.142243 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:19.142144    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct" event={"ID":"b0874b59-f95e-4e86-a7b5-dc1085c53222","Type":"ContainerStarted","Data":"f5df13ce36ddb6270183f26ece8342cf40d152835436dbbe71c92c7fb29808b9"}
Sep 10 15:30:19.142243 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:19.142220    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:19.142243 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:19.142240    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct" event={"ID":"b0874b59-f95e-4e86-a7b5-dc1085c53222","Type":"ContainerStarted","Data":"3650721ce31bc21e9fd6686b849de548496ade4ce2d085554902c43e351fbaaa"}
Sep 10 15:30:19.143887 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:19.143859    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6" event={"ID":"519bcbaa-86eb-4ead-97cf-3d300039723b","Type":"ContainerStarted","Data":"af09abd02e5121b7fc4a2e14b94129f6e96dc3f880aae2d8bc3e1657b55181a1"}
Sep 10 15:30:19.144030 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:19.143911    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6" event={"ID":"519bcbaa-86eb-4ead-97cf-3d300039723b","Type":"ContainerStarted","Data":"dab0363a677c879609b7a63dde30d702754664c6186de73796c9f3f6510a27c9"}
Sep 10 15:30:19.145675 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:19.145382    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm" event={"ID":"e0730bed-86f7-4307-9111-4671acf0da34","Type":"ContainerStarted","Data":"c8e1ad7620f4b9095a7fa77ca57d1c08b1d12f6893bc971933636031da1ded9f"}
Sep 10 15:30:19.145675 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:19.145433    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm" event={"ID":"e0730bed-86f7-4307-9111-4671acf0da34","Type":"ContainerStarted","Data":"864a1c3fb16c3efe417dc7c31325e975af871307c20fb7f9388432f380ea7712"}
Sep 10 15:30:19.145675 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:19.145522    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"
Sep 10 15:30:19.166358 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:19.166309    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct" podStartSLOduration=1.75989651 podStartE2EDuration="2.166292691s" totalImagesPullingTime="406.396181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:30:19.166111319 +0000 UTC m=+881.581907140" watchObservedRunningTime="2026-09-10 15:30:19.166292691 +0000 UTC m=+881.582088510"
Sep 10 15:30:19.185727 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:19.185661    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6" podStartSLOduration=1.185639793 podStartE2EDuration="1.185639793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:30:19.183423756 +0000 UTC m=+881.599219577" watchObservedRunningTime="2026-09-10 15:30:19.185639793 +0000 UTC m=+881.601435611"
Sep 10 15:30:19.204605 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:19.204543    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm" podStartSLOduration=1.204523327 podStartE2EDuration="1.204523327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:30:19.203586418 +0000 UTC m=+881.619382244" watchObservedRunningTime="2026-09-10 15:30:19.204523327 +0000 UTC m=+881.620319147"
Sep 10 15:30:20.150919 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:20.150881    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"
Sep 10 15:30:25.156848 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:25.156817    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:30:30.152889 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:30.152780    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"
Sep 10 15:30:31.157978 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:30:31.157948    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"
Sep 10 15:31:33.158084 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.158031    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"]
Sep 10 15:31:33.178042 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:31:33.178010    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-serving-cert: secret "maas-api-e2e-explicit-57c78a-serving-cert" not found
Sep 10 15:31:33.178222 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:31:33.178097    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-maas-api-tls podName:b0874b59-f95e-4e86-a7b5-dc1085c53222 nodeName:}" failed. No retries permitted until 2026-09-10 15:31:33.678079284 +0000 UTC m=+956.093875081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-maas-api-tls") pod "maas-api-e2e-explicit-57c78a-57b6556d95-c54ct" (UID: "b0874b59-f95e-4e86-a7b5-dc1085c53222") : secret "maas-api-e2e-explicit-57c78a-serving-cert" not found
Sep 10 15:31:33.233894 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.233852    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"]
Sep 10 15:31:33.234818 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.234743    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm" podUID="e0730bed-86f7-4307-9111-4671acf0da34" containerName="payload-processing" containerID="cri-o://c8e1ad7620f4b9095a7fa77ca57d1c08b1d12f6893bc971933636031da1ded9f" gracePeriod=30
Sep 10 15:31:33.237394 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.237368    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"]
Sep 10 15:31:33.237648 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.237595    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6" podUID="519bcbaa-86eb-4ead-97cf-3d300039723b" containerName="payload-pre-processing" containerID="cri-o://af09abd02e5121b7fc4a2e14b94129f6e96dc3f880aae2d8bc3e1657b55181a1" gracePeriod=30
Sep 10 15:31:33.514069 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.514037    2573 generic.go:356] "Generic (PLEG): container finished" podID="519bcbaa-86eb-4ead-97cf-3d300039723b" containerID="af09abd02e5121b7fc4a2e14b94129f6e96dc3f880aae2d8bc3e1657b55181a1" exitCode=0
Sep 10 15:31:33.514203 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.514144    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6" event={"ID":"519bcbaa-86eb-4ead-97cf-3d300039723b","Type":"ContainerDied","Data":"af09abd02e5121b7fc4a2e14b94129f6e96dc3f880aae2d8bc3e1657b55181a1"}
Sep 10 15:31:33.514203 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.514194    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6" event={"ID":"519bcbaa-86eb-4ead-97cf-3d300039723b","Type":"ContainerDied","Data":"dab0363a677c879609b7a63dde30d702754664c6186de73796c9f3f6510a27c9"}
Sep 10 15:31:33.514320 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.514214    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dab0363a677c879609b7a63dde30d702754664c6186de73796c9f3f6510a27c9"
Sep 10 15:31:33.516153 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.516129    2573 generic.go:356] "Generic (PLEG): container finished" podID="e0730bed-86f7-4307-9111-4671acf0da34" containerID="c8e1ad7620f4b9095a7fa77ca57d1c08b1d12f6893bc971933636031da1ded9f" exitCode=0
Sep 10 15:31:33.516255 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.516194    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm" event={"ID":"e0730bed-86f7-4307-9111-4671acf0da34","Type":"ContainerDied","Data":"c8e1ad7620f4b9095a7fa77ca57d1c08b1d12f6893bc971933636031da1ded9f"}
Sep 10 15:31:33.516324 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.516305    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct" podUID="b0874b59-f95e-4e86-a7b5-dc1085c53222" containerName="maas-api" containerID="cri-o://f5df13ce36ddb6270183f26ece8342cf40d152835436dbbe71c92c7fb29808b9" gracePeriod=30
Sep 10 15:31:33.602558 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.602534    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"
Sep 10 15:31:33.605571 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.605540    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"
Sep 10 15:31:33.683081 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.683003    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxx8s\" (UniqueName: \"kubernetes.io/projected/e0730bed-86f7-4307-9111-4671acf0da34-kube-api-access-wxx8s\") pod \"e0730bed-86f7-4307-9111-4671acf0da34\" (UID: \"e0730bed-86f7-4307-9111-4671acf0da34\") "
Sep 10 15:31:33.683081 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.683062    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw9fs\" (UniqueName: \"kubernetes.io/projected/519bcbaa-86eb-4ead-97cf-3d300039723b-kube-api-access-mw9fs\") pod \"519bcbaa-86eb-4ead-97cf-3d300039723b\" (UID: \"519bcbaa-86eb-4ead-97cf-3d300039723b\") "
Sep 10 15:31:33.683284 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.683087    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0730bed-86f7-4307-9111-4671acf0da34-plugins-config-volume\") pod \"e0730bed-86f7-4307-9111-4671acf0da34\" (UID: \"e0730bed-86f7-4307-9111-4671acf0da34\") "
Sep 10 15:31:33.683284 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.683138    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519bcbaa-86eb-4ead-97cf-3d300039723b-plugins-config-volume\") pod \"519bcbaa-86eb-4ead-97cf-3d300039723b\" (UID: \"519bcbaa-86eb-4ead-97cf-3d300039723b\") "
Sep 10 15:31:33.683474 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:31:33.683447    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-serving-cert: secret "maas-api-e2e-explicit-57c78a-serving-cert" not found
Sep 10 15:31:33.683580 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:31:33.683533    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-maas-api-tls podName:b0874b59-f95e-4e86-a7b5-dc1085c53222 nodeName:}" failed. No retries permitted until 2026-09-10 15:31:34.6835131 +0000 UTC m=+957.099308911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-maas-api-tls") pod "maas-api-e2e-explicit-57c78a-57b6556d95-c54ct" (UID: "b0874b59-f95e-4e86-a7b5-dc1085c53222") : secret "maas-api-e2e-explicit-57c78a-serving-cert" not found
Sep 10 15:31:33.683719 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.683675    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0730bed-86f7-4307-9111-4671acf0da34-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0730bed-86f7-4307-9111-4671acf0da34" (UID: "e0730bed-86f7-4307-9111-4671acf0da34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:31:33.683719 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.683686    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519bcbaa-86eb-4ead-97cf-3d300039723b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "519bcbaa-86eb-4ead-97cf-3d300039723b" (UID: "519bcbaa-86eb-4ead-97cf-3d300039723b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:31:33.685373 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.685347    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0730bed-86f7-4307-9111-4671acf0da34-kube-api-access-wxx8s" (OuterVolumeSpecName: "kube-api-access-wxx8s") pod "e0730bed-86f7-4307-9111-4671acf0da34" (UID: "e0730bed-86f7-4307-9111-4671acf0da34"). InnerVolumeSpecName "kube-api-access-wxx8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:31:33.685496 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.685477    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519bcbaa-86eb-4ead-97cf-3d300039723b-kube-api-access-mw9fs" (OuterVolumeSpecName: "kube-api-access-mw9fs") pod "519bcbaa-86eb-4ead-97cf-3d300039723b" (UID: "519bcbaa-86eb-4ead-97cf-3d300039723b"). InnerVolumeSpecName "kube-api-access-mw9fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:31:33.765660 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.765627    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:31:33.784721 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.784653    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxx8s\" (UniqueName: \"kubernetes.io/projected/e0730bed-86f7-4307-9111-4671acf0da34-kube-api-access-wxx8s\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:31:33.784721 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.784686    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw9fs\" (UniqueName: \"kubernetes.io/projected/519bcbaa-86eb-4ead-97cf-3d300039723b-kube-api-access-mw9fs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:31:33.784721 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.784702    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0730bed-86f7-4307-9111-4671acf0da34-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:31:33.784721 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.784717    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519bcbaa-86eb-4ead-97cf-3d300039723b-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:31:33.885283 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.885247    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0874b59-f95e-4e86-a7b5-dc1085c53222-openshift-service-ca\") pod \"b0874b59-f95e-4e86-a7b5-dc1085c53222\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") "
Sep 10 15:31:33.885457 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.885325    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-metrics-certs\") pod \"b0874b59-f95e-4e86-a7b5-dc1085c53222\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") "
Sep 10 15:31:33.885457 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.885376    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-maas-api-tls\") pod \"b0874b59-f95e-4e86-a7b5-dc1085c53222\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") "
Sep 10 15:31:33.885457 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.885407    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4ftz\" (UniqueName: \"kubernetes.io/projected/b0874b59-f95e-4e86-a7b5-dc1085c53222-kube-api-access-z4ftz\") pod \"b0874b59-f95e-4e86-a7b5-dc1085c53222\" (UID: \"b0874b59-f95e-4e86-a7b5-dc1085c53222\") "
Sep 10 15:31:33.885662 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.885637    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0874b59-f95e-4e86-a7b5-dc1085c53222-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0874b59-f95e-4e86-a7b5-dc1085c53222" (UID: "b0874b59-f95e-4e86-a7b5-dc1085c53222"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:31:33.885743 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.885712    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0874b59-f95e-4e86-a7b5-dc1085c53222-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:31:33.887711 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.887676    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0874b59-f95e-4e86-a7b5-dc1085c53222-kube-api-access-z4ftz" (OuterVolumeSpecName: "kube-api-access-z4ftz") pod "b0874b59-f95e-4e86-a7b5-dc1085c53222" (UID: "b0874b59-f95e-4e86-a7b5-dc1085c53222"). InnerVolumeSpecName "kube-api-access-z4ftz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:31:33.887711 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.887699    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b0874b59-f95e-4e86-a7b5-dc1085c53222" (UID: "b0874b59-f95e-4e86-a7b5-dc1085c53222"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:31:33.887889 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.887684    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0874b59-f95e-4e86-a7b5-dc1085c53222" (UID: "b0874b59-f95e-4e86-a7b5-dc1085c53222"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:31:33.986718 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.986629    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:31:33.986718 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.986664    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0874b59-f95e-4e86-a7b5-dc1085c53222-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:31:33.986718 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:33.986679    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4ftz\" (UniqueName: \"kubernetes.io/projected/b0874b59-f95e-4e86-a7b5-dc1085c53222-kube-api-access-z4ftz\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:31:34.522502 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.522413    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"
Sep 10 15:31:34.522502 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.522444    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm" event={"ID":"e0730bed-86f7-4307-9111-4671acf0da34","Type":"ContainerDied","Data":"864a1c3fb16c3efe417dc7c31325e975af871307c20fb7f9388432f380ea7712"}
Sep 10 15:31:34.523044 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.522516    2573 scope.go:117] "RemoveContainer" containerID="c8e1ad7620f4b9095a7fa77ca57d1c08b1d12f6893bc971933636031da1ded9f"
Sep 10 15:31:34.524015 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.523840    2573 generic.go:356] "Generic (PLEG): container finished" podID="b0874b59-f95e-4e86-a7b5-dc1085c53222" containerID="f5df13ce36ddb6270183f26ece8342cf40d152835436dbbe71c92c7fb29808b9" exitCode=0
Sep 10 15:31:34.524015 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.523879    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct" event={"ID":"b0874b59-f95e-4e86-a7b5-dc1085c53222","Type":"ContainerDied","Data":"f5df13ce36ddb6270183f26ece8342cf40d152835436dbbe71c92c7fb29808b9"}
Sep 10 15:31:34.524015 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.523896    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"
Sep 10 15:31:34.524015 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.523936    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct" event={"ID":"b0874b59-f95e-4e86-a7b5-dc1085c53222","Type":"ContainerDied","Data":"3650721ce31bc21e9fd6686b849de548496ade4ce2d085554902c43e351fbaaa"}
Sep 10 15:31:34.524015 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.523980    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"
Sep 10 15:31:34.532183 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.532169    2573 scope.go:117] "RemoveContainer" containerID="f5df13ce36ddb6270183f26ece8342cf40d152835436dbbe71c92c7fb29808b9"
Sep 10 15:31:34.540198 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.540183    2573 scope.go:117] "RemoveContainer" containerID="f5df13ce36ddb6270183f26ece8342cf40d152835436dbbe71c92c7fb29808b9"
Sep 10 15:31:34.540476 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:31:34.540455    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5df13ce36ddb6270183f26ece8342cf40d152835436dbbe71c92c7fb29808b9\": container with ID starting with f5df13ce36ddb6270183f26ece8342cf40d152835436dbbe71c92c7fb29808b9 not found: ID does not exist" containerID="f5df13ce36ddb6270183f26ece8342cf40d152835436dbbe71c92c7fb29808b9"
Sep 10 15:31:34.540545 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.540487    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5df13ce36ddb6270183f26ece8342cf40d152835436dbbe71c92c7fb29808b9"} err="failed to get container status \"f5df13ce36ddb6270183f26ece8342cf40d152835436dbbe71c92c7fb29808b9\": rpc error: code = NotFound desc = could not find container \"f5df13ce36ddb6270183f26ece8342cf40d152835436dbbe71c92c7fb29808b9\": container with ID starting with f5df13ce36ddb6270183f26ece8342cf40d152835436dbbe71c92c7fb29808b9 not found: ID does not exist"
Sep 10 15:31:34.551762 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.551738    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"]
Sep 10 15:31:34.557858 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.557838    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-57c78a-57b6556d95-c54ct"]
Sep 10 15:31:34.578524 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.578493    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"]
Sep 10 15:31:34.581372 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.581352    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57c78a-67ddfd86b6-n8kz6"]
Sep 10 15:31:34.598584 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.598564    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"]
Sep 10 15:31:34.601444 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:34.601422    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57c78a-76dfc56969-x9rzm"]
Sep 10 15:31:36.126156 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:36.126126    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="519bcbaa-86eb-4ead-97cf-3d300039723b" path="/var/lib/kubelet/pods/519bcbaa-86eb-4ead-97cf-3d300039723b/volumes"
Sep 10 15:31:36.126505 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:36.126491    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0874b59-f95e-4e86-a7b5-dc1085c53222" path="/var/lib/kubelet/pods/b0874b59-f95e-4e86-a7b5-dc1085c53222/volumes"
Sep 10 15:31:36.126805 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:36.126775    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0730bed-86f7-4307-9111-4671acf0da34" path="/var/lib/kubelet/pods/e0730bed-86f7-4307-9111-4671acf0da34/volumes"
Sep 10 15:31:53.928049 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:53.928007    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"]
Sep 10 15:31:53.928649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:53.928404    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="519bcbaa-86eb-4ead-97cf-3d300039723b" containerName="payload-pre-processing"
Sep 10 15:31:53.928649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:53.928416    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="519bcbaa-86eb-4ead-97cf-3d300039723b" containerName="payload-pre-processing"
Sep 10 15:31:53.928649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:53.928446    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0874b59-f95e-4e86-a7b5-dc1085c53222" containerName="maas-api"
Sep 10 15:31:53.928649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:53.928455    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0874b59-f95e-4e86-a7b5-dc1085c53222" containerName="maas-api"
Sep 10 15:31:53.928649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:53.928477    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0730bed-86f7-4307-9111-4671acf0da34" containerName="payload-processing"
Sep 10 15:31:53.928649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:53.928486    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0730bed-86f7-4307-9111-4671acf0da34" containerName="payload-processing"
Sep 10 15:31:53.928649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:53.928569    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0730bed-86f7-4307-9111-4671acf0da34" containerName="payload-processing"
Sep 10 15:31:53.928649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:53.928584    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="519bcbaa-86eb-4ead-97cf-3d300039723b" containerName="payload-pre-processing"
Sep 10 15:31:53.928649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:53.928594    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0874b59-f95e-4e86-a7b5-dc1085c53222" containerName="maas-api"
Sep 10 15:31:53.933603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:53.933585    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:53.936712 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:53.936680    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-83557387-serving-cert\""
Sep 10 15:31:53.941723 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:53.941486    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"]
Sep 10 15:31:54.062182 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.062138    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-maas-api-tls\") pod \"maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.062375 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.062189    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-metrics-certs\") pod \"maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.062375 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.062282    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdk6k\" (UniqueName: \"kubernetes.io/projected/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-kube-api-access-jdk6k\") pod \"maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.062375 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.062328    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-openshift-service-ca\") pod \"maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.162988 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.162956    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdk6k\" (UniqueName: \"kubernetes.io/projected/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-kube-api-access-jdk6k\") pod \"maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.163180 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.163001    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-openshift-service-ca\") pod \"maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.163180 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.163045    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-maas-api-tls\") pod \"maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.163180 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.163070    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-metrics-certs\") pod \"maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.163348 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:31:54.163198    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-83557387-serving-cert: secret "maas-api-e2e-mt-83557387-serving-cert" not found
Sep 10 15:31:54.163348 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:31:54.163291    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-maas-api-tls podName:cfd44e94-4bf7-469d-9c0b-7e970621ca8e nodeName:}" failed. No retries permitted until 2026-09-10 15:31:54.663266918 +0000 UTC m=+977.079062729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-maas-api-tls") pod "maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b" (UID: "cfd44e94-4bf7-469d-9c0b-7e970621ca8e") : secret "maas-api-e2e-mt-83557387-serving-cert" not found
Sep 10 15:31:54.163705 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.163684    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-openshift-service-ca\") pod \"maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.165606 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.165586    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-metrics-certs\") pod \"maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.177072 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.177045    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdk6k\" (UniqueName: \"kubernetes.io/projected/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-kube-api-access-jdk6k\") pod \"maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.589301 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.589260    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"]
Sep 10 15:31:54.593535 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.593513    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"
Sep 10 15:31:54.596074 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.596049    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-83557387-dockercfg-48l7h\""
Sep 10 15:31:54.596191 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.596070    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-83557387\""
Sep 10 15:31:54.606357 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.606330    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"]
Sep 10 15:31:54.668583 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.668529    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mpw9\" (UniqueName: \"kubernetes.io/projected/953ea721-d960-4925-a1df-673a32c11c5a-kube-api-access-6mpw9\") pod \"payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp\" (UID: \"953ea721-d960-4925-a1df-673a32c11c5a\") " pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"
Sep 10 15:31:54.668784 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.668603    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953ea721-d960-4925-a1df-673a32c11c5a-plugins-config-volume\") pod \"payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp\" (UID: \"953ea721-d960-4925-a1df-673a32c11c5a\") " pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"
Sep 10 15:31:54.668906 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.668829    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-maas-api-tls\") pod \"maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.671857 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.671830    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-maas-api-tls\") pod \"maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.770272 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.770221    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mpw9\" (UniqueName: \"kubernetes.io/projected/953ea721-d960-4925-a1df-673a32c11c5a-kube-api-access-6mpw9\") pod \"payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp\" (UID: \"953ea721-d960-4925-a1df-673a32c11c5a\") " pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"
Sep 10 15:31:54.770476 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.770287    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953ea721-d960-4925-a1df-673a32c11c5a-plugins-config-volume\") pod \"payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp\" (UID: \"953ea721-d960-4925-a1df-673a32c11c5a\") " pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"
Sep 10 15:31:54.771289 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.771262    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953ea721-d960-4925-a1df-673a32c11c5a-plugins-config-volume\") pod \"payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp\" (UID: \"953ea721-d960-4925-a1df-673a32c11c5a\") " pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"
Sep 10 15:31:54.781245 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.781209    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mpw9\" (UniqueName: \"kubernetes.io/projected/953ea721-d960-4925-a1df-673a32c11c5a-kube-api-access-6mpw9\") pod \"payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp\" (UID: \"953ea721-d960-4925-a1df-673a32c11c5a\") " pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"
Sep 10 15:31:54.846186 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.846093    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:31:54.905090 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:54.905059    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"
Sep 10 15:31:55.120247 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:55.120225    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"]
Sep 10 15:31:55.122295 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:31:55.122269    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd44e94_4bf7_469d_9c0b_7e970621ca8e.slice/crio-ae9d87c53c13c3ea4c62ca130f3e3823fc74fcf3e3f8ba868863ebb3278716c5 WatchSource:0}: Error finding container ae9d87c53c13c3ea4c62ca130f3e3823fc74fcf3e3f8ba868863ebb3278716c5: Status 404 returned error can't find the container with id ae9d87c53c13c3ea4c62ca130f3e3823fc74fcf3e3f8ba868863ebb3278716c5
Sep 10 15:31:55.182896 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:55.182870    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"]
Sep 10 15:31:55.185133 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:31:55.185107    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953ea721_d960_4925_a1df_673a32c11c5a.slice/crio-cd06128835111537d2ab6891cff32c6fbb8cd1039691b7c2efeab06e36b3479d WatchSource:0}: Error finding container cd06128835111537d2ab6891cff32c6fbb8cd1039691b7c2efeab06e36b3479d: Status 404 returned error can't find the container with id cd06128835111537d2ab6891cff32c6fbb8cd1039691b7c2efeab06e36b3479d
Sep 10 15:31:55.627852 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:55.627822    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b" event={"ID":"cfd44e94-4bf7-469d-9c0b-7e970621ca8e","Type":"ContainerStarted","Data":"ae9d87c53c13c3ea4c62ca130f3e3823fc74fcf3e3f8ba868863ebb3278716c5"}
Sep 10 15:31:55.629565 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:55.629537    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp" event={"ID":"953ea721-d960-4925-a1df-673a32c11c5a","Type":"ContainerStarted","Data":"20b3d45b22c542fe6a0fc988ae2434209c43b891101bd011d8410b34a83013a1"}
Sep 10 15:31:55.629707 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:55.629590    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp" event={"ID":"953ea721-d960-4925-a1df-673a32c11c5a","Type":"ContainerStarted","Data":"cd06128835111537d2ab6891cff32c6fbb8cd1039691b7c2efeab06e36b3479d"}
Sep 10 15:31:55.629707 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:55.629617    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"
Sep 10 15:31:55.649743 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:55.649695    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp" podStartSLOduration=1.649674439 podStartE2EDuration="1.649674439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:31:55.648614811 +0000 UTC m=+978.064410651" watchObservedRunningTime="2026-09-10 15:31:55.649674439 +0000 UTC m=+978.065470261"
Sep 10 15:31:56.635890 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:56.635845    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b" event={"ID":"cfd44e94-4bf7-469d-9c0b-7e970621ca8e","Type":"ContainerStarted","Data":"e7335034940e52a9dabc82c65e0dacf1707339db6e55124fb177eda640045e53"}
Sep 10 15:31:56.655236 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:56.655184    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b" podStartSLOduration=3.199698267 podStartE2EDuration="3.655168367s" totalImagesPullingTime="455.4701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:31:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:31:56.653628501 +0000 UTC m=+979.069424331" watchObservedRunningTime="2026-09-10 15:31:56.655168367 +0000 UTC m=+979.070964185"
Sep 10 15:31:57.640922 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:31:57.640891    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:32:03.652584 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:03.652509    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:32:06.638821 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:06.638763    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"
Sep 10 15:32:13.195374 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:13.195342    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"]
Sep 10 15:32:28.215587 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.215508    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89" podUID="54ddddad-710f-4251-a0a4-fd3f0b702f57" containerName="main" containerID="cri-o://2c902eff69f59874fff4b4090e8f8ace13250294d9f0a64880a73d74e7b56a38" gracePeriod=45
Sep 10 15:32:28.463238 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.463213    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:32:28.560456 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.560368    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ddddad-710f-4251-a0a4-fd3f0b702f57-tls-certs\") pod \"54ddddad-710f-4251-a0a4-fd3f0b702f57\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") "
Sep 10 15:32:28.560456 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.560417    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-tmp-dir\") pod \"54ddddad-710f-4251-a0a4-fd3f0b702f57\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") "
Sep 10 15:32:28.560456 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.560456    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-home\") pod \"54ddddad-710f-4251-a0a4-fd3f0b702f57\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") "
Sep 10 15:32:28.560685 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.560506    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-model-cache\") pod \"54ddddad-710f-4251-a0a4-fd3f0b702f57\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") "
Sep 10 15:32:28.560685 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.560534    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-dshm\") pod \"54ddddad-710f-4251-a0a4-fd3f0b702f57\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") "
Sep 10 15:32:28.560685 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.560577    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb7wp\" (UniqueName: \"kubernetes.io/projected/54ddddad-710f-4251-a0a4-fd3f0b702f57-kube-api-access-lb7wp\") pod \"54ddddad-710f-4251-a0a4-fd3f0b702f57\" (UID: \"54ddddad-710f-4251-a0a4-fd3f0b702f57\") "
Sep 10 15:32:28.560887 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.560712    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54ddddad-710f-4251-a0a4-fd3f0b702f57" (UID: "54ddddad-710f-4251-a0a4-fd3f0b702f57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:32:28.560887 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.560767    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-home" (OuterVolumeSpecName: "home") pod "54ddddad-710f-4251-a0a4-fd3f0b702f57" (UID: "54ddddad-710f-4251-a0a4-fd3f0b702f57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:32:28.560887 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.560818    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-model-cache" (OuterVolumeSpecName: "model-cache") pod "54ddddad-710f-4251-a0a4-fd3f0b702f57" (UID: "54ddddad-710f-4251-a0a4-fd3f0b702f57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:32:28.561046 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.560904    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-home\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:28.561046 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.560919    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-model-cache\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:28.561046 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.560949    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-tmp-dir\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:28.562825 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.562783    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ddddad-710f-4251-a0a4-fd3f0b702f57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54ddddad-710f-4251-a0a4-fd3f0b702f57" (UID: "54ddddad-710f-4251-a0a4-fd3f0b702f57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:32:28.562905 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.562829    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-dshm" (OuterVolumeSpecName: "dshm") pod "54ddddad-710f-4251-a0a4-fd3f0b702f57" (UID: "54ddddad-710f-4251-a0a4-fd3f0b702f57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:32:28.562942 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.562914    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ddddad-710f-4251-a0a4-fd3f0b702f57-kube-api-access-lb7wp" (OuterVolumeSpecName: "kube-api-access-lb7wp") pod "54ddddad-710f-4251-a0a4-fd3f0b702f57" (UID: "54ddddad-710f-4251-a0a4-fd3f0b702f57"). InnerVolumeSpecName "kube-api-access-lb7wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:32:28.661393 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.661358    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ddddad-710f-4251-a0a4-fd3f0b702f57-tls-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:28.661393 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.661384    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ddddad-710f-4251-a0a4-fd3f0b702f57-dshm\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:28.661393 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.661394    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb7wp\" (UniqueName: \"kubernetes.io/projected/54ddddad-710f-4251-a0a4-fd3f0b702f57-kube-api-access-lb7wp\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:28.800669 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.800631    2573 generic.go:356] "Generic (PLEG): container finished" podID="54ddddad-710f-4251-a0a4-fd3f0b702f57" containerID="2c902eff69f59874fff4b4090e8f8ace13250294d9f0a64880a73d74e7b56a38" exitCode=0
Sep 10 15:32:28.800856 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.800677    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"
Sep 10 15:32:28.800856 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.800686    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89" event={"ID":"54ddddad-710f-4251-a0a4-fd3f0b702f57","Type":"ContainerDied","Data":"2c902eff69f59874fff4b4090e8f8ace13250294d9f0a64880a73d74e7b56a38"}
Sep 10 15:32:28.800856 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.800717    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89" event={"ID":"54ddddad-710f-4251-a0a4-fd3f0b702f57","Type":"ContainerDied","Data":"8ee7a01fa296c5487c2a5601a5c6859a0ca928e1bf8f175c65c961696c48eff2"}
Sep 10 15:32:28.800856 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.800733    2573 scope.go:117] "RemoveContainer" containerID="2c902eff69f59874fff4b4090e8f8ace13250294d9f0a64880a73d74e7b56a38"
Sep 10 15:32:28.809937 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.809915    2573 scope.go:117] "RemoveContainer" containerID="2c902eff69f59874fff4b4090e8f8ace13250294d9f0a64880a73d74e7b56a38"
Sep 10 15:32:28.810201 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:32:28.810181    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c902eff69f59874fff4b4090e8f8ace13250294d9f0a64880a73d74e7b56a38\": container with ID starting with 2c902eff69f59874fff4b4090e8f8ace13250294d9f0a64880a73d74e7b56a38 not found: ID does not exist" containerID="2c902eff69f59874fff4b4090e8f8ace13250294d9f0a64880a73d74e7b56a38"
Sep 10 15:32:28.810266 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.810209    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c902eff69f59874fff4b4090e8f8ace13250294d9f0a64880a73d74e7b56a38"} err="failed to get container status \"2c902eff69f59874fff4b4090e8f8ace13250294d9f0a64880a73d74e7b56a38\": rpc error: code = NotFound desc = could not find container \"2c902eff69f59874fff4b4090e8f8ace13250294d9f0a64880a73d74e7b56a38\": container with ID starting with 2c902eff69f59874fff4b4090e8f8ace13250294d9f0a64880a73d74e7b56a38 not found: ID does not exist"
Sep 10 15:32:28.822211 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.822148    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"]
Sep 10 15:32:28.827752 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:28.827727    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-229c42/test-model-229c42-kserve-5d9f98c985-kcp89"]
Sep 10 15:32:30.126001 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:30.125957    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54ddddad-710f-4251-a0a4-fd3f0b702f57" path="/var/lib/kubelet/pods/54ddddad-710f-4251-a0a4-fd3f0b702f57/volumes"
Sep 10 15:32:33.869762 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:33.869724    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"]
Sep 10 15:32:33.870205 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:33.870003    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m" podUID="0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d" containerName="maas-api" containerID="cri-o://1ccfc56e2b9cfb9656db58889bae9fd39391ba5e1df89f7bb0b83912c6681da4" gracePeriod=30
Sep 10 15:32:33.911387 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:33.911321    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"]
Sep 10 15:32:33.911732 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:33.911695    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck" podUID="237414a6-4bdf-4185-985d-c66709c67d27" containerName="payload-processing" containerID="cri-o://73512bbecfa1c4832487502c2b33cbe13ea541af8399e814b30f61e922c90954" gracePeriod=30
Sep 10 15:32:34.173724 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.173688    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:32:34.203516 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.203488    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"
Sep 10 15:32:34.207359 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.207223    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-openshift-service-ca\") pod \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") "
Sep 10 15:32:34.207359 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.207293    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk2zg\" (UniqueName: \"kubernetes.io/projected/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-kube-api-access-pk2zg\") pod \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") "
Sep 10 15:32:34.207359 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.207329    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-metrics-certs\") pod \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") "
Sep 10 15:32:34.207359 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.207356    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-maas-api-tls\") pod \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\" (UID: \"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d\") "
Sep 10 15:32:34.207681 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.207576    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d" (UID: "0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:32:34.207741 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.207688    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:34.209960 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.209869    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d" (UID: "0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:32:34.209960 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.209949    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d" (UID: "0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:32:34.210098 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.209958    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-kube-api-access-pk2zg" (OuterVolumeSpecName: "kube-api-access-pk2zg") pod "0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d" (UID: "0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d"). InnerVolumeSpecName "kube-api-access-pk2zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:32:34.308308 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.308274    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-682md\" (UniqueName: \"kubernetes.io/projected/237414a6-4bdf-4185-985d-c66709c67d27-kube-api-access-682md\") pod \"237414a6-4bdf-4185-985d-c66709c67d27\" (UID: \"237414a6-4bdf-4185-985d-c66709c67d27\") "
Sep 10 15:32:34.308455 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.308332    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237414a6-4bdf-4185-985d-c66709c67d27-plugins-config-volume\") pod \"237414a6-4bdf-4185-985d-c66709c67d27\" (UID: \"237414a6-4bdf-4185-985d-c66709c67d27\") "
Sep 10 15:32:34.308603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.308580    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk2zg\" (UniqueName: \"kubernetes.io/projected/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-kube-api-access-pk2zg\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:34.308661 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.308604    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:34.308661 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.308619    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:34.308751 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.308732    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237414a6-4bdf-4185-985d-c66709c67d27-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "237414a6-4bdf-4185-985d-c66709c67d27" (UID: "237414a6-4bdf-4185-985d-c66709c67d27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:32:34.310496 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.310471    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237414a6-4bdf-4185-985d-c66709c67d27-kube-api-access-682md" (OuterVolumeSpecName: "kube-api-access-682md") pod "237414a6-4bdf-4185-985d-c66709c67d27" (UID: "237414a6-4bdf-4185-985d-c66709c67d27"). InnerVolumeSpecName "kube-api-access-682md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:32:34.409735 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.409640    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-682md\" (UniqueName: \"kubernetes.io/projected/237414a6-4bdf-4185-985d-c66709c67d27-kube-api-access-682md\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:34.409735 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.409678    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237414a6-4bdf-4185-985d-c66709c67d27-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:34.831682 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.831645    2573 generic.go:356] "Generic (PLEG): container finished" podID="0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d" containerID="1ccfc56e2b9cfb9656db58889bae9fd39391ba5e1df89f7bb0b83912c6681da4" exitCode=0
Sep 10 15:32:34.831913 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.831695    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"
Sep 10 15:32:34.831913 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.831731    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m" event={"ID":"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d","Type":"ContainerDied","Data":"1ccfc56e2b9cfb9656db58889bae9fd39391ba5e1df89f7bb0b83912c6681da4"}
Sep 10 15:32:34.831913 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.831813    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m" event={"ID":"0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d","Type":"ContainerDied","Data":"44561dd2e3f5d391cbe196860157ec7f822bfe5b8bf71e61de0ea43faf8a90f9"}
Sep 10 15:32:34.831913 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.831844    2573 scope.go:117] "RemoveContainer" containerID="1ccfc56e2b9cfb9656db58889bae9fd39391ba5e1df89f7bb0b83912c6681da4"
Sep 10 15:32:34.833170 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.833151    2573 generic.go:356] "Generic (PLEG): container finished" podID="237414a6-4bdf-4185-985d-c66709c67d27" containerID="73512bbecfa1c4832487502c2b33cbe13ea541af8399e814b30f61e922c90954" exitCode=0
Sep 10 15:32:34.833310 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.833183    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck" event={"ID":"237414a6-4bdf-4185-985d-c66709c67d27","Type":"ContainerDied","Data":"73512bbecfa1c4832487502c2b33cbe13ea541af8399e814b30f61e922c90954"}
Sep 10 15:32:34.833310 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.833230    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck" event={"ID":"237414a6-4bdf-4185-985d-c66709c67d27","Type":"ContainerDied","Data":"a98949f77eb19375cb73580f22d25cda3489501fdfb3fb8004166e09b93d258d"}
Sep 10 15:32:34.833310 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.833190    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"
Sep 10 15:32:34.843376 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.843361    2573 scope.go:117] "RemoveContainer" containerID="1ccfc56e2b9cfb9656db58889bae9fd39391ba5e1df89f7bb0b83912c6681da4"
Sep 10 15:32:34.843719 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:32:34.843693    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccfc56e2b9cfb9656db58889bae9fd39391ba5e1df89f7bb0b83912c6681da4\": container with ID starting with 1ccfc56e2b9cfb9656db58889bae9fd39391ba5e1df89f7bb0b83912c6681da4 not found: ID does not exist" containerID="1ccfc56e2b9cfb9656db58889bae9fd39391ba5e1df89f7bb0b83912c6681da4"
Sep 10 15:32:34.843873 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.843730    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccfc56e2b9cfb9656db58889bae9fd39391ba5e1df89f7bb0b83912c6681da4"} err="failed to get container status \"1ccfc56e2b9cfb9656db58889bae9fd39391ba5e1df89f7bb0b83912c6681da4\": rpc error: code = NotFound desc = could not find container \"1ccfc56e2b9cfb9656db58889bae9fd39391ba5e1df89f7bb0b83912c6681da4\": container with ID starting with 1ccfc56e2b9cfb9656db58889bae9fd39391ba5e1df89f7bb0b83912c6681da4 not found: ID does not exist"
Sep 10 15:32:34.843873 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.843754    2573 scope.go:117] "RemoveContainer" containerID="73512bbecfa1c4832487502c2b33cbe13ea541af8399e814b30f61e922c90954"
Sep 10 15:32:34.852416 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.852398    2573 scope.go:117] "RemoveContainer" containerID="73512bbecfa1c4832487502c2b33cbe13ea541af8399e814b30f61e922c90954"
Sep 10 15:32:34.852714 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:32:34.852678    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73512bbecfa1c4832487502c2b33cbe13ea541af8399e814b30f61e922c90954\": container with ID starting with 73512bbecfa1c4832487502c2b33cbe13ea541af8399e814b30f61e922c90954 not found: ID does not exist" containerID="73512bbecfa1c4832487502c2b33cbe13ea541af8399e814b30f61e922c90954"
Sep 10 15:32:34.852763 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.852721    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73512bbecfa1c4832487502c2b33cbe13ea541af8399e814b30f61e922c90954"} err="failed to get container status \"73512bbecfa1c4832487502c2b33cbe13ea541af8399e814b30f61e922c90954\": rpc error: code = NotFound desc = could not find container \"73512bbecfa1c4832487502c2b33cbe13ea541af8399e814b30f61e922c90954\": container with ID starting with 73512bbecfa1c4832487502c2b33cbe13ea541af8399e814b30f61e922c90954 not found: ID does not exist"
Sep 10 15:32:34.853847 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.853829    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"]
Sep 10 15:32:34.858399 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.858379    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-229c42-59f475f5d9-58n7m"]
Sep 10 15:32:34.870673 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.870651    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"]
Sep 10 15:32:34.874995 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:34.874976    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-229c42-6f96c8d78f-kj9ck"]
Sep 10 15:32:36.125302 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.125267    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d" path="/var/lib/kubelet/pods/0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d/volumes"
Sep 10 15:32:36.125667 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.125603    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="237414a6-4bdf-4185-985d-c66709c67d27" path="/var/lib/kubelet/pods/237414a6-4bdf-4185-985d-c66709c67d27/volumes"
Sep 10 15:32:36.504624 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.504544    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"]
Sep 10 15:32:36.504828 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.504768    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b" podUID="cfd44e94-4bf7-469d-9c0b-7e970621ca8e" containerName="maas-api" containerID="cri-o://e7335034940e52a9dabc82c65e0dacf1707339db6e55124fb177eda640045e53" gracePeriod=30
Sep 10 15:32:36.569576 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.569542    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"]
Sep 10 15:32:36.569891 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.569784    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp" podUID="953ea721-d960-4925-a1df-673a32c11c5a" containerName="payload-processing" containerID="cri-o://20b3d45b22c542fe6a0fc988ae2434209c43b891101bd011d8410b34a83013a1" gracePeriod=30
Sep 10 15:32:36.637230 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.637187    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp" podUID="953ea721-d960-4925-a1df-673a32c11c5a" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.62:9004: connect: connection refused"
Sep 10 15:32:36.815931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.815906    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:32:36.849429 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.849401    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"
Sep 10 15:32:36.850562 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.850538    2573 generic.go:356] "Generic (PLEG): container finished" podID="953ea721-d960-4925-a1df-673a32c11c5a" containerID="20b3d45b22c542fe6a0fc988ae2434209c43b891101bd011d8410b34a83013a1" exitCode=0
Sep 10 15:32:36.850657 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.850609    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp" event={"ID":"953ea721-d960-4925-a1df-673a32c11c5a","Type":"ContainerDied","Data":"20b3d45b22c542fe6a0fc988ae2434209c43b891101bd011d8410b34a83013a1"}
Sep 10 15:32:36.850709 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.850661    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp" event={"ID":"953ea721-d960-4925-a1df-673a32c11c5a","Type":"ContainerDied","Data":"cd06128835111537d2ab6891cff32c6fbb8cd1039691b7c2efeab06e36b3479d"}
Sep 10 15:32:36.850709 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.850690    2573 scope.go:117] "RemoveContainer" containerID="20b3d45b22c542fe6a0fc988ae2434209c43b891101bd011d8410b34a83013a1"
Sep 10 15:32:36.852145 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.852119    2573 generic.go:356] "Generic (PLEG): container finished" podID="cfd44e94-4bf7-469d-9c0b-7e970621ca8e" containerID="e7335034940e52a9dabc82c65e0dacf1707339db6e55124fb177eda640045e53" exitCode=0
Sep 10 15:32:36.852264 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.852156    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b" event={"ID":"cfd44e94-4bf7-469d-9c0b-7e970621ca8e","Type":"ContainerDied","Data":"e7335034940e52a9dabc82c65e0dacf1707339db6e55124fb177eda640045e53"}
Sep 10 15:32:36.852328 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.852309    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b" event={"ID":"cfd44e94-4bf7-469d-9c0b-7e970621ca8e","Type":"ContainerDied","Data":"ae9d87c53c13c3ea4c62ca130f3e3823fc74fcf3e3f8ba868863ebb3278716c5"}
Sep 10 15:32:36.852388 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.852179    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"
Sep 10 15:32:36.859658 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.859635    2573 scope.go:117] "RemoveContainer" containerID="20b3d45b22c542fe6a0fc988ae2434209c43b891101bd011d8410b34a83013a1"
Sep 10 15:32:36.859986 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:32:36.859962    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b3d45b22c542fe6a0fc988ae2434209c43b891101bd011d8410b34a83013a1\": container with ID starting with 20b3d45b22c542fe6a0fc988ae2434209c43b891101bd011d8410b34a83013a1 not found: ID does not exist" containerID="20b3d45b22c542fe6a0fc988ae2434209c43b891101bd011d8410b34a83013a1"
Sep 10 15:32:36.860041 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.859997    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b3d45b22c542fe6a0fc988ae2434209c43b891101bd011d8410b34a83013a1"} err="failed to get container status \"20b3d45b22c542fe6a0fc988ae2434209c43b891101bd011d8410b34a83013a1\": rpc error: code = NotFound desc = could not find container \"20b3d45b22c542fe6a0fc988ae2434209c43b891101bd011d8410b34a83013a1\": container with ID starting with 20b3d45b22c542fe6a0fc988ae2434209c43b891101bd011d8410b34a83013a1 not found: ID does not exist"
Sep 10 15:32:36.860041 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.860024    2573 scope.go:117] "RemoveContainer" containerID="e7335034940e52a9dabc82c65e0dacf1707339db6e55124fb177eda640045e53"
Sep 10 15:32:36.868290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.868240    2573 scope.go:117] "RemoveContainer" containerID="e7335034940e52a9dabc82c65e0dacf1707339db6e55124fb177eda640045e53"
Sep 10 15:32:36.868671 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:32:36.868640    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7335034940e52a9dabc82c65e0dacf1707339db6e55124fb177eda640045e53\": container with ID starting with e7335034940e52a9dabc82c65e0dacf1707339db6e55124fb177eda640045e53 not found: ID does not exist" containerID="e7335034940e52a9dabc82c65e0dacf1707339db6e55124fb177eda640045e53"
Sep 10 15:32:36.868815 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.868678    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7335034940e52a9dabc82c65e0dacf1707339db6e55124fb177eda640045e53"} err="failed to get container status \"e7335034940e52a9dabc82c65e0dacf1707339db6e55124fb177eda640045e53\": rpc error: code = NotFound desc = could not find container \"e7335034940e52a9dabc82c65e0dacf1707339db6e55124fb177eda640045e53\": container with ID starting with e7335034940e52a9dabc82c65e0dacf1707339db6e55124fb177eda640045e53 not found: ID does not exist"
Sep 10 15:32:36.928511 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.928482    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-openshift-service-ca\") pod \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") "
Sep 10 15:32:36.928712 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.928528    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdk6k\" (UniqueName: \"kubernetes.io/projected/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-kube-api-access-jdk6k\") pod \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") "
Sep 10 15:32:36.928712 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.928545    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953ea721-d960-4925-a1df-673a32c11c5a-plugins-config-volume\") pod \"953ea721-d960-4925-a1df-673a32c11c5a\" (UID: \"953ea721-d960-4925-a1df-673a32c11c5a\") "
Sep 10 15:32:36.928712 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.928569    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mpw9\" (UniqueName: \"kubernetes.io/projected/953ea721-d960-4925-a1df-673a32c11c5a-kube-api-access-6mpw9\") pod \"953ea721-d960-4925-a1df-673a32c11c5a\" (UID: \"953ea721-d960-4925-a1df-673a32c11c5a\") "
Sep 10 15:32:36.928712 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.928599    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-maas-api-tls\") pod \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") "
Sep 10 15:32:36.928712 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.928640    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-metrics-certs\") pod \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\" (UID: \"cfd44e94-4bf7-469d-9c0b-7e970621ca8e\") "
Sep 10 15:32:36.929020 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.928962    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953ea721-d960-4925-a1df-673a32c11c5a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "953ea721-d960-4925-a1df-673a32c11c5a" (UID: "953ea721-d960-4925-a1df-673a32c11c5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:32:36.929020 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.928976    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cfd44e94-4bf7-469d-9c0b-7e970621ca8e" (UID: "cfd44e94-4bf7-469d-9c0b-7e970621ca8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:32:36.930971 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.930941    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cfd44e94-4bf7-469d-9c0b-7e970621ca8e" (UID: "cfd44e94-4bf7-469d-9c0b-7e970621ca8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:32:36.931090 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.930994    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-kube-api-access-jdk6k" (OuterVolumeSpecName: "kube-api-access-jdk6k") pod "cfd44e94-4bf7-469d-9c0b-7e970621ca8e" (UID: "cfd44e94-4bf7-469d-9c0b-7e970621ca8e"). InnerVolumeSpecName "kube-api-access-jdk6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:32:36.931325 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.931307    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cfd44e94-4bf7-469d-9c0b-7e970621ca8e" (UID: "cfd44e94-4bf7-469d-9c0b-7e970621ca8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:32:36.931416 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:36.931401    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953ea721-d960-4925-a1df-673a32c11c5a-kube-api-access-6mpw9" (OuterVolumeSpecName: "kube-api-access-6mpw9") pod "953ea721-d960-4925-a1df-673a32c11c5a" (UID: "953ea721-d960-4925-a1df-673a32c11c5a"). InnerVolumeSpecName "kube-api-access-6mpw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:32:37.029505 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:37.029424    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:37.029505 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:37.029452    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdk6k\" (UniqueName: \"kubernetes.io/projected/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-kube-api-access-jdk6k\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:37.029505 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:37.029461    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953ea721-d960-4925-a1df-673a32c11c5a-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:37.029505 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:37.029470    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mpw9\" (UniqueName: \"kubernetes.io/projected/953ea721-d960-4925-a1df-673a32c11c5a-kube-api-access-6mpw9\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:37.029505 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:37.029480    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:37.029505 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:37.029488    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd44e94-4bf7-469d-9c0b-7e970621ca8e-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:37.175484 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:37.175448    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"]
Sep 10 15:32:37.179074 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:37.179045    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-83557387-5bdc47cdfc-zcg2b"]
Sep 10 15:32:37.858360 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:37.858332    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"
Sep 10 15:32:37.891506 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:37.891471    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"]
Sep 10 15:32:37.894918 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:37.894881    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-83557387-854b7d5b57-7ntsp"]
Sep 10 15:32:38.127427 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:38.127349    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="953ea721-d960-4925-a1df-673a32c11c5a" path="/var/lib/kubelet/pods/953ea721-d960-4925-a1df-673a32c11c5a/volumes"
Sep 10 15:32:38.127750 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:38.127736    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfd44e94-4bf7-469d-9c0b-7e970621ca8e" path="/var/lib/kubelet/pods/cfd44e94-4bf7-469d-9c0b-7e970621ca8e/volumes"
Sep 10 15:32:51.841263 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:51.841225    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"]
Sep 10 15:32:51.841737 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:51.841555    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln" podUID="d91c8248-5db1-4ce5-8930-ca9b1e7a84c9" containerName="maas-api" containerID="cri-o://dbf374aa88399ae1c162eb44697f60968379a8052374e3f71311b7c184aa6d7f" gracePeriod=30
Sep 10 15:32:52.105816 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.105734    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:32:52.173902 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.173868    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-metrics-certs\") pod \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") "
Sep 10 15:32:52.174092 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.173937    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-openshift-service-ca\") pod \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") "
Sep 10 15:32:52.174092 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.173976    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds4mg\" (UniqueName: \"kubernetes.io/projected/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-kube-api-access-ds4mg\") pod \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") "
Sep 10 15:32:52.174092 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.174016    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-maas-api-tls\") pod \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\" (UID: \"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9\") "
Sep 10 15:32:52.174374 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.174334    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d91c8248-5db1-4ce5-8930-ca9b1e7a84c9" (UID: "d91c8248-5db1-4ce5-8930-ca9b1e7a84c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:32:52.176366 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.176331    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d91c8248-5db1-4ce5-8930-ca9b1e7a84c9" (UID: "d91c8248-5db1-4ce5-8930-ca9b1e7a84c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:32:52.176664 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.176638    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d91c8248-5db1-4ce5-8930-ca9b1e7a84c9" (UID: "d91c8248-5db1-4ce5-8930-ca9b1e7a84c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:32:52.176664 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.176639    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-kube-api-access-ds4mg" (OuterVolumeSpecName: "kube-api-access-ds4mg") pod "d91c8248-5db1-4ce5-8930-ca9b1e7a84c9" (UID: "d91c8248-5db1-4ce5-8930-ca9b1e7a84c9"). InnerVolumeSpecName "kube-api-access-ds4mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:32:52.275244 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.275203    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds4mg\" (UniqueName: \"kubernetes.io/projected/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-kube-api-access-ds4mg\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:52.275244 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.275236    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:52.275244 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.275245    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:52.275244 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.275253    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:32:52.933053 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.933018    2573 generic.go:356] "Generic (PLEG): container finished" podID="d91c8248-5db1-4ce5-8930-ca9b1e7a84c9" containerID="dbf374aa88399ae1c162eb44697f60968379a8052374e3f71311b7c184aa6d7f" exitCode=0
Sep 10 15:32:52.933449 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.933065    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"
Sep 10 15:32:52.933449 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.933109    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln" event={"ID":"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9","Type":"ContainerDied","Data":"dbf374aa88399ae1c162eb44697f60968379a8052374e3f71311b7c184aa6d7f"}
Sep 10 15:32:52.933449 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.933172    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln" event={"ID":"d91c8248-5db1-4ce5-8930-ca9b1e7a84c9","Type":"ContainerDied","Data":"efa7d829825074a1ea6a368ee5fa1baa5ced76a9c307adf55aaf82e866860dde"}
Sep 10 15:32:52.933449 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.933195    2573 scope.go:117] "RemoveContainer" containerID="dbf374aa88399ae1c162eb44697f60968379a8052374e3f71311b7c184aa6d7f"
Sep 10 15:32:52.942479 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.942459    2573 scope.go:117] "RemoveContainer" containerID="dbf374aa88399ae1c162eb44697f60968379a8052374e3f71311b7c184aa6d7f"
Sep 10 15:32:52.942833 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:32:52.942778    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf374aa88399ae1c162eb44697f60968379a8052374e3f71311b7c184aa6d7f\": container with ID starting with dbf374aa88399ae1c162eb44697f60968379a8052374e3f71311b7c184aa6d7f not found: ID does not exist" containerID="dbf374aa88399ae1c162eb44697f60968379a8052374e3f71311b7c184aa6d7f"
Sep 10 15:32:52.942924 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.942844    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf374aa88399ae1c162eb44697f60968379a8052374e3f71311b7c184aa6d7f"} err="failed to get container status \"dbf374aa88399ae1c162eb44697f60968379a8052374e3f71311b7c184aa6d7f\": rpc error: code = NotFound desc = could not find container \"dbf374aa88399ae1c162eb44697f60968379a8052374e3f71311b7c184aa6d7f\": container with ID starting with dbf374aa88399ae1c162eb44697f60968379a8052374e3f71311b7c184aa6d7f not found: ID does not exist"
Sep 10 15:32:52.954176 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.954149    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"]
Sep 10 15:32:52.957915 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:52.957891    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50fe56-69f78b4667-nh7ln"]
Sep 10 15:32:54.127289 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:32:54.127249    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d91c8248-5db1-4ce5-8930-ca9b1e7a84c9" path="/var/lib/kubelet/pods/d91c8248-5db1-4ce5-8930-ca9b1e7a84c9/volumes"
Sep 10 15:33:00.168995 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.168959    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"]
Sep 10 15:33:00.169469 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169439    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="237414a6-4bdf-4185-985d-c66709c67d27" containerName="payload-processing"
Sep 10 15:33:00.169469 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169458    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="237414a6-4bdf-4185-985d-c66709c67d27" containerName="payload-processing"
Sep 10 15:33:00.169544 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169472    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d" containerName="maas-api"
Sep 10 15:33:00.169544 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169481    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d" containerName="maas-api"
Sep 10 15:33:00.169544 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169500    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d91c8248-5db1-4ce5-8930-ca9b1e7a84c9" containerName="maas-api"
Sep 10 15:33:00.169544 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169508    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d91c8248-5db1-4ce5-8930-ca9b1e7a84c9" containerName="maas-api"
Sep 10 15:33:00.169544 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169522    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="953ea721-d960-4925-a1df-673a32c11c5a" containerName="payload-processing"
Sep 10 15:33:00.169544 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169530    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="953ea721-d960-4925-a1df-673a32c11c5a" containerName="payload-processing"
Sep 10 15:33:00.169544 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169544    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ddddad-710f-4251-a0a4-fd3f0b702f57" containerName="main"
Sep 10 15:33:00.169749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169553    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ddddad-710f-4251-a0a4-fd3f0b702f57" containerName="main"
Sep 10 15:33:00.169749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169566    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd44e94-4bf7-469d-9c0b-7e970621ca8e" containerName="maas-api"
Sep 10 15:33:00.169749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169574    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd44e94-4bf7-469d-9c0b-7e970621ca8e" containerName="maas-api"
Sep 10 15:33:00.169749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169646    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd44e94-4bf7-469d-9c0b-7e970621ca8e" containerName="maas-api"
Sep 10 15:33:00.169749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169659    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d91c8248-5db1-4ce5-8930-ca9b1e7a84c9" containerName="maas-api"
Sep 10 15:33:00.169749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169667    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ddddad-710f-4251-a0a4-fd3f0b702f57" containerName="main"
Sep 10 15:33:00.169749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169677    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="953ea721-d960-4925-a1df-673a32c11c5a" containerName="payload-processing"
Sep 10 15:33:00.169749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169688    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb6fcc5-2830-4be4-90f3-003f3d4c4e3d" containerName="maas-api"
Sep 10 15:33:00.169749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.169697    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="237414a6-4bdf-4185-985d-c66709c67d27" containerName="payload-processing"
Sep 10 15:33:00.172733 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.172707    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.175596 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.175576    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 15:33:00.175716 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.175665    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-62d2f7-openshift-default-dockercfg-j5z5l\""
Sep 10 15:33:00.186145 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.186120    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"]
Sep 10 15:33:00.347253 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.347192    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-envoy\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.347450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.347275    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-workload-certs\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.347450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.347295    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-data\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.347450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.347314    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759fj\" (UniqueName: \"kubernetes.io/projected/d547ad4b-a726-422e-aaeb-a5293584e4ac-kube-api-access-759fj\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.347450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.347336    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d547ad4b-a726-422e-aaeb-a5293584e4ac-istiod-ca-cert\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.347450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.347362    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-token\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.347450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.347427    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-credential-socket\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.347727 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.347547    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-workload-socket\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.347727 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.347602    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-podinfo\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.448766 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.448664    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-workload-certs\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.448766 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.448700    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-data\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.448766 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.448720    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759fj\" (UniqueName: \"kubernetes.io/projected/d547ad4b-a726-422e-aaeb-a5293584e4ac-kube-api-access-759fj\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.448766 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.448741    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d547ad4b-a726-422e-aaeb-a5293584e4ac-istiod-ca-cert\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.448766 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.448762    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-token\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.449195 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.448813    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-credential-socket\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.449195 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.448866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-workload-socket\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.449195 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.448904    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-podinfo\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.449195 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.448942    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-envoy\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.449195 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.449181    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-data\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.449487 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.449302    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-workload-socket\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.449487 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.449416    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-workload-certs\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.449487 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.449432    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-credential-socket\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.449611 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.449586    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d547ad4b-a726-422e-aaeb-a5293584e4ac-istiod-ca-cert\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.451524 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.451499    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-envoy\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.451766 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.451738    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-podinfo\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.457091 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.457068    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-token\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.457351 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.457331    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-759fj\" (UniqueName: \"kubernetes.io/projected/d547ad4b-a726-422e-aaeb-a5293584e4ac-kube-api-access-759fj\") pod \"e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") " pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.485047 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.485015    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:00.766249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.766157    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"]
Sep 10 15:33:00.769550 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:33:00.769521    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd547ad4b_a726_422e_aaeb_a5293584e4ac.slice/crio-1b62e2314cca39df2ff0698b376424f178217fa36bbf16a78ce53aa5037ae3e2 WatchSource:0}: Error finding container 1b62e2314cca39df2ff0698b376424f178217fa36bbf16a78ce53aa5037ae3e2: Status 404 returned error can't find the container with id 1b62e2314cca39df2ff0698b376424f178217fa36bbf16a78ce53aa5037ae3e2
Sep 10 15:33:00.772132 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.772102    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:33:00.975410 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:00.975372    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" event={"ID":"d547ad4b-a726-422e-aaeb-a5293584e4ac","Type":"ContainerStarted","Data":"1b62e2314cca39df2ff0698b376424f178217fa36bbf16a78ce53aa5037ae3e2"}
Sep 10 15:33:02.629307 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.629271    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"]
Sep 10 15:33:02.638154 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.638121    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:02.642640 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.642399    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-5fe45b-serving-cert\""
Sep 10 15:33:02.648327 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.646483    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"]
Sep 10 15:33:02.769639 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.769595    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:02.769860 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.769706    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e038425b-805f-4030-9cf7-662c4189cf63-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:02.769860 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.769845    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82tv6\" (UniqueName: \"kubernetes.io/projected/e038425b-805f-4030-9cf7-662c4189cf63-kube-api-access-82tv6\") pod \"maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:02.770001 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.769877    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:02.871262 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.871229    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82tv6\" (UniqueName: \"kubernetes.io/projected/e038425b-805f-4030-9cf7-662c4189cf63-kube-api-access-82tv6\") pod \"maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:02.871262 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.871274    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:02.871531 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.871330    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:02.871594 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:33:02.871562    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-serving-cert: secret "maas-api-e2e-shared-a-w2-5fe45b-serving-cert" not found
Sep 10 15:33:02.871649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.871592    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e038425b-805f-4030-9cf7-662c4189cf63-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:02.871649 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:33:02.871637    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-maas-api-tls podName:e038425b-805f-4030-9cf7-662c4189cf63 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:03.371616358 +0000 UTC m=+1045.787412162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-maas-api-tls") pod "maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl" (UID: "e038425b-805f-4030-9cf7-662c4189cf63") : secret "maas-api-e2e-shared-a-w2-5fe45b-serving-cert" not found
Sep 10 15:33:02.872311 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.872282    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e038425b-805f-4030-9cf7-662c4189cf63-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:02.874508 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.874489    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:02.880479 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:02.880427    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82tv6\" (UniqueName: \"kubernetes.io/projected/e038425b-805f-4030-9cf7-662c4189cf63-kube-api-access-82tv6\") pod \"maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:03.247168 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.247082    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"]
Sep 10 15:33:03.254505 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.254479    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"
Sep 10 15:33:03.257080 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.257052    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-5fe45b-dockercfg-9wbxk\""
Sep 10 15:33:03.257080 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.257073    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5fe45b\""
Sep 10 15:33:03.260580 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.260513    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"]
Sep 10 15:33:03.377093 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.377057    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pm8d\" (UniqueName: \"kubernetes.io/projected/acb591d7-4afd-424d-9773-4d5812165d7a-kube-api-access-2pm8d\") pod \"payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls\" (UID: \"acb591d7-4afd-424d-9773-4d5812165d7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"
Sep 10 15:33:03.377313 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.377169    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:03.377313 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.377221    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb591d7-4afd-424d-9773-4d5812165d7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls\" (UID: \"acb591d7-4afd-424d-9773-4d5812165d7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"
Sep 10 15:33:03.380297 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.380262    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:03.480163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.478697    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb591d7-4afd-424d-9773-4d5812165d7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls\" (UID: \"acb591d7-4afd-424d-9773-4d5812165d7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"
Sep 10 15:33:03.480163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.478825    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pm8d\" (UniqueName: \"kubernetes.io/projected/acb591d7-4afd-424d-9773-4d5812165d7a-kube-api-access-2pm8d\") pod \"payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls\" (UID: \"acb591d7-4afd-424d-9773-4d5812165d7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"
Sep 10 15:33:03.480163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.480020    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb591d7-4afd-424d-9773-4d5812165d7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls\" (UID: \"acb591d7-4afd-424d-9773-4d5812165d7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"
Sep 10 15:33:03.488123 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.488100    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pm8d\" (UniqueName: \"kubernetes.io/projected/acb591d7-4afd-424d-9773-4d5812165d7a-kube-api-access-2pm8d\") pod \"payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls\" (UID: \"acb591d7-4afd-424d-9773-4d5812165d7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"
Sep 10 15:33:03.556917 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.556823    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:03.568564 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.568524    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"
Sep 10 15:33:03.735509 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.735460    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 10 15:33:03.735834 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.735572    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 10 15:33:03.735834 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.735621    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 10 15:33:03.935528 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.935502    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"]
Sep 10 15:33:03.938615 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.938586    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"]
Sep 10 15:33:03.939126 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:33:03.939091    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb591d7_4afd_424d_9773_4d5812165d7a.slice/crio-7ff6212cbb041449ed67b96288469b3e17acd931437a6904d6694778e842d2ef WatchSource:0}: Error finding container 7ff6212cbb041449ed67b96288469b3e17acd931437a6904d6694778e842d2ef: Status 404 returned error can't find the container with id 7ff6212cbb041449ed67b96288469b3e17acd931437a6904d6694778e842d2ef
Sep 10 15:33:03.946345 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:33:03.946314    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode038425b_805f_4030_9cf7_662c4189cf63.slice/crio-c409e913767a789f4f52bb43140669bba5c636a148fe1bc94667358343e10f68 WatchSource:0}: Error finding container c409e913767a789f4f52bb43140669bba5c636a148fe1bc94667358343e10f68: Status 404 returned error can't find the container with id c409e913767a789f4f52bb43140669bba5c636a148fe1bc94667358343e10f68
Sep 10 15:33:03.995280 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.995248    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" event={"ID":"d547ad4b-a726-422e-aaeb-a5293584e4ac","Type":"ContainerStarted","Data":"146b263e0470607ba52442e620822d608f6d944366fdb80b3d75d449aa21d8b6"}
Sep 10 15:33:03.996622 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.996596    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl" event={"ID":"e038425b-805f-4030-9cf7-662c4189cf63","Type":"ContainerStarted","Data":"c409e913767a789f4f52bb43140669bba5c636a148fe1bc94667358343e10f68"}
Sep 10 15:33:03.997833 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:03.997782    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls" event={"ID":"acb591d7-4afd-424d-9773-4d5812165d7a","Type":"ContainerStarted","Data":"7ff6212cbb041449ed67b96288469b3e17acd931437a6904d6694778e842d2ef"}
Sep 10 15:33:04.017057 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:04.017010    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" podStartSLOduration=1.054219154 podStartE2EDuration="4.016992166s" totalImagesPullingTime="2.962773012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:04.015921237 +0000 UTC m=+1046.431717056" watchObservedRunningTime="2026-09-10 15:33:04.016992166 +0000 UTC m=+1046.432787984"
Sep 10 15:33:04.485461 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:04.485421    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:04.491315 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:04.491288    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:05.004663 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:05.004631    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl" event={"ID":"e038425b-805f-4030-9cf7-662c4189cf63","Type":"ContainerStarted","Data":"97f64844518cb8ed99b67399f7cb36dffe79868bfea3dd7c69427a101e7a6631"}
Sep 10 15:33:05.005167 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:05.004745    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:05.006452 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:05.006426    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls" event={"ID":"acb591d7-4afd-424d-9773-4d5812165d7a","Type":"ContainerStarted","Data":"81f2ca389eb61b474f0a36e667617b688460ac8d3d2c360a323aa45edabeb4d8"}
Sep 10 15:33:05.006586 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:05.006572    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"
Sep 10 15:33:05.006747 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:05.006734    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:05.007638 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:05.007620    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:05.029237 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:05.029191    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl" podStartSLOduration=2.6748612019999998 podStartE2EDuration="3.029177013s" totalImagesPullingTime="354.315811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:05.028140559 +0000 UTC m=+1047.443936377" watchObservedRunningTime="2026-09-10 15:33:05.029177013 +0000 UTC m=+1047.444972918"
Sep 10 15:33:05.079670 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:05.079621    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls" podStartSLOduration=2.079604851 podStartE2EDuration="2.079604851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:05.078878095 +0000 UTC m=+1047.494673932" watchObservedRunningTime="2026-09-10 15:33:05.079604851 +0000 UTC m=+1047.495400669"
Sep 10 15:33:07.084453 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.084417    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"]
Sep 10 15:33:07.087988 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.087959    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:07.090469 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.090445    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-62d2f7-serving-cert\""
Sep 10 15:33:07.101509 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.101476    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"]
Sep 10 15:33:07.112747 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.112723    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:07.112877 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.112756    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:07.112877 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.112805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:07.112877 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.112856    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncgzt\" (UniqueName: \"kubernetes.io/projected/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-kube-api-access-ncgzt\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:07.213949 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.213908    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:07.214157 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.213957    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgzt\" (UniqueName: \"kubernetes.io/projected/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-kube-api-access-ncgzt\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:07.214157 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.214054    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:07.214157 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.214085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:07.214348 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:33:07.214200    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-serving-cert: secret "maas-api-e2e-shared-a-w3-62d2f7-serving-cert" not found
Sep 10 15:33:07.214348 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:33:07.214271    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls podName:9fd90bcf-76fa-42c2-bd3a-8a1834d05d00 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:07.714249392 +0000 UTC m=+1050.130045189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls") pod "maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f" (UID: "9fd90bcf-76fa-42c2-bd3a-8a1834d05d00") : secret "maas-api-e2e-shared-a-w3-62d2f7-serving-cert" not found
Sep 10 15:33:07.214606 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.214582    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:07.216716 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.216695    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:07.222068 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.222046    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgzt\" (UniqueName: \"kubernetes.io/projected/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-kube-api-access-ncgzt\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:07.718256 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.718221    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:07.718422 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:33:07.718372    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-serving-cert: secret "maas-api-e2e-shared-a-w3-62d2f7-serving-cert" not found
Sep 10 15:33:07.718466 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:33:07.718440    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls podName:9fd90bcf-76fa-42c2-bd3a-8a1834d05d00 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:08.718423949 +0000 UTC m=+1051.134219746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls") pod "maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f" (UID: "9fd90bcf-76fa-42c2-bd3a-8a1834d05d00") : secret "maas-api-e2e-shared-a-w3-62d2f7-serving-cert" not found
Sep 10 15:33:07.921505 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.921466    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"]
Sep 10 15:33:07.925035 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.925016    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"
Sep 10 15:33:07.927391 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.927358    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-62d2f7\""
Sep 10 15:33:07.927519 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.927409    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-62d2f7-dockercfg-9qxl6\""
Sep 10 15:33:07.934617 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:07.934590    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"]
Sep 10 15:33:08.019536 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:08.019454    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53f4421-3ed3-4cad-b985-11184c2447fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv\" (UID: \"c53f4421-3ed3-4cad-b985-11184c2447fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"
Sep 10 15:33:08.019536 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:08.019485    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66swq\" (UniqueName: \"kubernetes.io/projected/c53f4421-3ed3-4cad-b985-11184c2447fb-kube-api-access-66swq\") pod \"payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv\" (UID: \"c53f4421-3ed3-4cad-b985-11184c2447fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"
Sep 10 15:33:08.119980 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:08.119942    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53f4421-3ed3-4cad-b985-11184c2447fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv\" (UID: \"c53f4421-3ed3-4cad-b985-11184c2447fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"
Sep 10 15:33:08.120441 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:08.119993    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66swq\" (UniqueName: \"kubernetes.io/projected/c53f4421-3ed3-4cad-b985-11184c2447fb-kube-api-access-66swq\") pod \"payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv\" (UID: \"c53f4421-3ed3-4cad-b985-11184c2447fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"
Sep 10 15:33:08.120808 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:08.120766    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53f4421-3ed3-4cad-b985-11184c2447fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv\" (UID: \"c53f4421-3ed3-4cad-b985-11184c2447fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"
Sep 10 15:33:08.128522 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:08.128500    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66swq\" (UniqueName: \"kubernetes.io/projected/c53f4421-3ed3-4cad-b985-11184c2447fb-kube-api-access-66swq\") pod \"payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv\" (UID: \"c53f4421-3ed3-4cad-b985-11184c2447fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"
Sep 10 15:33:08.237057 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:08.237018    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"
Sep 10 15:33:08.514619 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:08.514596    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"]
Sep 10 15:33:08.516810 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:33:08.516758    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53f4421_3ed3_4cad_b985_11184c2447fb.slice/crio-957035ada1314ac3721f318cd42b72812805198692f7545e79b9a7545c38022b WatchSource:0}: Error finding container 957035ada1314ac3721f318cd42b72812805198692f7545e79b9a7545c38022b: Status 404 returned error can't find the container with id 957035ada1314ac3721f318cd42b72812805198692f7545e79b9a7545c38022b
Sep 10 15:33:08.726066 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:08.726027    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:08.726250 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:33:08.726197    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-serving-cert: secret "maas-api-e2e-shared-a-w3-62d2f7-serving-cert" not found
Sep 10 15:33:08.726339 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:33:08.726283    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls podName:9fd90bcf-76fa-42c2-bd3a-8a1834d05d00 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:10.726260269 +0000 UTC m=+1053.142056067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls") pod "maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f" (UID: "9fd90bcf-76fa-42c2-bd3a-8a1834d05d00") : secret "maas-api-e2e-shared-a-w3-62d2f7-serving-cert" not found
Sep 10 15:33:09.029093 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:09.029053    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv" event={"ID":"c53f4421-3ed3-4cad-b985-11184c2447fb","Type":"ContainerStarted","Data":"bbfc2435985a1dc3446d12f7fbd159a2542d73a0fc703b7051b5112105fbae8a"}
Sep 10 15:33:09.029282 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:09.029123    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv" event={"ID":"c53f4421-3ed3-4cad-b985-11184c2447fb","Type":"ContainerStarted","Data":"957035ada1314ac3721f318cd42b72812805198692f7545e79b9a7545c38022b"}
Sep 10 15:33:09.029351 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:09.029276    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"
Sep 10 15:33:09.049800 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:09.049740    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv" podStartSLOduration=2.049724391 podStartE2EDuration="2.049724391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:09.047099807 +0000 UTC m=+1051.462895627" watchObservedRunningTime="2026-09-10 15:33:09.049724391 +0000 UTC m=+1051.465520210"
Sep 10 15:33:10.741979 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:10.741933    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:10.744714 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:10.744688    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:10.999354 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:10.999274    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:11.018540 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:11.018517    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:33:11.280763 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:11.280734    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"]
Sep 10 15:33:11.282766 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:33:11.282736    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd90bcf_76fa_42c2_bd3a_8a1834d05d00.slice/crio-18b137a5f3d405f211d2a5cd536a6d156ebed11c7a7008643f0b21c1b24622ab WatchSource:0}: Error finding container 18b137a5f3d405f211d2a5cd536a6d156ebed11c7a7008643f0b21c1b24622ab: Status 404 returned error can't find the container with id 18b137a5f3d405f211d2a5cd536a6d156ebed11c7a7008643f0b21c1b24622ab
Sep 10 15:33:12.055524 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:12.055419    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f" event={"ID":"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00","Type":"ContainerStarted","Data":"d633dbc6a86aa860b00765bd6e36a9edeeb1b712c611c2b27178870a86e89aa4"}
Sep 10 15:33:12.055524 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:12.055476    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f" event={"ID":"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00","Type":"ContainerStarted","Data":"18b137a5f3d405f211d2a5cd536a6d156ebed11c7a7008643f0b21c1b24622ab"}
Sep 10 15:33:12.055524 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:12.055522    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:12.075044 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:12.074989    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f" podStartSLOduration=4.754566016 podStartE2EDuration="5.074967151s" totalImagesPullingTime="320.401135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:12.074844558 +0000 UTC m=+1054.490640383" watchObservedRunningTime="2026-09-10 15:33:12.074967151 +0000 UTC m=+1054.490762994"
Sep 10 15:33:16.014436 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:16.014405    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"
Sep 10 15:33:18.065012 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:18.064981    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:33:20.036644 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:20.036610    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"
Sep 10 15:33:21.047980 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:21.047940    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 15:33:21.048430 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:21.048006    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 15:33:22.118805 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:22.118748    2573 generic.go:356] "Generic (PLEG): container finished" podID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerID="146b263e0470607ba52442e620822d608f6d944366fdb80b3d75d449aa21d8b6" exitCode=0
Sep 10 15:33:22.118805 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:22.118807    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" event={"ID":"d547ad4b-a726-422e-aaeb-a5293584e4ac","Type":"ContainerDied","Data":"146b263e0470607ba52442e620822d608f6d944366fdb80b3d75d449aa21d8b6"}
Sep 10 15:33:22.119272 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:22.119195    2573 scope.go:117] "RemoveContainer" containerID="146b263e0470607ba52442e620822d608f6d944366fdb80b3d75d449aa21d8b6"
Sep 10 15:33:22.120323 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:22.120277    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 10 15:33:22.120463 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:22.120377    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 10 15:33:22.120463 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:22.120424    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 10 15:33:23.126059 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:23.126024    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" event={"ID":"d547ad4b-a726-422e-aaeb-a5293584e4ac","Type":"ContainerStarted","Data":"e47c6c4abf76a02bb727954b8d814d9a349df124fca3e9e9788d1d5a48f74d3b"}
Sep 10 15:33:23.485955 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:23.485845    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:23.486261 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:23.486231    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 10 15:33:23.486360 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:23.486332    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 10 15:33:24.486363 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:24.486314    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 10 15:33:24.486931 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:24.486402    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 10 15:33:25.486921 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:25.486444    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 10 15:33:25.486921 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:25.486514    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 10 15:33:26.486076 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:26.486028    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 10 15:33:26.486302 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:26.486100    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 10 15:33:27.485546 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:27.485506    2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 10 15:33:27.486173 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:27.486138    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 10 15:33:28.561031 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:28.560993    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:28.561689 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:28.561665    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:28.562861 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:28.562839    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:33:31.628824 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.628718    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"]
Sep 10 15:33:31.645527 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.645497    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"]
Sep 10 15:33:31.645666 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.645617    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:31.648163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.648128    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-d7a2b3-serving-cert\""
Sep 10 15:33:31.732618 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.732582    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9336a78f-f603-415f-8a79-ca64fd6eb9ac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:31.732827 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.732625    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:31.732827 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.732677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:31.732827 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.732710    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw7c9\" (UniqueName: \"kubernetes.io/projected/9336a78f-f603-415f-8a79-ca64fd6eb9ac-kube-api-access-mw7c9\") pod \"maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:31.833839 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.833774    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:31.834029 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.833853    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:31.834029 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.833879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw7c9\" (UniqueName: \"kubernetes.io/projected/9336a78f-f603-415f-8a79-ca64fd6eb9ac-kube-api-access-mw7c9\") pod \"maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:31.834029 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:33:31.834001    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-serving-cert: secret "maas-api-e2e-shared-b-w2-d7a2b3-serving-cert" not found
Sep 10 15:33:31.834150 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.834007    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9336a78f-f603-415f-8a79-ca64fd6eb9ac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:31.834150 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:33:31.834084    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-maas-api-tls podName:9336a78f-f603-415f-8a79-ca64fd6eb9ac nodeName:}" failed. No retries permitted until 2026-09-10 15:33:32.334059179 +0000 UTC m=+1074.749854977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-maas-api-tls") pod "maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d" (UID: "9336a78f-f603-415f-8a79-ca64fd6eb9ac") : secret "maas-api-e2e-shared-b-w2-d7a2b3-serving-cert" not found
Sep 10 15:33:31.834642 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.834624    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9336a78f-f603-415f-8a79-ca64fd6eb9ac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:31.836564 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.836544    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:31.845662 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:31.845635    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw7c9\" (UniqueName: \"kubernetes.io/projected/9336a78f-f603-415f-8a79-ca64fd6eb9ac-kube-api-access-mw7c9\") pod \"maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:32.338982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:32.338949    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:32.341509 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:32.341477    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:32.558102 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:32.558063    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:32.836009 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:32.835980    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"]
Sep 10 15:33:32.838216 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:33:32.838189    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9336a78f_f603_415f_8a79_ca64fd6eb9ac.slice/crio-fe6b02fb06fe97aec622b7949ed613a8c698bab5d9e073ead76d7591f1305d07 WatchSource:0}: Error finding container fe6b02fb06fe97aec622b7949ed613a8c698bab5d9e073ead76d7591f1305d07: Status 404 returned error can't find the container with id fe6b02fb06fe97aec622b7949ed613a8c698bab5d9e073ead76d7591f1305d07
Sep 10 15:33:33.176863 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:33.176773    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d" event={"ID":"9336a78f-f603-415f-8a79-ca64fd6eb9ac","Type":"ContainerStarted","Data":"fe6b02fb06fe97aec622b7949ed613a8c698bab5d9e073ead76d7591f1305d07"}
Sep 10 15:33:34.183499 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:34.183465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d" event={"ID":"9336a78f-f603-415f-8a79-ca64fd6eb9ac","Type":"ContainerStarted","Data":"367b32b209166361219056e71a634e09471cd16e9e45ffbb14053a3a514694a1"}
Sep 10 15:33:34.183899 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:34.183596    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:34.203953 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:34.203905    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d" podStartSLOduration=2.890404871 podStartE2EDuration="3.203889327s" totalImagesPullingTime="313.484456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:34.202023075 +0000 UTC m=+1076.617818896" watchObservedRunningTime="2026-09-10 15:33:34.203889327 +0000 UTC m=+1076.619685146"
Sep 10 15:33:35.530280 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.530245    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"]
Sep 10 15:33:35.534021 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.533997    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:35.536208 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.536185    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-fc3e54-serving-cert\""
Sep 10 15:33:35.543770 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.543743    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"]
Sep 10 15:33:35.567473 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.567424    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:35.567669 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.567573    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76s7p\" (UniqueName: \"kubernetes.io/projected/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-kube-api-access-76s7p\") pod \"maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:35.567669 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.567637    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:35.567822 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.567677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:35.668217 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.668178    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:35.668440 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.668263    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76s7p\" (UniqueName: \"kubernetes.io/projected/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-kube-api-access-76s7p\") pod \"maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:35.668440 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.668311    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:35.668440 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:33:35.668416    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-serving-cert: secret "maas-api-e2e-shared-b-w3-fc3e54-serving-cert" not found
Sep 10 15:33:35.668623 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:33:35.668478    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-maas-api-tls podName:37a07ee7-ef8b-46b2-9c4a-273bc91a63fb nodeName:}" failed. No retries permitted until 2026-09-10 15:33:36.168457799 +0000 UTC m=+1078.584253600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-maas-api-tls") pod "maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm" (UID: "37a07ee7-ef8b-46b2-9c4a-273bc91a63fb") : secret "maas-api-e2e-shared-b-w3-fc3e54-serving-cert" not found
Sep 10 15:33:35.668691 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.668616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:35.669303 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.669273    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:35.671660 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.671635    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:35.677586 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:35.677563    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76s7p\" (UniqueName: \"kubernetes.io/projected/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-kube-api-access-76s7p\") pod \"maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:36.174040 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:36.173989    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:36.176976 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:36.176942    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:36.445706 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:36.445614    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:36.720674 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:36.720636    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"]
Sep 10 15:33:36.723237 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:33:36.723209    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a07ee7_ef8b_46b2_9c4a_273bc91a63fb.slice/crio-96ae830847327f56e432384edefb7c7a47a36f40951e1a0716f810d823fb74ae WatchSource:0}: Error finding container 96ae830847327f56e432384edefb7c7a47a36f40951e1a0716f810d823fb74ae: Status 404 returned error can't find the container with id 96ae830847327f56e432384edefb7c7a47a36f40951e1a0716f810d823fb74ae
Sep 10 15:33:37.200573 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:37.200532    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm" event={"ID":"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb","Type":"ContainerStarted","Data":"e7011929cac828bab4745c249768efaf83e4448d376e781d6a338db3b6bb7dc6"}
Sep 10 15:33:37.200745 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:37.200598    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:33:37.200745 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:37.200617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm" event={"ID":"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb","Type":"ContainerStarted","Data":"96ae830847327f56e432384edefb7c7a47a36f40951e1a0716f810d823fb74ae"}
Sep 10 15:33:37.219369 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:37.219313    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm" podStartSLOduration=1.919176825 podStartE2EDuration="2.219296742s" totalImagesPullingTime="300.119917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:37.218742597 +0000 UTC m=+1079.634538415" watchObservedRunningTime="2026-09-10 15:33:37.219296742 +0000 UTC m=+1079.635092561"
Sep 10 15:33:40.194430 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:40.194395    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:33:43.212010 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:33:43.211976    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:34:00.396001 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:00.395969    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7821-a4ed95"]
Sep 10 15:34:00.400418 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:00.400399    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-a4ed95"
Sep 10 15:34:00.402884 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:00.402862    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h7f4v\""
Sep 10 15:34:00.409498 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:00.409474    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7821-a4ed95"]
Sep 10 15:34:00.497877 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:00.497839    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrjbf\" (UniqueName: \"kubernetes.io/projected/d431efd0-c797-46ff-b865-195b04ef8c49-kube-api-access-qrjbf\") pod \"test-curl-7821-a4ed95\" (UID: \"d431efd0-c797-46ff-b865-195b04ef8c49\") " pod="opendatahub/test-curl-7821-a4ed95"
Sep 10 15:34:00.598634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:00.598597    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrjbf\" (UniqueName: \"kubernetes.io/projected/d431efd0-c797-46ff-b865-195b04ef8c49-kube-api-access-qrjbf\") pod \"test-curl-7821-a4ed95\" (UID: \"d431efd0-c797-46ff-b865-195b04ef8c49\") " pod="opendatahub/test-curl-7821-a4ed95"
Sep 10 15:34:00.608112 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:00.608082    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrjbf\" (UniqueName: \"kubernetes.io/projected/d431efd0-c797-46ff-b865-195b04ef8c49-kube-api-access-qrjbf\") pod \"test-curl-7821-a4ed95\" (UID: \"d431efd0-c797-46ff-b865-195b04ef8c49\") " pod="opendatahub/test-curl-7821-a4ed95"
Sep 10 15:34:00.709953 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:00.709854    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-a4ed95"
Sep 10 15:34:00.987000 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:00.986974    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7821-a4ed95"]
Sep 10 15:34:00.989588 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:34:00.989564    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd431efd0_c797_46ff_b865_195b04ef8c49.slice/crio-9f24cd2aa54d9640a68fbf9e76ab320691e10ce8b7750bab6fff8277fb439d99 WatchSource:0}: Error finding container 9f24cd2aa54d9640a68fbf9e76ab320691e10ce8b7750bab6fff8277fb439d99: Status 404 returned error can't find the container with id 9f24cd2aa54d9640a68fbf9e76ab320691e10ce8b7750bab6fff8277fb439d99
Sep 10 15:34:01.322854 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:01.322764    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7821-a4ed95" event={"ID":"d431efd0-c797-46ff-b865-195b04ef8c49","Type":"ContainerStarted","Data":"a37e039c53c4d15a5da75c25f0ce44034d61c3ac1d1bf7e6b4dcf5c342b9a62e"}
Sep 10 15:34:01.322854 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:01.322826    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7821-a4ed95" event={"ID":"d431efd0-c797-46ff-b865-195b04ef8c49","Type":"ContainerStarted","Data":"9f24cd2aa54d9640a68fbf9e76ab320691e10ce8b7750bab6fff8277fb439d99"}
Sep 10 15:34:01.343084 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:01.343032    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7821-a4ed95" podStartSLOduration=1.34301604 podStartE2EDuration="1.34301604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:34:01.34191859 +0000 UTC m=+1103.757714408" watchObservedRunningTime="2026-09-10 15:34:01.34301604 +0000 UTC m=+1103.758811882"
Sep 10 15:34:01.962829 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:01.960987    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7821-a4ed95"]
Sep 10 15:34:01.971974 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:01.971937    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7821-a4ed95"]
Sep 10 15:34:02.328413 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:34:02.328379    2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7821-a4ed95" container="test-curl-7821-a4ed95"
Sep 10 15:34:03.333150 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:03.333104    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7821-a4ed95" podUID="d431efd0-c797-46ff-b865-195b04ef8c49" containerName="test-curl-7821-a4ed95" containerID="cri-o://a37e039c53c4d15a5da75c25f0ce44034d61c3ac1d1bf7e6b4dcf5c342b9a62e" gracePeriod=2
Sep 10 15:34:05.513189 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:05.513163    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-a4ed95"
Sep 10 15:34:05.643734 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:05.643642    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrjbf\" (UniqueName: \"kubernetes.io/projected/d431efd0-c797-46ff-b865-195b04ef8c49-kube-api-access-qrjbf\") pod \"d431efd0-c797-46ff-b865-195b04ef8c49\" (UID: \"d431efd0-c797-46ff-b865-195b04ef8c49\") "
Sep 10 15:34:05.645979 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:05.645957    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d431efd0-c797-46ff-b865-195b04ef8c49-kube-api-access-qrjbf" (OuterVolumeSpecName: "kube-api-access-qrjbf") pod "d431efd0-c797-46ff-b865-195b04ef8c49" (UID: "d431efd0-c797-46ff-b865-195b04ef8c49"). InnerVolumeSpecName "kube-api-access-qrjbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:34:05.745029 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:05.744987    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrjbf\" (UniqueName: \"kubernetes.io/projected/d431efd0-c797-46ff-b865-195b04ef8c49-kube-api-access-qrjbf\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:34:06.126966 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:06.126937    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d431efd0-c797-46ff-b865-195b04ef8c49" path="/var/lib/kubelet/pods/d431efd0-c797-46ff-b865-195b04ef8c49/volumes"
Sep 10 15:34:06.351405 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:06.351370    2573 generic.go:356] "Generic (PLEG): container finished" podID="d431efd0-c797-46ff-b865-195b04ef8c49" containerID="a37e039c53c4d15a5da75c25f0ce44034d61c3ac1d1bf7e6b4dcf5c342b9a62e" exitCode=137
Sep 10 15:34:06.351593 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:06.351414    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7821-a4ed95"
Sep 10 15:34:06.351593 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:06.351467    2573 scope.go:117] "RemoveContainer" containerID="a37e039c53c4d15a5da75c25f0ce44034d61c3ac1d1bf7e6b4dcf5c342b9a62e"
Sep 10 15:34:06.360893 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:06.360871    2573 scope.go:117] "RemoveContainer" containerID="a37e039c53c4d15a5da75c25f0ce44034d61c3ac1d1bf7e6b4dcf5c342b9a62e"
Sep 10 15:34:06.361182 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:34:06.361160    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37e039c53c4d15a5da75c25f0ce44034d61c3ac1d1bf7e6b4dcf5c342b9a62e\": container with ID starting with a37e039c53c4d15a5da75c25f0ce44034d61c3ac1d1bf7e6b4dcf5c342b9a62e not found: ID does not exist" containerID="a37e039c53c4d15a5da75c25f0ce44034d61c3ac1d1bf7e6b4dcf5c342b9a62e"
Sep 10 15:34:06.361237 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:06.361194    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37e039c53c4d15a5da75c25f0ce44034d61c3ac1d1bf7e6b4dcf5c342b9a62e"} err="failed to get container status \"a37e039c53c4d15a5da75c25f0ce44034d61c3ac1d1bf7e6b4dcf5c342b9a62e\": rpc error: code = NotFound desc = could not find container \"a37e039c53c4d15a5da75c25f0ce44034d61c3ac1d1bf7e6b4dcf5c342b9a62e\": container with ID starting with a37e039c53c4d15a5da75c25f0ce44034d61c3ac1d1bf7e6b4dcf5c342b9a62e not found: ID does not exist"
Sep 10 15:34:09.923577 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:09.923545    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"]
Sep 10 15:34:09.924106 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:09.924061    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d431efd0-c797-46ff-b865-195b04ef8c49" containerName="test-curl-7821-a4ed95"
Sep 10 15:34:09.924106 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:09.924080    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d431efd0-c797-46ff-b865-195b04ef8c49" containerName="test-curl-7821-a4ed95"
Sep 10 15:34:09.924194 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:09.924173    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d431efd0-c797-46ff-b865-195b04ef8c49" containerName="test-curl-7821-a4ed95"
Sep 10 15:34:09.928481 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:09.928462    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:09.932420 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:09.932397    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-26935da7-serving-cert\""
Sep 10 15:34:09.947023 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:09.946994    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"]
Sep 10 15:34:09.983849 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:09.983812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee58980-1f0a-4c03-b0be-b4b410b034fb-openshift-service-ca\") pod \"maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:09.984009 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:09.983867    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee58980-1f0a-4c03-b0be-b4b410b034fb-maas-api-tls\") pod \"maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:09.984009 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:09.983885    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrhrq\" (UniqueName: \"kubernetes.io/projected/8ee58980-1f0a-4c03-b0be-b4b410b034fb-kube-api-access-qrhrq\") pod \"maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:09.984009 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:09.983918    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee58980-1f0a-4c03-b0be-b4b410b034fb-metrics-certs\") pod \"maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:10.084632 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:10.084596    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee58980-1f0a-4c03-b0be-b4b410b034fb-openshift-service-ca\") pod \"maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:10.084825 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:10.084652    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee58980-1f0a-4c03-b0be-b4b410b034fb-maas-api-tls\") pod \"maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:10.084825 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:10.084673    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrhrq\" (UniqueName: \"kubernetes.io/projected/8ee58980-1f0a-4c03-b0be-b4b410b034fb-kube-api-access-qrhrq\") pod \"maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:10.084825 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:10.084710    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee58980-1f0a-4c03-b0be-b4b410b034fb-metrics-certs\") pod \"maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:10.085352 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:10.085321    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee58980-1f0a-4c03-b0be-b4b410b034fb-openshift-service-ca\") pod \"maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:10.087274 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:10.087254    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee58980-1f0a-4c03-b0be-b4b410b034fb-metrics-certs\") pod \"maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:10.087370 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:10.087289    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee58980-1f0a-4c03-b0be-b4b410b034fb-maas-api-tls\") pod \"maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:10.097877 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:10.097855    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrhrq\" (UniqueName: \"kubernetes.io/projected/8ee58980-1f0a-4c03-b0be-b4b410b034fb-kube-api-access-qrhrq\") pod \"maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:10.238477 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:10.238385    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:10.537402 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:10.537374    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"]
Sep 10 15:34:10.540588 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:34:10.540555    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee58980_1f0a_4c03_b0be_b4b410b034fb.slice/crio-4e73f6376c776bf76216ce14edf4b244b00c6f86c021b5105021871fc337e27f WatchSource:0}: Error finding container 4e73f6376c776bf76216ce14edf4b244b00c6f86c021b5105021871fc337e27f: Status 404 returned error can't find the container with id 4e73f6376c776bf76216ce14edf4b244b00c6f86c021b5105021871fc337e27f
Sep 10 15:34:11.230886 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.230848    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"]
Sep 10 15:34:11.234611 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.234585    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"
Sep 10 15:34:11.237634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.237599    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-26935da7\""
Sep 10 15:34:11.237769 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.237671    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-26935da7-dockercfg-l6knj\""
Sep 10 15:34:11.247553 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.247529    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"]
Sep 10 15:34:11.294044 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.294001    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjwf4\" (UniqueName: \"kubernetes.io/projected/0f960473-ffd5-4da1-b7a2-2fdbbe7597d8-kube-api-access-qjwf4\") pod \"payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6\" (UID: \"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"
Sep 10 15:34:11.294198 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.294129    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f960473-ffd5-4da1-b7a2-2fdbbe7597d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6\" (UID: \"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"
Sep 10 15:34:11.378847 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.378805    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j" event={"ID":"8ee58980-1f0a-4c03-b0be-b4b410b034fb","Type":"ContainerStarted","Data":"68c998d5f082a5e0a1b0e965ac4f186e8828928850841a9cc23f5a73788ea703"}
Sep 10 15:34:11.379035 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.378873    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:11.379035 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.378891    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j" event={"ID":"8ee58980-1f0a-4c03-b0be-b4b410b034fb","Type":"ContainerStarted","Data":"4e73f6376c776bf76216ce14edf4b244b00c6f86c021b5105021871fc337e27f"}
Sep 10 15:34:11.395157 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.395121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjwf4\" (UniqueName: \"kubernetes.io/projected/0f960473-ffd5-4da1-b7a2-2fdbbe7597d8-kube-api-access-qjwf4\") pod \"payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6\" (UID: \"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"
Sep 10 15:34:11.395338 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.395193    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f960473-ffd5-4da1-b7a2-2fdbbe7597d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6\" (UID: \"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"
Sep 10 15:34:11.396027 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.396004    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f960473-ffd5-4da1-b7a2-2fdbbe7597d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6\" (UID: \"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"
Sep 10 15:34:11.416906 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.416852    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j" podStartSLOduration=1.894758012 podStartE2EDuration="2.41683615s" totalImagesPullingTime="522.078138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:34:11.416366019 +0000 UTC m=+1113.832161854" watchObservedRunningTime="2026-09-10 15:34:11.41683615 +0000 UTC m=+1113.832631968"
Sep 10 15:34:11.419447 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.419420    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjwf4\" (UniqueName: \"kubernetes.io/projected/0f960473-ffd5-4da1-b7a2-2fdbbe7597d8-kube-api-access-qjwf4\") pod \"payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6\" (UID: \"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"
Sep 10 15:34:11.546089 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.546044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"
Sep 10 15:34:11.836539 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:11.836507    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"]
Sep 10 15:34:11.837650 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:34:11.837619    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f960473_ffd5_4da1_b7a2_2fdbbe7597d8.slice/crio-dc23e054622d11f051157e52a610746bd0a8b258d4e9f7c9cb9c844612a98820 WatchSource:0}: Error finding container dc23e054622d11f051157e52a610746bd0a8b258d4e9f7c9cb9c844612a98820: Status 404 returned error can't find the container with id dc23e054622d11f051157e52a610746bd0a8b258d4e9f7c9cb9c844612a98820
Sep 10 15:34:12.386428 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:12.386375    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6" event={"ID":"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8","Type":"ContainerStarted","Data":"01687f02474edf0a9920a52ba96e565d0c882019a3681868fb0cc47be93dcd45"}
Sep 10 15:34:12.386951 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:12.386449    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"
Sep 10 15:34:12.386951 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:12.386474    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6" event={"ID":"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8","Type":"ContainerStarted","Data":"dc23e054622d11f051157e52a610746bd0a8b258d4e9f7c9cb9c844612a98820"}
Sep 10 15:34:12.413008 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:12.412953    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6" podStartSLOduration=1.412936443 podStartE2EDuration="1.412936443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:34:12.411523507 +0000 UTC m=+1114.827319327" watchObservedRunningTime="2026-09-10 15:34:12.412936443 +0000 UTC m=+1114.828732263"
Sep 10 15:34:17.392971 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:17.392938    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:34:23.394734 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:23.394694    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"
Sep 10 15:34:43.459403 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.459368    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"]
Sep 10 15:34:43.464205 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.464180    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.466805 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.466764    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-26935da7\"/\"e2e-lifecycle-model-26935da7-kserve-self-signed-certs\""
Sep 10 15:34:43.467406 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.467381    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-26935da7\"/\"kube-root-ca.crt\""
Sep 10 15:34:43.467522 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.467458    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-26935da7\"/\"default-dockercfg-bq2b2\""
Sep 10 15:34:43.467679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.467631    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-26935da7\"/\"openshift-service-ca.crt\""
Sep 10 15:34:43.479822 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.479778    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"]
Sep 10 15:34:43.594315 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.594283    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9kz7\" (UniqueName: \"kubernetes.io/projected/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-kube-api-access-x9kz7\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.594517 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.594319    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-dshm\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.594517 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.594362    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-model-cache\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.594517 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.594418    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-home\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.594517 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.594445    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-tmp-dir\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.594517 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.594480    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-tls-certs\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.695027 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.694995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kz7\" (UniqueName: \"kubernetes.io/projected/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-kube-api-access-x9kz7\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.695213 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.695034    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-dshm\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.695213 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.695193    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-model-cache\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.695336 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.695259    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-home\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.695336 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.695294    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-tmp-dir\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.695336 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.695324    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-tls-certs\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.695599 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.695572    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-model-cache\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.695730 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.695673    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-home\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.696104 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.696081    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-tmp-dir\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.697679 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.697656    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-dshm\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.698130 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.698114    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-tls-certs\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.704882 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.704858    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9kz7\" (UniqueName: \"kubernetes.io/projected/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-kube-api-access-x9kz7\") pod \"e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") " pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:43.775154 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:43.775065    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:44.061238 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:44.061212    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"]
Sep 10 15:34:44.062959 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:34:44.062927    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1708e2a4_3889_4b40_b5e5_cd949df6c0e2.slice/crio-f3e55e9e1a3d107886a64caf964b946471f31b8b5598df129898e5761185fbd9 WatchSource:0}: Error finding container f3e55e9e1a3d107886a64caf964b946471f31b8b5598df129898e5761185fbd9: Status 404 returned error can't find the container with id f3e55e9e1a3d107886a64caf964b946471f31b8b5598df129898e5761185fbd9
Sep 10 15:34:44.557927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:44.557883    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs" event={"ID":"1708e2a4-3889-4b40-b5e5-cd949df6c0e2","Type":"ContainerStarted","Data":"0ceb932038a14f897008f2e1155573abee6c31a6547a7e8640fcd9c0106479b6"}
Sep 10 15:34:44.557927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:44.557934    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs" event={"ID":"1708e2a4-3889-4b40-b5e5-cd949df6c0e2","Type":"ContainerStarted","Data":"f3e55e9e1a3d107886a64caf964b946471f31b8b5598df129898e5761185fbd9"}
Sep 10 15:34:44.581317 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:44.581257    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs" podStartSLOduration=1.5812418990000001 podStartE2EDuration="1.581241899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:34:44.578141406 +0000 UTC m=+1146.993937216" watchObservedRunningTime="2026-09-10 15:34:44.581241899 +0000 UTC m=+1146.997037717"
Sep 10 15:34:53.775251 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:53.775206    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:53.776025 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:53.775988    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:53.789000 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:53.788978    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:34:54.621537 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:34:54.621506    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:36:14.530983 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.530950    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"]
Sep 10 15:36:14.531582 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.531181    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j" podUID="8ee58980-1f0a-4c03-b0be-b4b410b034fb" containerName="maas-api" containerID="cri-o://68c998d5f082a5e0a1b0e965ac4f186e8828928850841a9cc23f5a73788ea703" gracePeriod=30
Sep 10 15:36:14.596615 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.596573    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"]
Sep 10 15:36:14.596988 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.596955    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6" podUID="0f960473-ffd5-4da1-b7a2-2fdbbe7597d8" containerName="payload-pre-processing" containerID="cri-o://01687f02474edf0a9920a52ba96e565d0c882019a3681868fb0cc47be93dcd45" gracePeriod=30
Sep 10 15:36:14.858705 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.858670    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:36:14.896325 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.896298    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"
Sep 10 15:36:14.929835 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.929773    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrhrq\" (UniqueName: \"kubernetes.io/projected/8ee58980-1f0a-4c03-b0be-b4b410b034fb-kube-api-access-qrhrq\") pod \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") "
Sep 10 15:36:14.930034 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.929890    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee58980-1f0a-4c03-b0be-b4b410b034fb-metrics-certs\") pod \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") "
Sep 10 15:36:14.930034 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.929926    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee58980-1f0a-4c03-b0be-b4b410b034fb-openshift-service-ca\") pod \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") "
Sep 10 15:36:14.930034 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.929941    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee58980-1f0a-4c03-b0be-b4b410b034fb-maas-api-tls\") pod \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\" (UID: \"8ee58980-1f0a-4c03-b0be-b4b410b034fb\") "
Sep 10 15:36:14.930315 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.930289    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee58980-1f0a-4c03-b0be-b4b410b034fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ee58980-1f0a-4c03-b0be-b4b410b034fb" (UID: "8ee58980-1f0a-4c03-b0be-b4b410b034fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:36:14.932162 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.932132    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee58980-1f0a-4c03-b0be-b4b410b034fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ee58980-1f0a-4c03-b0be-b4b410b034fb" (UID: "8ee58980-1f0a-4c03-b0be-b4b410b034fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:36:14.932162 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.932148    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee58980-1f0a-4c03-b0be-b4b410b034fb-kube-api-access-qrhrq" (OuterVolumeSpecName: "kube-api-access-qrhrq") pod "8ee58980-1f0a-4c03-b0be-b4b410b034fb" (UID: "8ee58980-1f0a-4c03-b0be-b4b410b034fb"). InnerVolumeSpecName "kube-api-access-qrhrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:36:14.932342 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:14.932274    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee58980-1f0a-4c03-b0be-b4b410b034fb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8ee58980-1f0a-4c03-b0be-b4b410b034fb" (UID: "8ee58980-1f0a-4c03-b0be-b4b410b034fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:36:15.028594 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.028558    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f960473-ffd5-4da1-b7a2-2fdbbe7597d8" containerID="01687f02474edf0a9920a52ba96e565d0c882019a3681868fb0cc47be93dcd45" exitCode=0
Sep 10 15:36:15.028809 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.028599    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"
Sep 10 15:36:15.028809 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.028653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6" event={"ID":"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8","Type":"ContainerDied","Data":"01687f02474edf0a9920a52ba96e565d0c882019a3681868fb0cc47be93dcd45"}
Sep 10 15:36:15.028809 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.028710    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6" event={"ID":"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8","Type":"ContainerDied","Data":"dc23e054622d11f051157e52a610746bd0a8b258d4e9f7c9cb9c844612a98820"}
Sep 10 15:36:15.028809 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.028736    2573 scope.go:117] "RemoveContainer" containerID="01687f02474edf0a9920a52ba96e565d0c882019a3681868fb0cc47be93dcd45"
Sep 10 15:36:15.030071 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.030051    2573 generic.go:356] "Generic (PLEG): container finished" podID="8ee58980-1f0a-4c03-b0be-b4b410b034fb" containerID="68c998d5f082a5e0a1b0e965ac4f186e8828928850841a9cc23f5a73788ea703" exitCode=0
Sep 10 15:36:15.030186 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.030098    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"
Sep 10 15:36:15.030186 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.030113    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j" event={"ID":"8ee58980-1f0a-4c03-b0be-b4b410b034fb","Type":"ContainerDied","Data":"68c998d5f082a5e0a1b0e965ac4f186e8828928850841a9cc23f5a73788ea703"}
Sep 10 15:36:15.030186 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.030156    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j" event={"ID":"8ee58980-1f0a-4c03-b0be-b4b410b034fb","Type":"ContainerDied","Data":"4e73f6376c776bf76216ce14edf4b244b00c6f86c021b5105021871fc337e27f"}
Sep 10 15:36:15.030708 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.030686    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f960473-ffd5-4da1-b7a2-2fdbbe7597d8-plugins-config-volume\") pod \"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8\" (UID: \"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8\") "
Sep 10 15:36:15.030901 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.030878    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjwf4\" (UniqueName: \"kubernetes.io/projected/0f960473-ffd5-4da1-b7a2-2fdbbe7597d8-kube-api-access-qjwf4\") pod \"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8\" (UID: \"0f960473-ffd5-4da1-b7a2-2fdbbe7597d8\") "
Sep 10 15:36:15.031078 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.031055    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f960473-ffd5-4da1-b7a2-2fdbbe7597d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f960473-ffd5-4da1-b7a2-2fdbbe7597d8" (UID: "0f960473-ffd5-4da1-b7a2-2fdbbe7597d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:36:15.031170 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.031157    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ee58980-1f0a-4c03-b0be-b4b410b034fb-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:36:15.031231 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.031173    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee58980-1f0a-4c03-b0be-b4b410b034fb-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:36:15.031231 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.031188    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ee58980-1f0a-4c03-b0be-b4b410b034fb-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:36:15.031231 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.031200    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f960473-ffd5-4da1-b7a2-2fdbbe7597d8-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:36:15.031231 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.031208    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrhrq\" (UniqueName: \"kubernetes.io/projected/8ee58980-1f0a-4c03-b0be-b4b410b034fb-kube-api-access-qrhrq\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:36:15.033088 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.033062    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f960473-ffd5-4da1-b7a2-2fdbbe7597d8-kube-api-access-qjwf4" (OuterVolumeSpecName: "kube-api-access-qjwf4") pod "0f960473-ffd5-4da1-b7a2-2fdbbe7597d8" (UID: "0f960473-ffd5-4da1-b7a2-2fdbbe7597d8"). InnerVolumeSpecName "kube-api-access-qjwf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:36:15.038241 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.038226    2573 scope.go:117] "RemoveContainer" containerID="01687f02474edf0a9920a52ba96e565d0c882019a3681868fb0cc47be93dcd45"
Sep 10 15:36:15.038538 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:36:15.038516    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01687f02474edf0a9920a52ba96e565d0c882019a3681868fb0cc47be93dcd45\": container with ID starting with 01687f02474edf0a9920a52ba96e565d0c882019a3681868fb0cc47be93dcd45 not found: ID does not exist" containerID="01687f02474edf0a9920a52ba96e565d0c882019a3681868fb0cc47be93dcd45"
Sep 10 15:36:15.038608 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.038546    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01687f02474edf0a9920a52ba96e565d0c882019a3681868fb0cc47be93dcd45"} err="failed to get container status \"01687f02474edf0a9920a52ba96e565d0c882019a3681868fb0cc47be93dcd45\": rpc error: code = NotFound desc = could not find container \"01687f02474edf0a9920a52ba96e565d0c882019a3681868fb0cc47be93dcd45\": container with ID starting with 01687f02474edf0a9920a52ba96e565d0c882019a3681868fb0cc47be93dcd45 not found: ID does not exist"
Sep 10 15:36:15.038608 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.038562    2573 scope.go:117] "RemoveContainer" containerID="68c998d5f082a5e0a1b0e965ac4f186e8828928850841a9cc23f5a73788ea703"
Sep 10 15:36:15.047307 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.047292    2573 scope.go:117] "RemoveContainer" containerID="68c998d5f082a5e0a1b0e965ac4f186e8828928850841a9cc23f5a73788ea703"
Sep 10 15:36:15.047570 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:36:15.047552    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c998d5f082a5e0a1b0e965ac4f186e8828928850841a9cc23f5a73788ea703\": container with ID starting with 68c998d5f082a5e0a1b0e965ac4f186e8828928850841a9cc23f5a73788ea703 not found: ID does not exist" containerID="68c998d5f082a5e0a1b0e965ac4f186e8828928850841a9cc23f5a73788ea703"
Sep 10 15:36:15.047608 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.047577    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c998d5f082a5e0a1b0e965ac4f186e8828928850841a9cc23f5a73788ea703"} err="failed to get container status \"68c998d5f082a5e0a1b0e965ac4f186e8828928850841a9cc23f5a73788ea703\": rpc error: code = NotFound desc = could not find container \"68c998d5f082a5e0a1b0e965ac4f186e8828928850841a9cc23f5a73788ea703\": container with ID starting with 68c998d5f082a5e0a1b0e965ac4f186e8828928850841a9cc23f5a73788ea703 not found: ID does not exist"
Sep 10 15:36:15.052000 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.051979    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"]
Sep 10 15:36:15.056402 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.056343    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-26935da7-6f55bdfbcd-lml9j"]
Sep 10 15:36:15.131947 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.131917    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjwf4\" (UniqueName: \"kubernetes.io/projected/0f960473-ffd5-4da1-b7a2-2fdbbe7597d8-kube-api-access-qjwf4\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:36:15.353110 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.353079    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"]
Sep 10 15:36:15.357151 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:15.357127    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26935da7-6d777867f4-x6sb6"]
Sep 10 15:36:16.126612 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:16.126568    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f960473-ffd5-4da1-b7a2-2fdbbe7597d8" path="/var/lib/kubelet/pods/0f960473-ffd5-4da1-b7a2-2fdbbe7597d8/volumes"
Sep 10 15:36:16.127195 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:16.127172    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ee58980-1f0a-4c03-b0be-b4b410b034fb" path="/var/lib/kubelet/pods/8ee58980-1f0a-4c03-b0be-b4b410b034fb/volumes"
Sep 10 15:36:22.775664 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:22.775612    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"]
Sep 10 15:36:37.796426 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:37.796307    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs" podUID="1708e2a4-3889-4b40-b5e5-cd949df6c0e2" containerName="main" containerID="cri-o://0ceb932038a14f897008f2e1155573abee6c31a6547a7e8640fcd9c0106479b6" gracePeriod=45
Sep 10 15:36:38.061784 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.061757    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:36:38.144252 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.144227    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-dshm\") pod \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") "
Sep 10 15:36:38.144252 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.144256    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-home\") pod \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") "
Sep 10 15:36:38.144493 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.144273    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-tmp-dir\") pod \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") "
Sep 10 15:36:38.144493 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.144322    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-model-cache\") pod \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") "
Sep 10 15:36:38.144493 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.144399    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-tls-certs\") pod \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") "
Sep 10 15:36:38.144493 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.144455    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9kz7\" (UniqueName: \"kubernetes.io/projected/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-kube-api-access-x9kz7\") pod \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\" (UID: \"1708e2a4-3889-4b40-b5e5-cd949df6c0e2\") "
Sep 10 15:36:38.144714 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.144556    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-home" (OuterVolumeSpecName: "home") pod "1708e2a4-3889-4b40-b5e5-cd949df6c0e2" (UID: "1708e2a4-3889-4b40-b5e5-cd949df6c0e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:36:38.144714 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.144572    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1708e2a4-3889-4b40-b5e5-cd949df6c0e2" (UID: "1708e2a4-3889-4b40-b5e5-cd949df6c0e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:36:38.144848 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.144810    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "1708e2a4-3889-4b40-b5e5-cd949df6c0e2" (UID: "1708e2a4-3889-4b40-b5e5-cd949df6c0e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:36:38.144919 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.144861    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-home\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:36:38.144919 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.144875    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-tmp-dir\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:36:38.146846 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.146818    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-dshm" (OuterVolumeSpecName: "dshm") pod "1708e2a4-3889-4b40-b5e5-cd949df6c0e2" (UID: "1708e2a4-3889-4b40-b5e5-cd949df6c0e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:36:38.147053 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.147031    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-kube-api-access-x9kz7" (OuterVolumeSpecName: "kube-api-access-x9kz7") pod "1708e2a4-3889-4b40-b5e5-cd949df6c0e2" (UID: "1708e2a4-3889-4b40-b5e5-cd949df6c0e2"). InnerVolumeSpecName "kube-api-access-x9kz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:36:38.147294 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.147271    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1708e2a4-3889-4b40-b5e5-cd949df6c0e2" (UID: "1708e2a4-3889-4b40-b5e5-cd949df6c0e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:36:38.149378 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.149350    2573 generic.go:356] "Generic (PLEG): container finished" podID="1708e2a4-3889-4b40-b5e5-cd949df6c0e2" containerID="0ceb932038a14f897008f2e1155573abee6c31a6547a7e8640fcd9c0106479b6" exitCode=0
Sep 10 15:36:38.149483 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.149450    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"
Sep 10 15:36:38.149525 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.149494    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs" event={"ID":"1708e2a4-3889-4b40-b5e5-cd949df6c0e2","Type":"ContainerDied","Data":"0ceb932038a14f897008f2e1155573abee6c31a6547a7e8640fcd9c0106479b6"}
Sep 10 15:36:38.149569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.149546    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs" event={"ID":"1708e2a4-3889-4b40-b5e5-cd949df6c0e2","Type":"ContainerDied","Data":"f3e55e9e1a3d107886a64caf964b946471f31b8b5598df129898e5761185fbd9"}
Sep 10 15:36:38.149602 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.149575    2573 scope.go:117] "RemoveContainer" containerID="0ceb932038a14f897008f2e1155573abee6c31a6547a7e8640fcd9c0106479b6"
Sep 10 15:36:38.165119 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.165102    2573 scope.go:117] "RemoveContainer" containerID="0ceb932038a14f897008f2e1155573abee6c31a6547a7e8640fcd9c0106479b6"
Sep 10 15:36:38.165387 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:36:38.165371    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ceb932038a14f897008f2e1155573abee6c31a6547a7e8640fcd9c0106479b6\": container with ID starting with 0ceb932038a14f897008f2e1155573abee6c31a6547a7e8640fcd9c0106479b6 not found: ID does not exist" containerID="0ceb932038a14f897008f2e1155573abee6c31a6547a7e8640fcd9c0106479b6"
Sep 10 15:36:38.165459 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.165395    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ceb932038a14f897008f2e1155573abee6c31a6547a7e8640fcd9c0106479b6"} err="failed to get container status \"0ceb932038a14f897008f2e1155573abee6c31a6547a7e8640fcd9c0106479b6\": rpc error: code = NotFound desc = could not find container \"0ceb932038a14f897008f2e1155573abee6c31a6547a7e8640fcd9c0106479b6\": container with ID starting with 0ceb932038a14f897008f2e1155573abee6c31a6547a7e8640fcd9c0106479b6 not found: ID does not exist"
Sep 10 15:36:38.176025 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.175997    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"]
Sep 10 15:36:38.179830 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.179808    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-26935da7/e2e-lifecycle-model-26935da7-kserve-d6b7d9555-cksxs"]
Sep 10 15:36:38.246282 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.246253    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9kz7\" (UniqueName: \"kubernetes.io/projected/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-kube-api-access-x9kz7\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:36:38.246282 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.246278    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-dshm\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:36:38.246466 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.246291    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-model-cache\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:36:38.246466 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.246300    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1708e2a4-3889-4b40-b5e5-cd949df6c0e2-tls-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:36:38.255112 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:38.255087    2573 scope.go:117] "RemoveContainer" containerID="af09abd02e5121b7fc4a2e14b94129f6e96dc3f880aae2d8bc3e1657b55181a1"
Sep 10 15:36:40.127285 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:36:40.127250    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1708e2a4-3889-4b40-b5e5-cd949df6c0e2" path="/var/lib/kubelet/pods/1708e2a4-3889-4b40-b5e5-cd949df6c0e2/volumes"
Sep 10 15:37:51.066579 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.066543    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7824-ef7013"]
Sep 10 15:37:51.067120 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.066940    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1708e2a4-3889-4b40-b5e5-cd949df6c0e2" containerName="main"
Sep 10 15:37:51.067120 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.066953    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1708e2a4-3889-4b40-b5e5-cd949df6c0e2" containerName="main"
Sep 10 15:37:51.067120 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.066965    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee58980-1f0a-4c03-b0be-b4b410b034fb" containerName="maas-api"
Sep 10 15:37:51.067120 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.066970    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee58980-1f0a-4c03-b0be-b4b410b034fb" containerName="maas-api"
Sep 10 15:37:51.067120 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.066980    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f960473-ffd5-4da1-b7a2-2fdbbe7597d8" containerName="payload-pre-processing"
Sep 10 15:37:51.067120 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.066985    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f960473-ffd5-4da1-b7a2-2fdbbe7597d8" containerName="payload-pre-processing"
Sep 10 15:37:51.067120 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.067042    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f960473-ffd5-4da1-b7a2-2fdbbe7597d8" containerName="payload-pre-processing"
Sep 10 15:37:51.067120 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.067050    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ee58980-1f0a-4c03-b0be-b4b410b034fb" containerName="maas-api"
Sep 10 15:37:51.067120 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.067056    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1708e2a4-3889-4b40-b5e5-cd949df6c0e2" containerName="main"
Sep 10 15:37:51.070445 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.070420    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7824-ef7013"
Sep 10 15:37:51.070868 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.070839    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7824-ef7013"]
Sep 10 15:37:51.073062 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.073036    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h7f4v\""
Sep 10 15:37:51.192207 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.192164    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm4qh\" (UniqueName: \"kubernetes.io/projected/5e65e3ca-1379-4481-8f30-99228b7960fb-kube-api-access-rm4qh\") pod \"test-curl-7824-ef7013\" (UID: \"5e65e3ca-1379-4481-8f30-99228b7960fb\") " pod="opendatahub/test-curl-7824-ef7013"
Sep 10 15:37:51.293265 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.293212    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm4qh\" (UniqueName: \"kubernetes.io/projected/5e65e3ca-1379-4481-8f30-99228b7960fb-kube-api-access-rm4qh\") pod \"test-curl-7824-ef7013\" (UID: \"5e65e3ca-1379-4481-8f30-99228b7960fb\") " pod="opendatahub/test-curl-7824-ef7013"
Sep 10 15:37:51.302116 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.302086    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm4qh\" (UniqueName: \"kubernetes.io/projected/5e65e3ca-1379-4481-8f30-99228b7960fb-kube-api-access-rm4qh\") pod \"test-curl-7824-ef7013\" (UID: \"5e65e3ca-1379-4481-8f30-99228b7960fb\") " pod="opendatahub/test-curl-7824-ef7013"
Sep 10 15:37:51.380637 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.380607    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7824-ef7013"
Sep 10 15:37:51.650310 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:51.650284    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7824-ef7013"]
Sep 10 15:37:51.652975 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:37:51.652947    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e65e3ca_1379_4481_8f30_99228b7960fb.slice/crio-81261ee3ffc6fc4ee91384e996a2de66464229fa980974fae1e19340644b92b5 WatchSource:0}: Error finding container 81261ee3ffc6fc4ee91384e996a2de66464229fa980974fae1e19340644b92b5: Status 404 returned error can't find the container with id 81261ee3ffc6fc4ee91384e996a2de66464229fa980974fae1e19340644b92b5
Sep 10 15:37:52.541991 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:52.541952    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7824-ef7013" event={"ID":"5e65e3ca-1379-4481-8f30-99228b7960fb","Type":"ContainerStarted","Data":"9aad401649454de3e4c57157a3c3d5a83faa17f57faf7804b08c7c99c8827441"}
Sep 10 15:37:52.542377 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:52.541998    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7824-ef7013" event={"ID":"5e65e3ca-1379-4481-8f30-99228b7960fb","Type":"ContainerStarted","Data":"81261ee3ffc6fc4ee91384e996a2de66464229fa980974fae1e19340644b92b5"}
Sep 10 15:37:52.556555 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:52.556490    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7824-ef7013" podStartSLOduration=1.556472607 podStartE2EDuration="1.556472607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:37:52.556284295 +0000 UTC m=+1334.972080130" watchObservedRunningTime="2026-09-10 15:37:52.556472607 +0000 UTC m=+1334.972268428"
Sep 10 15:37:53.148246 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:53.148208    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7824-ef7013"]
Sep 10 15:37:53.152647 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:53.152616    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7824-ef7013"]
Sep 10 15:37:53.547240 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:37:53.547208    2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7824-ef7013" container="test-curl-7824-ef7013"
Sep 10 15:37:54.552175 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:54.552131    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7824-ef7013" podUID="5e65e3ca-1379-4481-8f30-99228b7960fb" containerName="test-curl-7824-ef7013" containerID="cri-o://9aad401649454de3e4c57157a3c3d5a83faa17f57faf7804b08c7c99c8827441" gracePeriod=2
Sep 10 15:37:56.728034 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:56.728006    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7824-ef7013"
Sep 10 15:37:56.842874 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:56.842843    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm4qh\" (UniqueName: \"kubernetes.io/projected/5e65e3ca-1379-4481-8f30-99228b7960fb-kube-api-access-rm4qh\") pod \"5e65e3ca-1379-4481-8f30-99228b7960fb\" (UID: \"5e65e3ca-1379-4481-8f30-99228b7960fb\") "
Sep 10 15:37:56.845122 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:56.845095    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e65e3ca-1379-4481-8f30-99228b7960fb-kube-api-access-rm4qh" (OuterVolumeSpecName: "kube-api-access-rm4qh") pod "5e65e3ca-1379-4481-8f30-99228b7960fb" (UID: "5e65e3ca-1379-4481-8f30-99228b7960fb"). InnerVolumeSpecName "kube-api-access-rm4qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:37:56.944261 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:56.944223    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm4qh\" (UniqueName: \"kubernetes.io/projected/5e65e3ca-1379-4481-8f30-99228b7960fb-kube-api-access-rm4qh\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:37:57.568169 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:57.568130    2573 generic.go:356] "Generic (PLEG): container finished" podID="5e65e3ca-1379-4481-8f30-99228b7960fb" containerID="9aad401649454de3e4c57157a3c3d5a83faa17f57faf7804b08c7c99c8827441" exitCode=137
Sep 10 15:37:57.568335 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:57.568181    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7824-ef7013"
Sep 10 15:37:57.568335 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:57.568208    2573 scope.go:117] "RemoveContainer" containerID="9aad401649454de3e4c57157a3c3d5a83faa17f57faf7804b08c7c99c8827441"
Sep 10 15:37:57.578056 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:57.578039    2573 scope.go:117] "RemoveContainer" containerID="9aad401649454de3e4c57157a3c3d5a83faa17f57faf7804b08c7c99c8827441"
Sep 10 15:37:57.578361 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:37:57.578340    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aad401649454de3e4c57157a3c3d5a83faa17f57faf7804b08c7c99c8827441\": container with ID starting with 9aad401649454de3e4c57157a3c3d5a83faa17f57faf7804b08c7c99c8827441 not found: ID does not exist" containerID="9aad401649454de3e4c57157a3c3d5a83faa17f57faf7804b08c7c99c8827441"
Sep 10 15:37:57.578422 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:57.578371    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aad401649454de3e4c57157a3c3d5a83faa17f57faf7804b08c7c99c8827441"} err="failed to get container status \"9aad401649454de3e4c57157a3c3d5a83faa17f57faf7804b08c7c99c8827441\": rpc error: code = NotFound desc = could not find container \"9aad401649454de3e4c57157a3c3d5a83faa17f57faf7804b08c7c99c8827441\": container with ID starting with 9aad401649454de3e4c57157a3c3d5a83faa17f57faf7804b08c7c99c8827441 not found: ID does not exist"
Sep 10 15:37:57.579062 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:37:57.579036    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7824-ef7013\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="5e65e3ca-1379-4481-8f30-99228b7960fb" pod="opendatahub/test-curl-7824-ef7013"
Sep 10 15:37:58.123523 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:37:58.123482    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7824-ef7013\" is forbidden: User \"system:node:ip-10-0-134-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-115.ec2.internal' and this object" podUID="5e65e3ca-1379-4481-8f30-99228b7960fb" pod="opendatahub/test-curl-7824-ef7013"
Sep 10 15:37:58.125497 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:37:58.125473    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e65e3ca-1379-4481-8f30-99228b7960fb" path="/var/lib/kubelet/pods/5e65e3ca-1379-4481-8f30-99228b7960fb/volumes"
Sep 10 15:38:06.307904 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.307828    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"]
Sep 10 15:38:06.308257 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.308206    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e65e3ca-1379-4481-8f30-99228b7960fb" containerName="test-curl-7824-ef7013"
Sep 10 15:38:06.308257 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.308217    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e65e3ca-1379-4481-8f30-99228b7960fb" containerName="test-curl-7824-ef7013"
Sep 10 15:38:06.308374 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.308283    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e65e3ca-1379-4481-8f30-99228b7960fb" containerName="test-curl-7824-ef7013"
Sep 10 15:38:06.311175 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.311158    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:06.314071 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.314051    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dd499d-serving-cert\""
Sep 10 15:38:06.326260 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.326230    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"]
Sep 10 15:38:06.428848 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.428815    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-metrics-certs\") pod \"maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:06.429050 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.428871    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/395e577d-4b93-4c90-b751-7e3f3feb491f-openshift-service-ca\") pod \"maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:06.429050 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.428906    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-maas-api-tls\") pod \"maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:06.429050 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.428974    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxbgh\" (UniqueName: \"kubernetes.io/projected/395e577d-4b93-4c90-b751-7e3f3feb491f-kube-api-access-lxbgh\") pod \"maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:06.530371 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.530334    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/395e577d-4b93-4c90-b751-7e3f3feb491f-openshift-service-ca\") pod \"maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:06.530565 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.530384    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-maas-api-tls\") pod \"maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:06.530565 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.530413    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxbgh\" (UniqueName: \"kubernetes.io/projected/395e577d-4b93-4c90-b751-7e3f3feb491f-kube-api-access-lxbgh\") pod \"maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:06.530565 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.530457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-metrics-certs\") pod \"maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:06.530741 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:38:06.530570    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-serving-cert: secret "maas-api-e2e-api-a-dd499d-serving-cert" not found
Sep 10 15:38:06.530741 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:38:06.530676    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-maas-api-tls podName:395e577d-4b93-4c90-b751-7e3f3feb491f nodeName:}" failed. No retries permitted until 2026-09-10 15:38:07.030654344 +0000 UTC m=+1349.446450146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-maas-api-tls") pod "maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2" (UID: "395e577d-4b93-4c90-b751-7e3f3feb491f") : secret "maas-api-e2e-api-a-dd499d-serving-cert" not found
Sep 10 15:38:06.531056 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.531036    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/395e577d-4b93-4c90-b751-7e3f3feb491f-openshift-service-ca\") pod \"maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:06.532956 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.532937    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-metrics-certs\") pod \"maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:06.541302 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:06.541270    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxbgh\" (UniqueName: \"kubernetes.io/projected/395e577d-4b93-4c90-b751-7e3f3feb491f-kube-api-access-lxbgh\") pod \"maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:07.035693 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.035659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-maas-api-tls\") pod \"maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:07.038649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.038620    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-maas-api-tls\") pod \"maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:07.221257 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.221218    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:07.358011 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.357973    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"]
Sep 10 15:38:07.361531 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.361514    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"
Sep 10 15:38:07.366537 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.366514    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dd499d\""
Sep 10 15:38:07.366653 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.366540    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dd499d-dockercfg-2rnsz\""
Sep 10 15:38:07.383728 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.383697    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"]
Sep 10 15:38:07.438998 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.438955    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a6a12b-5cf4-48d1-b3b9-795436f56960-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd\" (UID: \"f2a6a12b-5cf4-48d1-b3b9-795436f56960\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"
Sep 10 15:38:07.439194 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.439021    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psk5v\" (UniqueName: \"kubernetes.io/projected/f2a6a12b-5cf4-48d1-b3b9-795436f56960-kube-api-access-psk5v\") pod \"payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd\" (UID: \"f2a6a12b-5cf4-48d1-b3b9-795436f56960\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"
Sep 10 15:38:07.513821 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.513772    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"]
Sep 10 15:38:07.515749 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:38:07.515718    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395e577d_4b93_4c90_b751_7e3f3feb491f.slice/crio-c2e5774c676a2b74c1e3084fb6047b0be71f69c8f45535426b90e7d23c39a79a WatchSource:0}: Error finding container c2e5774c676a2b74c1e3084fb6047b0be71f69c8f45535426b90e7d23c39a79a: Status 404 returned error can't find the container with id c2e5774c676a2b74c1e3084fb6047b0be71f69c8f45535426b90e7d23c39a79a
Sep 10 15:38:07.517360 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.517337    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:38:07.540255 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.540225    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a6a12b-5cf4-48d1-b3b9-795436f56960-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd\" (UID: \"f2a6a12b-5cf4-48d1-b3b9-795436f56960\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"
Sep 10 15:38:07.540409 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.540307    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psk5v\" (UniqueName: \"kubernetes.io/projected/f2a6a12b-5cf4-48d1-b3b9-795436f56960-kube-api-access-psk5v\") pod \"payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd\" (UID: \"f2a6a12b-5cf4-48d1-b3b9-795436f56960\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"
Sep 10 15:38:07.541024 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.541003    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a6a12b-5cf4-48d1-b3b9-795436f56960-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd\" (UID: \"f2a6a12b-5cf4-48d1-b3b9-795436f56960\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"
Sep 10 15:38:07.548827 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.548782    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psk5v\" (UniqueName: \"kubernetes.io/projected/f2a6a12b-5cf4-48d1-b3b9-795436f56960-kube-api-access-psk5v\") pod \"payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd\" (UID: \"f2a6a12b-5cf4-48d1-b3b9-795436f56960\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"
Sep 10 15:38:07.619130 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.619042    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2" event={"ID":"395e577d-4b93-4c90-b751-7e3f3feb491f","Type":"ContainerStarted","Data":"c2e5774c676a2b74c1e3084fb6047b0be71f69c8f45535426b90e7d23c39a79a"}
Sep 10 15:38:07.674167 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.674125    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"
Sep 10 15:38:07.975945 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:07.975916    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"]
Sep 10 15:38:07.977341 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:38:07.977312    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a6a12b_5cf4_48d1_b3b9_795436f56960.slice/crio-eb9cc3d1cd395e8b42a6b87c44f170406ecc47a4ec26363de2fa6bbf5b9e214b WatchSource:0}: Error finding container eb9cc3d1cd395e8b42a6b87c44f170406ecc47a4ec26363de2fa6bbf5b9e214b: Status 404 returned error can't find the container with id eb9cc3d1cd395e8b42a6b87c44f170406ecc47a4ec26363de2fa6bbf5b9e214b
Sep 10 15:38:08.626070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:08.626034    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd" event={"ID":"f2a6a12b-5cf4-48d1-b3b9-795436f56960","Type":"ContainerStarted","Data":"387c45d0a5bdbe2abecf23b6f66efe4cb9442f9c3397be898aff48b1d5c13e0c"}
Sep 10 15:38:08.626553 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:08.626098    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd" event={"ID":"f2a6a12b-5cf4-48d1-b3b9-795436f56960","Type":"ContainerStarted","Data":"eb9cc3d1cd395e8b42a6b87c44f170406ecc47a4ec26363de2fa6bbf5b9e214b"}
Sep 10 15:38:08.626553 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:08.626213    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"
Sep 10 15:38:08.627652 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:08.627627    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2" event={"ID":"395e577d-4b93-4c90-b751-7e3f3feb491f","Type":"ContainerStarted","Data":"af6fa92bee52195cc3faeb303f927959d2ab0a731a19958a8c7d84ed66216a90"}
Sep 10 15:38:08.627814 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:08.627682    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:08.731059 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:08.730984    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2" podStartSLOduration=2.246330627 podStartE2EDuration="2.730960471s" totalImagesPullingTime="484.629844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:38:08.730452413 +0000 UTC m=+1351.146248257" watchObservedRunningTime="2026-09-10 15:38:08.730960471 +0000 UTC m=+1351.146756290"
Sep 10 15:38:08.732044 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:08.732013    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd" podStartSLOduration=1.732001712 podStartE2EDuration="1.732001712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:38:08.673689639 +0000 UTC m=+1351.089485460" watchObservedRunningTime="2026-09-10 15:38:08.732001712 +0000 UTC m=+1351.147797532"
Sep 10 15:38:14.638454 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:14.638423    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:38:19.635282 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:19.635253    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"
Sep 10 15:38:41.142713 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.142652    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"]
Sep 10 15:38:41.147410 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.147388    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"
Sep 10 15:38:41.150673 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.150642    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bbc967\""
Sep 10 15:38:41.151142 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.151086    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bbc967-dockercfg-bmd29\""
Sep 10 15:38:41.156994 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.156970    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"]
Sep 10 15:38:41.243224 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.243185    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znwfn\" (UniqueName: \"kubernetes.io/projected/063a6467-2ac1-4583-8a9b-e28cfe1571f7-kube-api-access-znwfn\") pod \"payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt\" (UID: \"063a6467-2ac1-4583-8a9b-e28cfe1571f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"
Sep 10 15:38:41.243509 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.243477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063a6467-2ac1-4583-8a9b-e28cfe1571f7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt\" (UID: \"063a6467-2ac1-4583-8a9b-e28cfe1571f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"
Sep 10 15:38:41.344402 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.344361    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znwfn\" (UniqueName: \"kubernetes.io/projected/063a6467-2ac1-4583-8a9b-e28cfe1571f7-kube-api-access-znwfn\") pod \"payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt\" (UID: \"063a6467-2ac1-4583-8a9b-e28cfe1571f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"
Sep 10 15:38:41.344402 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.344401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063a6467-2ac1-4583-8a9b-e28cfe1571f7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt\" (UID: \"063a6467-2ac1-4583-8a9b-e28cfe1571f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"
Sep 10 15:38:41.345136 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.345118    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063a6467-2ac1-4583-8a9b-e28cfe1571f7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt\" (UID: \"063a6467-2ac1-4583-8a9b-e28cfe1571f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"
Sep 10 15:38:41.353899 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.353866    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znwfn\" (UniqueName: \"kubernetes.io/projected/063a6467-2ac1-4583-8a9b-e28cfe1571f7-kube-api-access-znwfn\") pod \"payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt\" (UID: \"063a6467-2ac1-4583-8a9b-e28cfe1571f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"
Sep 10 15:38:41.461386 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.461283    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"
Sep 10 15:38:41.743940 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.743915    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"]
Sep 10 15:38:41.745741 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:38:41.745714    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063a6467_2ac1_4583_8a9b_e28cfe1571f7.slice/crio-9a109280f718ab9e00131c4d3c0f72de6b23c2c9ec86292d4d7ca13cd5a099b2 WatchSource:0}: Error finding container 9a109280f718ab9e00131c4d3c0f72de6b23c2c9ec86292d4d7ca13cd5a099b2: Status 404 returned error can't find the container with id 9a109280f718ab9e00131c4d3c0f72de6b23c2c9ec86292d4d7ca13cd5a099b2
Sep 10 15:38:41.809449 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:41.809424    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt" event={"ID":"063a6467-2ac1-4583-8a9b-e28cfe1571f7","Type":"ContainerStarted","Data":"9a109280f718ab9e00131c4d3c0f72de6b23c2c9ec86292d4d7ca13cd5a099b2"}
Sep 10 15:38:42.817500 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:42.817456    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt" event={"ID":"063a6467-2ac1-4583-8a9b-e28cfe1571f7","Type":"ContainerStarted","Data":"5cac2d1c6fd66e67c9164521af6ac9a8b21809e2f6ff4bdeb9c92f29f3f8b7f7"}
Sep 10 15:38:42.817957 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:42.817610    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"
Sep 10 15:38:42.833950 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:42.833901    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt" podStartSLOduration=1.833885172 podStartE2EDuration="1.833885172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:38:42.833343461 +0000 UTC m=+1385.249139281" watchObservedRunningTime="2026-09-10 15:38:42.833885172 +0000 UTC m=+1385.249680991"
Sep 10 15:38:53.825388 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:38:53.825349    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"
Sep 10 15:39:22.746691 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:22.746655    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"]
Sep 10 15:39:22.747521 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:22.747459    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2" podUID="395e577d-4b93-4c90-b751-7e3f3feb491f" containerName="maas-api" containerID="cri-o://af6fa92bee52195cc3faeb303f927959d2ab0a731a19958a8c7d84ed66216a90" gracePeriod=30
Sep 10 15:39:22.813656 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:22.813614    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"]
Sep 10 15:39:22.813917 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:22.813871    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd" podUID="f2a6a12b-5cf4-48d1-b3b9-795436f56960" containerName="payload-processing" containerID="cri-o://387c45d0a5bdbe2abecf23b6f66efe4cb9442f9c3397be898aff48b1d5c13e0c" gracePeriod=30
Sep 10 15:39:23.034019 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.033983    2573 generic.go:356] "Generic (PLEG): container finished" podID="f2a6a12b-5cf4-48d1-b3b9-795436f56960" containerID="387c45d0a5bdbe2abecf23b6f66efe4cb9442f9c3397be898aff48b1d5c13e0c" exitCode=0
Sep 10 15:39:23.034188 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.034059    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd" event={"ID":"f2a6a12b-5cf4-48d1-b3b9-795436f56960","Type":"ContainerDied","Data":"387c45d0a5bdbe2abecf23b6f66efe4cb9442f9c3397be898aff48b1d5c13e0c"}
Sep 10 15:39:23.036224 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.036200    2573 generic.go:356] "Generic (PLEG): container finished" podID="395e577d-4b93-4c90-b751-7e3f3feb491f" containerID="af6fa92bee52195cc3faeb303f927959d2ab0a731a19958a8c7d84ed66216a90" exitCode=0
Sep 10 15:39:23.036328 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.036274    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2" event={"ID":"395e577d-4b93-4c90-b751-7e3f3feb491f","Type":"ContainerDied","Data":"af6fa92bee52195cc3faeb303f927959d2ab0a731a19958a8c7d84ed66216a90"}
Sep 10 15:39:23.078162 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.078136    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:39:23.090546 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.090521    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"
Sep 10 15:39:23.138501 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.138473    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psk5v\" (UniqueName: \"kubernetes.io/projected/f2a6a12b-5cf4-48d1-b3b9-795436f56960-kube-api-access-psk5v\") pod \"f2a6a12b-5cf4-48d1-b3b9-795436f56960\" (UID: \"f2a6a12b-5cf4-48d1-b3b9-795436f56960\") "
Sep 10 15:39:23.138660 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.138520    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-maas-api-tls\") pod \"395e577d-4b93-4c90-b751-7e3f3feb491f\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") "
Sep 10 15:39:23.138710 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.138691    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/395e577d-4b93-4c90-b751-7e3f3feb491f-openshift-service-ca\") pod \"395e577d-4b93-4c90-b751-7e3f3feb491f\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") "
Sep 10 15:39:23.138743 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.138725    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-metrics-certs\") pod \"395e577d-4b93-4c90-b751-7e3f3feb491f\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") "
Sep 10 15:39:23.138819 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.138757    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxbgh\" (UniqueName: \"kubernetes.io/projected/395e577d-4b93-4c90-b751-7e3f3feb491f-kube-api-access-lxbgh\") pod \"395e577d-4b93-4c90-b751-7e3f3feb491f\" (UID: \"395e577d-4b93-4c90-b751-7e3f3feb491f\") "
Sep 10 15:39:23.138819 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.138780    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a6a12b-5cf4-48d1-b3b9-795436f56960-plugins-config-volume\") pod \"f2a6a12b-5cf4-48d1-b3b9-795436f56960\" (UID: \"f2a6a12b-5cf4-48d1-b3b9-795436f56960\") "
Sep 10 15:39:23.139191 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.139160    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395e577d-4b93-4c90-b751-7e3f3feb491f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "395e577d-4b93-4c90-b751-7e3f3feb491f" (UID: "395e577d-4b93-4c90-b751-7e3f3feb491f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:39:23.139440 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.139410    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a6a12b-5cf4-48d1-b3b9-795436f56960-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2a6a12b-5cf4-48d1-b3b9-795436f56960" (UID: "f2a6a12b-5cf4-48d1-b3b9-795436f56960"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:39:23.141047 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.141015    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "395e577d-4b93-4c90-b751-7e3f3feb491f" (UID: "395e577d-4b93-4c90-b751-7e3f3feb491f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:39:23.141047 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.141028    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a6a12b-5cf4-48d1-b3b9-795436f56960-kube-api-access-psk5v" (OuterVolumeSpecName: "kube-api-access-psk5v") pod "f2a6a12b-5cf4-48d1-b3b9-795436f56960" (UID: "f2a6a12b-5cf4-48d1-b3b9-795436f56960"). InnerVolumeSpecName "kube-api-access-psk5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:39:23.141211 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.141064    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395e577d-4b93-4c90-b751-7e3f3feb491f-kube-api-access-lxbgh" (OuterVolumeSpecName: "kube-api-access-lxbgh") pod "395e577d-4b93-4c90-b751-7e3f3feb491f" (UID: "395e577d-4b93-4c90-b751-7e3f3feb491f"). InnerVolumeSpecName "kube-api-access-lxbgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:39:23.141211 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.141094    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "395e577d-4b93-4c90-b751-7e3f3feb491f" (UID: "395e577d-4b93-4c90-b751-7e3f3feb491f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:39:23.240519 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.240486    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psk5v\" (UniqueName: \"kubernetes.io/projected/f2a6a12b-5cf4-48d1-b3b9-795436f56960-kube-api-access-psk5v\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:39:23.240519 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.240512    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:39:23.240519 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.240522    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/395e577d-4b93-4c90-b751-7e3f3feb491f-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:39:23.240752 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.240531    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395e577d-4b93-4c90-b751-7e3f3feb491f-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:39:23.240752 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.240539    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxbgh\" (UniqueName: \"kubernetes.io/projected/395e577d-4b93-4c90-b751-7e3f3feb491f-kube-api-access-lxbgh\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:39:23.240752 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:23.240548    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a6a12b-5cf4-48d1-b3b9-795436f56960-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:39:24.049756 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:24.049672    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"
Sep 10 15:39:24.049756 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:24.049678    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd" event={"ID":"f2a6a12b-5cf4-48d1-b3b9-795436f56960","Type":"ContainerDied","Data":"eb9cc3d1cd395e8b42a6b87c44f170406ecc47a4ec26363de2fa6bbf5b9e214b"}
Sep 10 15:39:24.050281 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:24.049768    2573 scope.go:117] "RemoveContainer" containerID="387c45d0a5bdbe2abecf23b6f66efe4cb9442f9c3397be898aff48b1d5c13e0c"
Sep 10 15:39:24.051262 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:24.051246    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"
Sep 10 15:39:24.051349 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:24.051277    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2" event={"ID":"395e577d-4b93-4c90-b751-7e3f3feb491f","Type":"ContainerDied","Data":"c2e5774c676a2b74c1e3084fb6047b0be71f69c8f45535426b90e7d23c39a79a"}
Sep 10 15:39:24.060187 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:24.060172    2573 scope.go:117] "RemoveContainer" containerID="af6fa92bee52195cc3faeb303f927959d2ab0a731a19958a8c7d84ed66216a90"
Sep 10 15:39:24.084686 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:24.084655    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"]
Sep 10 15:39:24.088895 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:24.088868    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd499d-7b57797cd7-pczxd"]
Sep 10 15:39:24.097771 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:24.097750    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"]
Sep 10 15:39:24.106736 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:24.106715    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dd499d-f776dbf7d-q9kr2"]
Sep 10 15:39:24.127078 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:24.127043    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="395e577d-4b93-4c90-b751-7e3f3feb491f" path="/var/lib/kubelet/pods/395e577d-4b93-4c90-b751-7e3f3feb491f/volumes"
Sep 10 15:39:24.127413 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:24.127400    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2a6a12b-5cf4-48d1-b3b9-795436f56960" path="/var/lib/kubelet/pods/f2a6a12b-5cf4-48d1-b3b9-795436f56960/volumes"
Sep 10 15:39:45.842270 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:45.842237    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"]
Sep 10 15:39:45.842767 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:45.842541    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt" podUID="063a6467-2ac1-4583-8a9b-e28cfe1571f7" containerName="payload-processing" containerID="cri-o://5cac2d1c6fd66e67c9164521af6ac9a8b21809e2f6ff4bdeb9c92f29f3f8b7f7" gracePeriod=30
Sep 10 15:39:46.099465 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.099397    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"
Sep 10 15:39:46.170053 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.170013    2573 generic.go:356] "Generic (PLEG): container finished" podID="063a6467-2ac1-4583-8a9b-e28cfe1571f7" containerID="5cac2d1c6fd66e67c9164521af6ac9a8b21809e2f6ff4bdeb9c92f29f3f8b7f7" exitCode=0
Sep 10 15:39:46.170250 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.170074    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"
Sep 10 15:39:46.170250 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.170077    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt" event={"ID":"063a6467-2ac1-4583-8a9b-e28cfe1571f7","Type":"ContainerDied","Data":"5cac2d1c6fd66e67c9164521af6ac9a8b21809e2f6ff4bdeb9c92f29f3f8b7f7"}
Sep 10 15:39:46.170250 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.170200    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt" event={"ID":"063a6467-2ac1-4583-8a9b-e28cfe1571f7","Type":"ContainerDied","Data":"9a109280f718ab9e00131c4d3c0f72de6b23c2c9ec86292d4d7ca13cd5a099b2"}
Sep 10 15:39:46.170250 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.170229    2573 scope.go:117] "RemoveContainer" containerID="5cac2d1c6fd66e67c9164521af6ac9a8b21809e2f6ff4bdeb9c92f29f3f8b7f7"
Sep 10 15:39:46.180551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.180524    2573 scope.go:117] "RemoveContainer" containerID="5cac2d1c6fd66e67c9164521af6ac9a8b21809e2f6ff4bdeb9c92f29f3f8b7f7"
Sep 10 15:39:46.180930 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:39:46.180906    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cac2d1c6fd66e67c9164521af6ac9a8b21809e2f6ff4bdeb9c92f29f3f8b7f7\": container with ID starting with 5cac2d1c6fd66e67c9164521af6ac9a8b21809e2f6ff4bdeb9c92f29f3f8b7f7 not found: ID does not exist" containerID="5cac2d1c6fd66e67c9164521af6ac9a8b21809e2f6ff4bdeb9c92f29f3f8b7f7"
Sep 10 15:39:46.181010 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.180940    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cac2d1c6fd66e67c9164521af6ac9a8b21809e2f6ff4bdeb9c92f29f3f8b7f7"} err="failed to get container status \"5cac2d1c6fd66e67c9164521af6ac9a8b21809e2f6ff4bdeb9c92f29f3f8b7f7\": rpc error: code = NotFound desc = could not find container \"5cac2d1c6fd66e67c9164521af6ac9a8b21809e2f6ff4bdeb9c92f29f3f8b7f7\": container with ID starting with 5cac2d1c6fd66e67c9164521af6ac9a8b21809e2f6ff4bdeb9c92f29f3f8b7f7 not found: ID does not exist"
Sep 10 15:39:46.250380 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.250343    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znwfn\" (UniqueName: \"kubernetes.io/projected/063a6467-2ac1-4583-8a9b-e28cfe1571f7-kube-api-access-znwfn\") pod \"063a6467-2ac1-4583-8a9b-e28cfe1571f7\" (UID: \"063a6467-2ac1-4583-8a9b-e28cfe1571f7\") "
Sep 10 15:39:46.250553 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.250484    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063a6467-2ac1-4583-8a9b-e28cfe1571f7-plugins-config-volume\") pod \"063a6467-2ac1-4583-8a9b-e28cfe1571f7\" (UID: \"063a6467-2ac1-4583-8a9b-e28cfe1571f7\") "
Sep 10 15:39:46.250866 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.250842    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063a6467-2ac1-4583-8a9b-e28cfe1571f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "063a6467-2ac1-4583-8a9b-e28cfe1571f7" (UID: "063a6467-2ac1-4583-8a9b-e28cfe1571f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:39:46.252569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.252541    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063a6467-2ac1-4583-8a9b-e28cfe1571f7-kube-api-access-znwfn" (OuterVolumeSpecName: "kube-api-access-znwfn") pod "063a6467-2ac1-4583-8a9b-e28cfe1571f7" (UID: "063a6467-2ac1-4583-8a9b-e28cfe1571f7"). InnerVolumeSpecName "kube-api-access-znwfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:39:46.351556 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.351475    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063a6467-2ac1-4583-8a9b-e28cfe1571f7-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:39:46.351556 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.351503    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znwfn\" (UniqueName: \"kubernetes.io/projected/063a6467-2ac1-4583-8a9b-e28cfe1571f7-kube-api-access-znwfn\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:39:46.491514 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.491476    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"]
Sep 10 15:39:46.495401 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:46.495365    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bbc967-fb99bb99-nx2pt"]
Sep 10 15:39:48.125610 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:48.125573    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="063a6467-2ac1-4583-8a9b-e28cfe1571f7" path="/var/lib/kubelet/pods/063a6467-2ac1-4583-8a9b-e28cfe1571f7/volumes"
Sep 10 15:39:52.739126 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.739094    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"]
Sep 10 15:39:52.739594 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.739497    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a6a12b-5cf4-48d1-b3b9-795436f56960" containerName="payload-processing"
Sep 10 15:39:52.739594 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.739511    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a6a12b-5cf4-48d1-b3b9-795436f56960" containerName="payload-processing"
Sep 10 15:39:52.739594 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.739525    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063a6467-2ac1-4583-8a9b-e28cfe1571f7" containerName="payload-processing"
Sep 10 15:39:52.739594 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.739531    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="063a6467-2ac1-4583-8a9b-e28cfe1571f7" containerName="payload-processing"
Sep 10 15:39:52.739594 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.739537    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="395e577d-4b93-4c90-b751-7e3f3feb491f" containerName="maas-api"
Sep 10 15:39:52.739594 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.739543    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="395e577d-4b93-4c90-b751-7e3f3feb491f" containerName="maas-api"
Sep 10 15:39:52.740062 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.739619    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="395e577d-4b93-4c90-b751-7e3f3feb491f" containerName="maas-api"
Sep 10 15:39:52.740062 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.739628    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="063a6467-2ac1-4583-8a9b-e28cfe1571f7" containerName="payload-processing"
Sep 10 15:39:52.740062 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.739634    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a6a12b-5cf4-48d1-b3b9-795436f56960" containerName="payload-processing"
Sep 10 15:39:52.744133 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.744117    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:52.747362 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.747338    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-62d2f7\"/\"openshift-service-ca.crt\""
Sep 10 15:39:52.747525 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.747397    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-62d2f7\"/\"default-dockercfg-mfpk6\""
Sep 10 15:39:52.747525 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.747422    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-62d2f7\"/\"kube-root-ca.crt\""
Sep 10 15:39:52.747525 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.747418    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-62d2f7\"/\"rate-test-model-62d2f7-kserve-self-signed-certs\""
Sep 10 15:39:52.753574 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.753553    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"]
Sep 10 15:39:52.912953 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.912911    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-home\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:52.913156 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.912957    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-dshm\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:52.913156 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.913054    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-model-cache\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:52.913156 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.913091    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thqnq\" (UniqueName: \"kubernetes.io/projected/06101ebd-fc0e-44db-acbc-a7e03898db7f-kube-api-access-thqnq\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:52.913286 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.913154    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06101ebd-fc0e-44db-acbc-a7e03898db7f-tls-certs\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:52.913286 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:52.913218    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-tmp-dir\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.014255 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.014159    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thqnq\" (UniqueName: \"kubernetes.io/projected/06101ebd-fc0e-44db-acbc-a7e03898db7f-kube-api-access-thqnq\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.014255 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.014205    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06101ebd-fc0e-44db-acbc-a7e03898db7f-tls-certs\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.014255 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.014247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-tmp-dir\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.014524 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.014287    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-home\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.014524 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.014303    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-dshm\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.014524 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.014334    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-model-cache\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.014662 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.014639    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-tmp-dir\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.014821 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.014766    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-model-cache\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.014909 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.014844    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-home\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.016744 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.016721    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-dshm\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.017040 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.017022    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06101ebd-fc0e-44db-acbc-a7e03898db7f-tls-certs\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.022528 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.022505    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thqnq\" (UniqueName: \"kubernetes.io/projected/06101ebd-fc0e-44db-acbc-a7e03898db7f-kube-api-access-thqnq\") pod \"rate-test-model-62d2f7-kserve-b55bc9579-zpjxg\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") " pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.054073 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.054026    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:39:53.338368 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:53.338340    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"]
Sep 10 15:39:53.340733 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:39:53.340701    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06101ebd_fc0e_44db_acbc_a7e03898db7f.slice/crio-fe548134812f51c77fc6868b7b9a5e21cea3999c362cef6ab7b60e64b9c84a86 WatchSource:0}: Error finding container fe548134812f51c77fc6868b7b9a5e21cea3999c362cef6ab7b60e64b9c84a86: Status 404 returned error can't find the container with id fe548134812f51c77fc6868b7b9a5e21cea3999c362cef6ab7b60e64b9c84a86
Sep 10 15:39:54.220740 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:54.220696    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg" event={"ID":"06101ebd-fc0e-44db-acbc-a7e03898db7f","Type":"ContainerStarted","Data":"e2b5fd369c2ad2c4fe36c8fb6a3082aab7b04d00d719d457a827c7808df8e47d"}
Sep 10 15:39:54.220740 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:54.220755    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg" event={"ID":"06101ebd-fc0e-44db-acbc-a7e03898db7f","Type":"ContainerStarted","Data":"fe548134812f51c77fc6868b7b9a5e21cea3999c362cef6ab7b60e64b9c84a86"}
Sep 10 15:39:54.237278 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:39:54.237224    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg" podStartSLOduration=2.237208342 podStartE2EDuration="2.237208342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:39:54.235233724 +0000 UTC m=+1456.651029543" watchObservedRunningTime="2026-09-10 15:39:54.237208342 +0000 UTC m=+1456.653004189"
Sep 10 15:40:03.054479 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:03.054443    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:40:03.054988 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:03.054519    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:40:03.067590 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:03.067566    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:40:03.279078 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:03.279045    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:40:03.729760 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:03.729721    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"]
Sep 10 15:40:03.729958 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:03.729937    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl" podUID="e038425b-805f-4030-9cf7-662c4189cf63" containerName="maas-api" containerID="cri-o://97f64844518cb8ed99b67399f7cb36dffe79868bfea3dd7c69427a101e7a6631" gracePeriod=30
Sep 10 15:40:03.789924 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:03.789886    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"]
Sep 10 15:40:03.790177 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:03.790148    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls" podUID="acb591d7-4afd-424d-9773-4d5812165d7a" containerName="payload-pre-processing" containerID="cri-o://81f2ca389eb61b474f0a36e667617b688460ac8d3d2c360a323aa45edabeb4d8" gracePeriod=30
Sep 10 15:40:04.071004 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.070976    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:40:04.074948 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.074917    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"
Sep 10 15:40:04.122139 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.122111    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82tv6\" (UniqueName: \"kubernetes.io/projected/e038425b-805f-4030-9cf7-662c4189cf63-kube-api-access-82tv6\") pod \"e038425b-805f-4030-9cf7-662c4189cf63\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") "
Sep 10 15:40:04.122139 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.122143    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-maas-api-tls\") pod \"e038425b-805f-4030-9cf7-662c4189cf63\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") "
Sep 10 15:40:04.122390 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.122214    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb591d7-4afd-424d-9773-4d5812165d7a-plugins-config-volume\") pod \"acb591d7-4afd-424d-9773-4d5812165d7a\" (UID: \"acb591d7-4afd-424d-9773-4d5812165d7a\") "
Sep 10 15:40:04.122390 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.122249    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e038425b-805f-4030-9cf7-662c4189cf63-openshift-service-ca\") pod \"e038425b-805f-4030-9cf7-662c4189cf63\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") "
Sep 10 15:40:04.122390 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.122309    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pm8d\" (UniqueName: \"kubernetes.io/projected/acb591d7-4afd-424d-9773-4d5812165d7a-kube-api-access-2pm8d\") pod \"acb591d7-4afd-424d-9773-4d5812165d7a\" (UID: \"acb591d7-4afd-424d-9773-4d5812165d7a\") "
Sep 10 15:40:04.122390 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.122373    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-metrics-certs\") pod \"e038425b-805f-4030-9cf7-662c4189cf63\" (UID: \"e038425b-805f-4030-9cf7-662c4189cf63\") "
Sep 10 15:40:04.122670 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.122644    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb591d7-4afd-424d-9773-4d5812165d7a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "acb591d7-4afd-424d-9773-4d5812165d7a" (UID: "acb591d7-4afd-424d-9773-4d5812165d7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:40:04.122938 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.122907    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e038425b-805f-4030-9cf7-662c4189cf63-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e038425b-805f-4030-9cf7-662c4189cf63" (UID: "e038425b-805f-4030-9cf7-662c4189cf63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:40:04.124670 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.124644    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e038425b-805f-4030-9cf7-662c4189cf63" (UID: "e038425b-805f-4030-9cf7-662c4189cf63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:40:04.125251 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.125223    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb591d7-4afd-424d-9773-4d5812165d7a-kube-api-access-2pm8d" (OuterVolumeSpecName: "kube-api-access-2pm8d") pod "acb591d7-4afd-424d-9773-4d5812165d7a" (UID: "acb591d7-4afd-424d-9773-4d5812165d7a"). InnerVolumeSpecName "kube-api-access-2pm8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:40:04.125334 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.125312    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e038425b-805f-4030-9cf7-662c4189cf63" (UID: "e038425b-805f-4030-9cf7-662c4189cf63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:40:04.125404 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.125385    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e038425b-805f-4030-9cf7-662c4189cf63-kube-api-access-82tv6" (OuterVolumeSpecName: "kube-api-access-82tv6") pod "e038425b-805f-4030-9cf7-662c4189cf63" (UID: "e038425b-805f-4030-9cf7-662c4189cf63"). InnerVolumeSpecName "kube-api-access-82tv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:40:04.223804 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.223766    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82tv6\" (UniqueName: \"kubernetes.io/projected/e038425b-805f-4030-9cf7-662c4189cf63-kube-api-access-82tv6\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:40:04.223982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.223826    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:40:04.223982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.223841    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb591d7-4afd-424d-9773-4d5812165d7a-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:40:04.223982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.223856    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e038425b-805f-4030-9cf7-662c4189cf63-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:40:04.223982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.223865    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pm8d\" (UniqueName: \"kubernetes.io/projected/acb591d7-4afd-424d-9773-4d5812165d7a-kube-api-access-2pm8d\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:40:04.223982 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.223873    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e038425b-805f-4030-9cf7-662c4189cf63-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:40:04.276513 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.276420    2573 generic.go:356] "Generic (PLEG): container finished" podID="e038425b-805f-4030-9cf7-662c4189cf63" containerID="97f64844518cb8ed99b67399f7cb36dffe79868bfea3dd7c69427a101e7a6631" exitCode=0
Sep 10 15:40:04.276513 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.276462    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"
Sep 10 15:40:04.276747 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.276508    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl" event={"ID":"e038425b-805f-4030-9cf7-662c4189cf63","Type":"ContainerDied","Data":"97f64844518cb8ed99b67399f7cb36dffe79868bfea3dd7c69427a101e7a6631"}
Sep 10 15:40:04.276747 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.276572    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl" event={"ID":"e038425b-805f-4030-9cf7-662c4189cf63","Type":"ContainerDied","Data":"c409e913767a789f4f52bb43140669bba5c636a148fe1bc94667358343e10f68"}
Sep 10 15:40:04.276747 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.276603    2573 scope.go:117] "RemoveContainer" containerID="97f64844518cb8ed99b67399f7cb36dffe79868bfea3dd7c69427a101e7a6631"
Sep 10 15:40:04.278100 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.278073    2573 generic.go:356] "Generic (PLEG): container finished" podID="acb591d7-4afd-424d-9773-4d5812165d7a" containerID="81f2ca389eb61b474f0a36e667617b688460ac8d3d2c360a323aa45edabeb4d8" exitCode=0
Sep 10 15:40:04.278228 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.278125    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"
Sep 10 15:40:04.278228 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.278164    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls" event={"ID":"acb591d7-4afd-424d-9773-4d5812165d7a","Type":"ContainerDied","Data":"81f2ca389eb61b474f0a36e667617b688460ac8d3d2c360a323aa45edabeb4d8"}
Sep 10 15:40:04.278345 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.278224    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls" event={"ID":"acb591d7-4afd-424d-9773-4d5812165d7a","Type":"ContainerDied","Data":"7ff6212cbb041449ed67b96288469b3e17acd931437a6904d6694778e842d2ef"}
Sep 10 15:40:04.286233 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.286203    2573 scope.go:117] "RemoveContainer" containerID="97f64844518cb8ed99b67399f7cb36dffe79868bfea3dd7c69427a101e7a6631"
Sep 10 15:40:04.286518 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:40:04.286497    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f64844518cb8ed99b67399f7cb36dffe79868bfea3dd7c69427a101e7a6631\": container with ID starting with 97f64844518cb8ed99b67399f7cb36dffe79868bfea3dd7c69427a101e7a6631 not found: ID does not exist" containerID="97f64844518cb8ed99b67399f7cb36dffe79868bfea3dd7c69427a101e7a6631"
Sep 10 15:40:04.286582 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.286527    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f64844518cb8ed99b67399f7cb36dffe79868bfea3dd7c69427a101e7a6631"} err="failed to get container status \"97f64844518cb8ed99b67399f7cb36dffe79868bfea3dd7c69427a101e7a6631\": rpc error: code = NotFound desc = could not find container \"97f64844518cb8ed99b67399f7cb36dffe79868bfea3dd7c69427a101e7a6631\": container with ID starting with 97f64844518cb8ed99b67399f7cb36dffe79868bfea3dd7c69427a101e7a6631 not found: ID does not exist"
Sep 10 15:40:04.286582 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.286546    2573 scope.go:117] "RemoveContainer" containerID="81f2ca389eb61b474f0a36e667617b688460ac8d3d2c360a323aa45edabeb4d8"
Sep 10 15:40:04.294549 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.294522    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"]
Sep 10 15:40:04.295237 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.295218    2573 scope.go:117] "RemoveContainer" containerID="81f2ca389eb61b474f0a36e667617b688460ac8d3d2c360a323aa45edabeb4d8"
Sep 10 15:40:04.295543 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:40:04.295517    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f2ca389eb61b474f0a36e667617b688460ac8d3d2c360a323aa45edabeb4d8\": container with ID starting with 81f2ca389eb61b474f0a36e667617b688460ac8d3d2c360a323aa45edabeb4d8 not found: ID does not exist" containerID="81f2ca389eb61b474f0a36e667617b688460ac8d3d2c360a323aa45edabeb4d8"
Sep 10 15:40:04.295591 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.295550    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f2ca389eb61b474f0a36e667617b688460ac8d3d2c360a323aa45edabeb4d8"} err="failed to get container status \"81f2ca389eb61b474f0a36e667617b688460ac8d3d2c360a323aa45edabeb4d8\": rpc error: code = NotFound desc = could not find container \"81f2ca389eb61b474f0a36e667617b688460ac8d3d2c360a323aa45edabeb4d8\": container with ID starting with 81f2ca389eb61b474f0a36e667617b688460ac8d3d2c360a323aa45edabeb4d8 not found: ID does not exist"
Sep 10 15:40:04.299750 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.299729    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fe45b-d4bfb654-zkqvl"]
Sep 10 15:40:04.309355 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.309329    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"]
Sep 10 15:40:04.312916 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:04.312891    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fe45b-68c9d64b7d-qdzls"]
Sep 10 15:40:06.126729 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:06.126697    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acb591d7-4afd-424d-9773-4d5812165d7a" path="/var/lib/kubelet/pods/acb591d7-4afd-424d-9773-4d5812165d7a/volumes"
Sep 10 15:40:06.127125 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:06.127111    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e038425b-805f-4030-9cf7-662c4189cf63" path="/var/lib/kubelet/pods/e038425b-805f-4030-9cf7-662c4189cf63/volumes"
Sep 10 15:40:21.781087 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:21.781052    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"]
Sep 10 15:40:21.781572 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:21.781312    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d" podUID="9336a78f-f603-415f-8a79-ca64fd6eb9ac" containerName="maas-api" containerID="cri-o://367b32b209166361219056e71a634e09471cd16e9e45ffbb14053a3a514694a1" gracePeriod=30
Sep 10 15:40:22.034785 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.034720    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:40:22.184508 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.184471    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw7c9\" (UniqueName: \"kubernetes.io/projected/9336a78f-f603-415f-8a79-ca64fd6eb9ac-kube-api-access-mw7c9\") pod \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") "
Sep 10 15:40:22.184701 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.184536    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-maas-api-tls\") pod \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") "
Sep 10 15:40:22.184701 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.184566    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-metrics-certs\") pod \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") "
Sep 10 15:40:22.184862 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.184728    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9336a78f-f603-415f-8a79-ca64fd6eb9ac-openshift-service-ca\") pod \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\" (UID: \"9336a78f-f603-415f-8a79-ca64fd6eb9ac\") "
Sep 10 15:40:22.185075 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.185045    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9336a78f-f603-415f-8a79-ca64fd6eb9ac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9336a78f-f603-415f-8a79-ca64fd6eb9ac" (UID: "9336a78f-f603-415f-8a79-ca64fd6eb9ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:40:22.186868 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.186847    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9336a78f-f603-415f-8a79-ca64fd6eb9ac-kube-api-access-mw7c9" (OuterVolumeSpecName: "kube-api-access-mw7c9") pod "9336a78f-f603-415f-8a79-ca64fd6eb9ac" (UID: "9336a78f-f603-415f-8a79-ca64fd6eb9ac"). InnerVolumeSpecName "kube-api-access-mw7c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:40:22.187238 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.187220    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9336a78f-f603-415f-8a79-ca64fd6eb9ac" (UID: "9336a78f-f603-415f-8a79-ca64fd6eb9ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:40:22.187284 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.187248    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9336a78f-f603-415f-8a79-ca64fd6eb9ac" (UID: "9336a78f-f603-415f-8a79-ca64fd6eb9ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:40:22.286155 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.286076    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:40:22.286155 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.286104    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9336a78f-f603-415f-8a79-ca64fd6eb9ac-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:40:22.286155 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.286114    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw7c9\" (UniqueName: \"kubernetes.io/projected/9336a78f-f603-415f-8a79-ca64fd6eb9ac-kube-api-access-mw7c9\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:40:22.286155 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.286122    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9336a78f-f603-415f-8a79-ca64fd6eb9ac-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:40:22.378363 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.378326    2573 generic.go:356] "Generic (PLEG): container finished" podID="9336a78f-f603-415f-8a79-ca64fd6eb9ac" containerID="367b32b209166361219056e71a634e09471cd16e9e45ffbb14053a3a514694a1" exitCode=0
Sep 10 15:40:22.378531 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.378384    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d" event={"ID":"9336a78f-f603-415f-8a79-ca64fd6eb9ac","Type":"ContainerDied","Data":"367b32b209166361219056e71a634e09471cd16e9e45ffbb14053a3a514694a1"}
Sep 10 15:40:22.378531 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.378422    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d" event={"ID":"9336a78f-f603-415f-8a79-ca64fd6eb9ac","Type":"ContainerDied","Data":"fe6b02fb06fe97aec622b7949ed613a8c698bab5d9e073ead76d7591f1305d07"}
Sep 10 15:40:22.378531 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.378441    2573 scope.go:117] "RemoveContainer" containerID="367b32b209166361219056e71a634e09471cd16e9e45ffbb14053a3a514694a1"
Sep 10 15:40:22.378531 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.378385    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"
Sep 10 15:40:22.388337 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.388316    2573 scope.go:117] "RemoveContainer" containerID="367b32b209166361219056e71a634e09471cd16e9e45ffbb14053a3a514694a1"
Sep 10 15:40:22.388612 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:40:22.388593    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"367b32b209166361219056e71a634e09471cd16e9e45ffbb14053a3a514694a1\": container with ID starting with 367b32b209166361219056e71a634e09471cd16e9e45ffbb14053a3a514694a1 not found: ID does not exist" containerID="367b32b209166361219056e71a634e09471cd16e9e45ffbb14053a3a514694a1"
Sep 10 15:40:22.388691 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.388620    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367b32b209166361219056e71a634e09471cd16e9e45ffbb14053a3a514694a1"} err="failed to get container status \"367b32b209166361219056e71a634e09471cd16e9e45ffbb14053a3a514694a1\": rpc error: code = NotFound desc = could not find container \"367b32b209166361219056e71a634e09471cd16e9e45ffbb14053a3a514694a1\": container with ID starting with 367b32b209166361219056e71a634e09471cd16e9e45ffbb14053a3a514694a1 not found: ID does not exist"
Sep 10 15:40:22.399256 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.399231    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"]
Sep 10 15:40:22.403330 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:22.403309    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d7a2b3-7ccb9bbb89-z2t8d"]
Sep 10 15:40:24.125984 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:24.125949    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9336a78f-f603-415f-8a79-ca64fd6eb9ac" path="/var/lib/kubelet/pods/9336a78f-f603-415f-8a79-ca64fd6eb9ac/volumes"
Sep 10 15:40:59.696933 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:40:59.696852    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"]
Sep 10 15:41:06.084692 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.084033    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"]
Sep 10 15:41:06.084692 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.084601    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acb591d7-4afd-424d-9773-4d5812165d7a" containerName="payload-pre-processing"
Sep 10 15:41:06.084692 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.084619    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="acb591d7-4afd-424d-9773-4d5812165d7a" containerName="payload-pre-processing"
Sep 10 15:41:06.084692 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.084639    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e038425b-805f-4030-9cf7-662c4189cf63" containerName="maas-api"
Sep 10 15:41:06.084692 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.084647    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e038425b-805f-4030-9cf7-662c4189cf63" containerName="maas-api"
Sep 10 15:41:06.084692 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.084656    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9336a78f-f603-415f-8a79-ca64fd6eb9ac" containerName="maas-api"
Sep 10 15:41:06.084692 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.084666    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9336a78f-f603-415f-8a79-ca64fd6eb9ac" containerName="maas-api"
Sep 10 15:41:06.085758 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.084748    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="acb591d7-4afd-424d-9773-4d5812165d7a" containerName="payload-pre-processing"
Sep 10 15:41:06.085758 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.084757    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9336a78f-f603-415f-8a79-ca64fd6eb9ac" containerName="maas-api"
Sep 10 15:41:06.085758 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.084768    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e038425b-805f-4030-9cf7-662c4189cf63" containerName="maas-api"
Sep 10 15:41:06.089486 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.089449    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"]
Sep 10 15:41:06.089613 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.089572    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"
Sep 10 15:41:06.091872 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.091841    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ba5c7e\""
Sep 10 15:41:06.092035 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.092017    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ba5c7e-dockercfg-fwm9l\""
Sep 10 15:41:06.180882 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.180833    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d79d08-e4f4-4cfd-b605-b2349ef1615e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8\" (UID: \"14d79d08-e4f4-4cfd-b605-b2349ef1615e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"
Sep 10 15:41:06.180882 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.180886    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq6c8\" (UniqueName: \"kubernetes.io/projected/14d79d08-e4f4-4cfd-b605-b2349ef1615e-kube-api-access-xq6c8\") pod \"payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8\" (UID: \"14d79d08-e4f4-4cfd-b605-b2349ef1615e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"
Sep 10 15:41:06.281782 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.281752    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d79d08-e4f4-4cfd-b605-b2349ef1615e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8\" (UID: \"14d79d08-e4f4-4cfd-b605-b2349ef1615e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"
Sep 10 15:41:06.281975 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.281814    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq6c8\" (UniqueName: \"kubernetes.io/projected/14d79d08-e4f4-4cfd-b605-b2349ef1615e-kube-api-access-xq6c8\") pod \"payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8\" (UID: \"14d79d08-e4f4-4cfd-b605-b2349ef1615e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"
Sep 10 15:41:06.282529 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.282507    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d79d08-e4f4-4cfd-b605-b2349ef1615e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8\" (UID: \"14d79d08-e4f4-4cfd-b605-b2349ef1615e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"
Sep 10 15:41:06.289768 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.289740    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq6c8\" (UniqueName: \"kubernetes.io/projected/14d79d08-e4f4-4cfd-b605-b2349ef1615e-kube-api-access-xq6c8\") pod \"payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8\" (UID: \"14d79d08-e4f4-4cfd-b605-b2349ef1615e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"
Sep 10 15:41:06.400267 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.400177    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"
Sep 10 15:41:06.679459 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:06.679433    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"]
Sep 10 15:41:06.682306 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:41:06.682274    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d79d08_e4f4_4cfd_b605_b2349ef1615e.slice/crio-f72d6850cc8d66095ae7f31cb91141c1c131739bacb530286634ce1d14ced09f WatchSource:0}: Error finding container f72d6850cc8d66095ae7f31cb91141c1c131739bacb530286634ce1d14ced09f: Status 404 returned error can't find the container with id f72d6850cc8d66095ae7f31cb91141c1c131739bacb530286634ce1d14ced09f
Sep 10 15:41:07.615181 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:07.615149    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8" event={"ID":"14d79d08-e4f4-4cfd-b605-b2349ef1615e","Type":"ContainerStarted","Data":"ac1988b731213ada238ad2518254b7fa291e227f572310d7b1a5d04905277877"}
Sep 10 15:41:07.615638 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:07.615198    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8" event={"ID":"14d79d08-e4f4-4cfd-b605-b2349ef1615e","Type":"ContainerStarted","Data":"f72d6850cc8d66095ae7f31cb91141c1c131739bacb530286634ce1d14ced09f"}
Sep 10 15:41:07.615638 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:07.615296    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"
Sep 10 15:41:07.632318 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:07.632270    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8" podStartSLOduration=1.632253708 podStartE2EDuration="1.632253708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:07.630306789 +0000 UTC m=+1530.046102609" watchObservedRunningTime="2026-09-10 15:41:07.632253708 +0000 UTC m=+1530.048049527"
Sep 10 15:41:14.717348 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:14.717283    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg" podUID="06101ebd-fc0e-44db-acbc-a7e03898db7f" containerName="main" containerID="cri-o://e2b5fd369c2ad2c4fe36c8fb6a3082aab7b04d00d719d457a827c7808df8e47d" gracePeriod=45
Sep 10 15:41:14.980132 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:14.980056    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:41:15.061547 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.061503    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06101ebd-fc0e-44db-acbc-a7e03898db7f-tls-certs\") pod \"06101ebd-fc0e-44db-acbc-a7e03898db7f\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") "
Sep 10 15:41:15.061736 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.061593    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-dshm\") pod \"06101ebd-fc0e-44db-acbc-a7e03898db7f\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") "
Sep 10 15:41:15.061736 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.061635    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-tmp-dir\") pod \"06101ebd-fc0e-44db-acbc-a7e03898db7f\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") "
Sep 10 15:41:15.061736 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.061667    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-model-cache\") pod \"06101ebd-fc0e-44db-acbc-a7e03898db7f\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") "
Sep 10 15:41:15.061736 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.061688    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-home\") pod \"06101ebd-fc0e-44db-acbc-a7e03898db7f\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") "
Sep 10 15:41:15.061736 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.061716    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thqnq\" (UniqueName: \"kubernetes.io/projected/06101ebd-fc0e-44db-acbc-a7e03898db7f-kube-api-access-thqnq\") pod \"06101ebd-fc0e-44db-acbc-a7e03898db7f\" (UID: \"06101ebd-fc0e-44db-acbc-a7e03898db7f\") "
Sep 10 15:41:15.062043 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.061959    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06101ebd-fc0e-44db-acbc-a7e03898db7f" (UID: "06101ebd-fc0e-44db-acbc-a7e03898db7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:41:15.062043 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.061992    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-home" (OuterVolumeSpecName: "home") pod "06101ebd-fc0e-44db-acbc-a7e03898db7f" (UID: "06101ebd-fc0e-44db-acbc-a7e03898db7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:41:15.062043 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.061960    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "06101ebd-fc0e-44db-acbc-a7e03898db7f" (UID: "06101ebd-fc0e-44db-acbc-a7e03898db7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:41:15.064116 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.064088    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-dshm" (OuterVolumeSpecName: "dshm") pod "06101ebd-fc0e-44db-acbc-a7e03898db7f" (UID: "06101ebd-fc0e-44db-acbc-a7e03898db7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:41:15.064216 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.064089    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06101ebd-fc0e-44db-acbc-a7e03898db7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06101ebd-fc0e-44db-acbc-a7e03898db7f" (UID: "06101ebd-fc0e-44db-acbc-a7e03898db7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:41:15.064216 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.064080    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06101ebd-fc0e-44db-acbc-a7e03898db7f-kube-api-access-thqnq" (OuterVolumeSpecName: "kube-api-access-thqnq") pod "06101ebd-fc0e-44db-acbc-a7e03898db7f" (UID: "06101ebd-fc0e-44db-acbc-a7e03898db7f"). InnerVolumeSpecName "kube-api-access-thqnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:41:15.162634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.162597    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-tmp-dir\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:41:15.162634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.162627    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-model-cache\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:41:15.162634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.162635    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-home\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:41:15.162913 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.162644    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thqnq\" (UniqueName: \"kubernetes.io/projected/06101ebd-fc0e-44db-acbc-a7e03898db7f-kube-api-access-thqnq\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:41:15.162913 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.162656    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06101ebd-fc0e-44db-acbc-a7e03898db7f-tls-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:41:15.162913 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.162663    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06101ebd-fc0e-44db-acbc-a7e03898db7f-dshm\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:41:15.655614 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.655575    2573 generic.go:356] "Generic (PLEG): container finished" podID="06101ebd-fc0e-44db-acbc-a7e03898db7f" containerID="e2b5fd369c2ad2c4fe36c8fb6a3082aab7b04d00d719d457a827c7808df8e47d" exitCode=0
Sep 10 15:41:15.655614 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.655610    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"
Sep 10 15:41:15.655876 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.655636    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg" event={"ID":"06101ebd-fc0e-44db-acbc-a7e03898db7f","Type":"ContainerDied","Data":"e2b5fd369c2ad2c4fe36c8fb6a3082aab7b04d00d719d457a827c7808df8e47d"}
Sep 10 15:41:15.655876 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.655675    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg" event={"ID":"06101ebd-fc0e-44db-acbc-a7e03898db7f","Type":"ContainerDied","Data":"fe548134812f51c77fc6868b7b9a5e21cea3999c362cef6ab7b60e64b9c84a86"}
Sep 10 15:41:15.655876 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.655695    2573 scope.go:117] "RemoveContainer" containerID="e2b5fd369c2ad2c4fe36c8fb6a3082aab7b04d00d719d457a827c7808df8e47d"
Sep 10 15:41:15.665352 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.665330    2573 scope.go:117] "RemoveContainer" containerID="e2b5fd369c2ad2c4fe36c8fb6a3082aab7b04d00d719d457a827c7808df8e47d"
Sep 10 15:41:15.665670 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:41:15.665646    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b5fd369c2ad2c4fe36c8fb6a3082aab7b04d00d719d457a827c7808df8e47d\": container with ID starting with e2b5fd369c2ad2c4fe36c8fb6a3082aab7b04d00d719d457a827c7808df8e47d not found: ID does not exist" containerID="e2b5fd369c2ad2c4fe36c8fb6a3082aab7b04d00d719d457a827c7808df8e47d"
Sep 10 15:41:15.665767 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.665676    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b5fd369c2ad2c4fe36c8fb6a3082aab7b04d00d719d457a827c7808df8e47d"} err="failed to get container status \"e2b5fd369c2ad2c4fe36c8fb6a3082aab7b04d00d719d457a827c7808df8e47d\": rpc error: code = NotFound desc = could not find container \"e2b5fd369c2ad2c4fe36c8fb6a3082aab7b04d00d719d457a827c7808df8e47d\": container with ID starting with e2b5fd369c2ad2c4fe36c8fb6a3082aab7b04d00d719d457a827c7808df8e47d not found: ID does not exist"
Sep 10 15:41:15.677166 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.677133    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"]
Sep 10 15:41:15.681000 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:15.680967    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-62d2f7/rate-test-model-62d2f7-kserve-b55bc9579-zpjxg"]
Sep 10 15:41:16.126001 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:16.125956    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06101ebd-fc0e-44db-acbc-a7e03898db7f" path="/var/lib/kubelet/pods/06101ebd-fc0e-44db-acbc-a7e03898db7f/volumes"
Sep 10 15:41:18.623755 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:18.623722    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"
Sep 10 15:41:35.297752 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.297713    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"]
Sep 10 15:41:35.298242 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.298121    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06101ebd-fc0e-44db-acbc-a7e03898db7f" containerName="main"
Sep 10 15:41:35.298242 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.298134    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="06101ebd-fc0e-44db-acbc-a7e03898db7f" containerName="main"
Sep 10 15:41:35.298242 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.298220    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="06101ebd-fc0e-44db-acbc-a7e03898db7f" containerName="main"
Sep 10 15:41:35.302628 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.302606    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"
Sep 10 15:41:35.306544 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.306519    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-09b833-dockercfg-v44d8\""
Sep 10 15:41:35.306884 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.306866    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-09b833\""
Sep 10 15:41:35.334430 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.334397    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8qxk\" (UniqueName: \"kubernetes.io/projected/63af54d3-85f5-480c-b792-60619b27081e-kube-api-access-h8qxk\") pod \"payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb\" (UID: \"63af54d3-85f5-480c-b792-60619b27081e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"
Sep 10 15:41:35.334684 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.334664    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63af54d3-85f5-480c-b792-60619b27081e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb\" (UID: \"63af54d3-85f5-480c-b792-60619b27081e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"
Sep 10 15:41:35.336626 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.336604    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"]
Sep 10 15:41:35.435663 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.435628    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63af54d3-85f5-480c-b792-60619b27081e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb\" (UID: \"63af54d3-85f5-480c-b792-60619b27081e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"
Sep 10 15:41:35.435864 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.435708    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8qxk\" (UniqueName: \"kubernetes.io/projected/63af54d3-85f5-480c-b792-60619b27081e-kube-api-access-h8qxk\") pod \"payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb\" (UID: \"63af54d3-85f5-480c-b792-60619b27081e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"
Sep 10 15:41:35.436354 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.436331    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63af54d3-85f5-480c-b792-60619b27081e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb\" (UID: \"63af54d3-85f5-480c-b792-60619b27081e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"
Sep 10 15:41:35.444351 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.444324    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8qxk\" (UniqueName: \"kubernetes.io/projected/63af54d3-85f5-480c-b792-60619b27081e-kube-api-access-h8qxk\") pod \"payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb\" (UID: \"63af54d3-85f5-480c-b792-60619b27081e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"
Sep 10 15:41:35.613319 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.613281    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"
Sep 10 15:41:35.887866 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:35.887767    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"]
Sep 10 15:41:35.891190 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:41:35.891164    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63af54d3_85f5_480c_b792_60619b27081e.slice/crio-aa039d871ca77b3f319b782c4416dee0d86bb6e456b45cebc22244d4e2a1f1e5 WatchSource:0}: Error finding container aa039d871ca77b3f319b782c4416dee0d86bb6e456b45cebc22244d4e2a1f1e5: Status 404 returned error can't find the container with id aa039d871ca77b3f319b782c4416dee0d86bb6e456b45cebc22244d4e2a1f1e5
Sep 10 15:41:36.761284 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:36.761239    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb" event={"ID":"63af54d3-85f5-480c-b792-60619b27081e","Type":"ContainerStarted","Data":"f477c6514f15bdb32faa16c742fd66dd6a9384a1de8e3a80c3680702d2752206"}
Sep 10 15:41:36.761772 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:36.761306    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb" event={"ID":"63af54d3-85f5-480c-b792-60619b27081e","Type":"ContainerStarted","Data":"aa039d871ca77b3f319b782c4416dee0d86bb6e456b45cebc22244d4e2a1f1e5"}
Sep 10 15:41:36.761772 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:36.761361    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"
Sep 10 15:41:36.777364 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:36.777309    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb" podStartSLOduration=1.7772915660000002 podStartE2EDuration="1.777291566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:36.776348939 +0000 UTC m=+1559.192144760" watchObservedRunningTime="2026-09-10 15:41:36.777291566 +0000 UTC m=+1559.193087388"
Sep 10 15:41:47.768031 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:41:47.767994    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"
Sep 10 15:42:03.451245 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:03.451207    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kxctv_4e9b368b-9061-4e9d-be30-4b59ada419c8/payload-processing/0.log"
Sep 10 15:42:20.137772 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.137739    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"]
Sep 10 15:42:20.141295 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.141274    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.143424 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.143405    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ba5c7e\"/\"kube-root-ca.crt\""
Sep 10 15:42:20.144289 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.144273    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ba5c7e\"/\"openshift-service-ca.crt\""
Sep 10 15:42:20.144289 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.144279    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ba5c7e\"/\"ipp-route-ba5c7e-kserve-self-signed-certs\""
Sep 10 15:42:20.144450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.144280    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ba5c7e\"/\"default-dockercfg-r7vqt\""
Sep 10 15:42:20.152739 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.152715    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"]
Sep 10 15:42:20.228130 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.228095    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-home\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.228130 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.228133    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpndt\" (UniqueName: \"kubernetes.io/projected/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-kube-api-access-gpndt\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.228350 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.228151    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-tls-certs\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.228350 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.228193    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-dshm\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.228350 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.228225    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-tmp-dir\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.228350 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.228247    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-model-cache\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.328615 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.328570    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-home\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.328615 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.328616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpndt\" (UniqueName: \"kubernetes.io/projected/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-kube-api-access-gpndt\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.328927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.328642    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-tls-certs\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.328927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.328666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-dshm\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.328927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.328710    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-tmp-dir\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.328927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.328733    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-model-cache\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.329117 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.329089    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-home\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.329238 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.329155    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-model-cache\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.329238 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.329174    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-tmp-dir\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.331206 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.331185    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-dshm\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.331478 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.331460    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-tls-certs\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.336553 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.336533    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpndt\" (UniqueName: \"kubernetes.io/projected/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-kube-api-access-gpndt\") pod \"ipp-route-ba5c7e-kserve-747f7c4879-rpsnv\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") " pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.452169 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.452071    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:20.730777 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.730750    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"]
Sep 10 15:42:20.732850 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:42:20.732822    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21c98154_61fb_4c9b_98f9_bbf1f41c9ed1.slice/crio-80db679573cc759941123fd47965f8133ff514a8b6698cffad5a549a6b327bc0 WatchSource:0}: Error finding container 80db679573cc759941123fd47965f8133ff514a8b6698cffad5a549a6b327bc0: Status 404 returned error can't find the container with id 80db679573cc759941123fd47965f8133ff514a8b6698cffad5a549a6b327bc0
Sep 10 15:42:20.987484 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.987394    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv" event={"ID":"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1","Type":"ContainerStarted","Data":"3500b7002a4b053aee814f5d37d5ad01d3ab5ae17ae8f456003afd3d6566c7e4"}
Sep 10 15:42:20.987484 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:20.987443    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv" event={"ID":"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1","Type":"ContainerStarted","Data":"80db679573cc759941123fd47965f8133ff514a8b6698cffad5a549a6b327bc0"}
Sep 10 15:42:21.003831 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:21.003770    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv" podStartSLOduration=1.003754108 podStartE2EDuration="1.003754108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:21.001619526 +0000 UTC m=+1603.417415347" watchObservedRunningTime="2026-09-10 15:42:21.003754108 +0000 UTC m=+1603.419549955"
Sep 10 15:42:30.452450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:30.452361    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:30.452450 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:30.452418    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:30.465128 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:30.465104    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:31.046215 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:31.046183    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:42:49.354709 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:49.354675    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8_14d79d08-e4f4-4cfd-b605-b2349ef1615e/payload-processing/0.log"
Sep 10 15:42:49.469184 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:49.469152    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kxctv_4e9b368b-9061-4e9d-be30-4b59ada419c8/payload-processing/0.log"
Sep 10 15:42:54.139372 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.139338    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"]
Sep 10 15:42:54.143192 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.143165    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:54.147042 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.147019    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-810029-serving-cert\""
Sep 10 15:42:54.164100 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.164059    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"]
Sep 10 15:42:54.230710 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.230673    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3506c9f-c807-4311-8520-6e9f9bb930d0-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:54.230909 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.230753    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:54.230996 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.230960    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:54.231084 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.231060    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfwsn\" (UniqueName: \"kubernetes.io/projected/f3506c9f-c807-4311-8520-6e9f9bb930d0-kube-api-access-sfwsn\") pod \"maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:54.332563 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.332504    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:54.332563 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.332576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfwsn\" (UniqueName: \"kubernetes.io/projected/f3506c9f-c807-4311-8520-6e9f9bb930d0-kube-api-access-sfwsn\") pod \"maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:54.332839 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.332611    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3506c9f-c807-4311-8520-6e9f9bb930d0-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:54.332839 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:42:54.332668    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-serving-cert: secret "maas-api-e2e-ipp-cleanup-810029-serving-cert" not found
Sep 10 15:42:54.332839 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:42:54.332749    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-maas-api-tls podName:f3506c9f-c807-4311-8520-6e9f9bb930d0 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:54.832728557 +0000 UTC m=+1637.248524357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4" (UID: "f3506c9f-c807-4311-8520-6e9f9bb930d0") : secret "maas-api-e2e-ipp-cleanup-810029-serving-cert" not found
Sep 10 15:42:54.332839 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.332671    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:54.333334 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.333306    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3506c9f-c807-4311-8520-6e9f9bb930d0-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:54.335289 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.335267    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:54.341007 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.340984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfwsn\" (UniqueName: \"kubernetes.io/projected/f3506c9f-c807-4311-8520-6e9f9bb930d0-kube-api-access-sfwsn\") pod \"maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:54.836851 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.836815    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:54.839511 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:54.839487    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:55.054308 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:55.054265    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:55.328976 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:55.328727    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"]
Sep 10 15:42:55.332107 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:42:55.332078    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3506c9f_c807_4311_8520_6e9f9bb930d0.slice/crio-ab52f89e79eec3d2da76c7bd7f65b86ff62df329559cab8316e5aec7713e5a30 WatchSource:0}: Error finding container ab52f89e79eec3d2da76c7bd7f65b86ff62df329559cab8316e5aec7713e5a30: Status 404 returned error can't find the container with id ab52f89e79eec3d2da76c7bd7f65b86ff62df329559cab8316e5aec7713e5a30
Sep 10 15:42:56.163112 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:56.163063    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4" event={"ID":"f3506c9f-c807-4311-8520-6e9f9bb930d0","Type":"ContainerStarted","Data":"c5609bd0634fb7a83176424d1e36a8467891e97e9061f88b1a4c71eb7920e0b7"}
Sep 10 15:42:56.163281 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:56.163135    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4" event={"ID":"f3506c9f-c807-4311-8520-6e9f9bb930d0","Type":"ContainerStarted","Data":"ab52f89e79eec3d2da76c7bd7f65b86ff62df329559cab8316e5aec7713e5a30"}
Sep 10 15:42:56.163281 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:56.163162    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:42:56.179187 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:42:56.179118    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4" podStartSLOduration=1.79095947 podStartE2EDuration="2.179099126s" totalImagesPullingTime="388.139656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:56.178333369 +0000 UTC m=+1638.594129208" watchObservedRunningTime="2026-09-10 15:42:56.179099126 +0000 UTC m=+1638.594894945"
Sep 10 15:43:02.173487 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:02.173455    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:43:24.701645 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:24.701606    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"]
Sep 10 15:43:24.702177 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:24.701880    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4" podUID="f3506c9f-c807-4311-8520-6e9f9bb930d0" containerName="maas-api" containerID="cri-o://c5609bd0634fb7a83176424d1e36a8467891e97e9061f88b1a4c71eb7920e0b7" gracePeriod=30
Sep 10 15:43:24.961896 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:24.961830    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:43:25.017396 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.017361    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-metrics-certs\") pod \"f3506c9f-c807-4311-8520-6e9f9bb930d0\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") "
Sep 10 15:43:25.017576 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.017422    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3506c9f-c807-4311-8520-6e9f9bb930d0-openshift-service-ca\") pod \"f3506c9f-c807-4311-8520-6e9f9bb930d0\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") "
Sep 10 15:43:25.017576 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.017452    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-maas-api-tls\") pod \"f3506c9f-c807-4311-8520-6e9f9bb930d0\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") "
Sep 10 15:43:25.017670 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.017579    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfwsn\" (UniqueName: \"kubernetes.io/projected/f3506c9f-c807-4311-8520-6e9f9bb930d0-kube-api-access-sfwsn\") pod \"f3506c9f-c807-4311-8520-6e9f9bb930d0\" (UID: \"f3506c9f-c807-4311-8520-6e9f9bb930d0\") "
Sep 10 15:43:25.017839 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.017771    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3506c9f-c807-4311-8520-6e9f9bb930d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3506c9f-c807-4311-8520-6e9f9bb930d0" (UID: "f3506c9f-c807-4311-8520-6e9f9bb930d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:43:25.018034 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.018014    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3506c9f-c807-4311-8520-6e9f9bb930d0-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:43:25.019725 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.019696    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3506c9f-c807-4311-8520-6e9f9bb930d0" (UID: "f3506c9f-c807-4311-8520-6e9f9bb930d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:43:25.019892 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.019750    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f3506c9f-c807-4311-8520-6e9f9bb930d0" (UID: "f3506c9f-c807-4311-8520-6e9f9bb930d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:43:25.019892 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.019762    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3506c9f-c807-4311-8520-6e9f9bb930d0-kube-api-access-sfwsn" (OuterVolumeSpecName: "kube-api-access-sfwsn") pod "f3506c9f-c807-4311-8520-6e9f9bb930d0" (UID: "f3506c9f-c807-4311-8520-6e9f9bb930d0"). InnerVolumeSpecName "kube-api-access-sfwsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:43:25.118803 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.118766    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:43:25.118972 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.118824    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfwsn\" (UniqueName: \"kubernetes.io/projected/f3506c9f-c807-4311-8520-6e9f9bb930d0-kube-api-access-sfwsn\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:43:25.118972 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.118835    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3506c9f-c807-4311-8520-6e9f9bb930d0-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:43:25.309769 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.309672    2573 generic.go:356] "Generic (PLEG): container finished" podID="f3506c9f-c807-4311-8520-6e9f9bb930d0" containerID="c5609bd0634fb7a83176424d1e36a8467891e97e9061f88b1a4c71eb7920e0b7" exitCode=0
Sep 10 15:43:25.309769 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.309724    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"
Sep 10 15:43:25.310038 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.309760    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4" event={"ID":"f3506c9f-c807-4311-8520-6e9f9bb930d0","Type":"ContainerDied","Data":"c5609bd0634fb7a83176424d1e36a8467891e97e9061f88b1a4c71eb7920e0b7"}
Sep 10 15:43:25.310038 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.309846    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4" event={"ID":"f3506c9f-c807-4311-8520-6e9f9bb930d0","Type":"ContainerDied","Data":"ab52f89e79eec3d2da76c7bd7f65b86ff62df329559cab8316e5aec7713e5a30"}
Sep 10 15:43:25.310038 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.309880    2573 scope.go:117] "RemoveContainer" containerID="c5609bd0634fb7a83176424d1e36a8467891e97e9061f88b1a4c71eb7920e0b7"
Sep 10 15:43:25.319422 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.319403    2573 scope.go:117] "RemoveContainer" containerID="c5609bd0634fb7a83176424d1e36a8467891e97e9061f88b1a4c71eb7920e0b7"
Sep 10 15:43:25.319692 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:43:25.319673    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5609bd0634fb7a83176424d1e36a8467891e97e9061f88b1a4c71eb7920e0b7\": container with ID starting with c5609bd0634fb7a83176424d1e36a8467891e97e9061f88b1a4c71eb7920e0b7 not found: ID does not exist" containerID="c5609bd0634fb7a83176424d1e36a8467891e97e9061f88b1a4c71eb7920e0b7"
Sep 10 15:43:25.319741 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.319700    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5609bd0634fb7a83176424d1e36a8467891e97e9061f88b1a4c71eb7920e0b7"} err="failed to get container status \"c5609bd0634fb7a83176424d1e36a8467891e97e9061f88b1a4c71eb7920e0b7\": rpc error: code = NotFound desc = could not find container \"c5609bd0634fb7a83176424d1e36a8467891e97e9061f88b1a4c71eb7920e0b7\": container with ID starting with c5609bd0634fb7a83176424d1e36a8467891e97e9061f88b1a4c71eb7920e0b7 not found: ID does not exist"
Sep 10 15:43:25.333294 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.333269    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"]
Sep 10 15:43:25.340964 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:25.340942    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-810029-75cd7674bb-422m4"]
Sep 10 15:43:26.126187 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:26.126153    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3506c9f-c807-4311-8520-6e9f9bb930d0" path="/var/lib/kubelet/pods/f3506c9f-c807-4311-8520-6e9f9bb930d0/volumes"
Sep 10 15:43:51.087558 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.087524    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"]
Sep 10 15:43:51.088106 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.087717    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8" podUID="14d79d08-e4f4-4cfd-b605-b2349ef1615e" containerName="payload-processing" containerID="cri-o://ac1988b731213ada238ad2518254b7fa291e227f572310d7b1a5d04905277877" gracePeriod=30
Sep 10 15:43:51.346429 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.346366    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"
Sep 10 15:43:51.443874 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.443841    2573 generic.go:356] "Generic (PLEG): container finished" podID="14d79d08-e4f4-4cfd-b605-b2349ef1615e" containerID="ac1988b731213ada238ad2518254b7fa291e227f572310d7b1a5d04905277877" exitCode=0
Sep 10 15:43:51.444035 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.443884    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"
Sep 10 15:43:51.444035 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.443923    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8" event={"ID":"14d79d08-e4f4-4cfd-b605-b2349ef1615e","Type":"ContainerDied","Data":"ac1988b731213ada238ad2518254b7fa291e227f572310d7b1a5d04905277877"}
Sep 10 15:43:51.444035 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.443981    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8" event={"ID":"14d79d08-e4f4-4cfd-b605-b2349ef1615e","Type":"ContainerDied","Data":"f72d6850cc8d66095ae7f31cb91141c1c131739bacb530286634ce1d14ced09f"}
Sep 10 15:43:51.444035 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.444005    2573 scope.go:117] "RemoveContainer" containerID="ac1988b731213ada238ad2518254b7fa291e227f572310d7b1a5d04905277877"
Sep 10 15:43:51.447967 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.447944    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq6c8\" (UniqueName: \"kubernetes.io/projected/14d79d08-e4f4-4cfd-b605-b2349ef1615e-kube-api-access-xq6c8\") pod \"14d79d08-e4f4-4cfd-b605-b2349ef1615e\" (UID: \"14d79d08-e4f4-4cfd-b605-b2349ef1615e\") "
Sep 10 15:43:51.448095 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.447981    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d79d08-e4f4-4cfd-b605-b2349ef1615e-plugins-config-volume\") pod \"14d79d08-e4f4-4cfd-b605-b2349ef1615e\" (UID: \"14d79d08-e4f4-4cfd-b605-b2349ef1615e\") "
Sep 10 15:43:51.448387 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.448366    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d79d08-e4f4-4cfd-b605-b2349ef1615e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14d79d08-e4f4-4cfd-b605-b2349ef1615e" (UID: "14d79d08-e4f4-4cfd-b605-b2349ef1615e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:43:51.450258 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.450238    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d79d08-e4f4-4cfd-b605-b2349ef1615e-kube-api-access-xq6c8" (OuterVolumeSpecName: "kube-api-access-xq6c8") pod "14d79d08-e4f4-4cfd-b605-b2349ef1615e" (UID: "14d79d08-e4f4-4cfd-b605-b2349ef1615e"). InnerVolumeSpecName "kube-api-access-xq6c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:43:51.452945 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.452911    2573 scope.go:117] "RemoveContainer" containerID="ac1988b731213ada238ad2518254b7fa291e227f572310d7b1a5d04905277877"
Sep 10 15:43:51.453244 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:43:51.453226    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1988b731213ada238ad2518254b7fa291e227f572310d7b1a5d04905277877\": container with ID starting with ac1988b731213ada238ad2518254b7fa291e227f572310d7b1a5d04905277877 not found: ID does not exist" containerID="ac1988b731213ada238ad2518254b7fa291e227f572310d7b1a5d04905277877"
Sep 10 15:43:51.453309 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.453252    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1988b731213ada238ad2518254b7fa291e227f572310d7b1a5d04905277877"} err="failed to get container status \"ac1988b731213ada238ad2518254b7fa291e227f572310d7b1a5d04905277877\": rpc error: code = NotFound desc = could not find container \"ac1988b731213ada238ad2518254b7fa291e227f572310d7b1a5d04905277877\": container with ID starting with ac1988b731213ada238ad2518254b7fa291e227f572310d7b1a5d04905277877 not found: ID does not exist"
Sep 10 15:43:51.548880 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.548845    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq6c8\" (UniqueName: \"kubernetes.io/projected/14d79d08-e4f4-4cfd-b605-b2349ef1615e-kube-api-access-xq6c8\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:43:51.548880 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.548874    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d79d08-e4f4-4cfd-b605-b2349ef1615e-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:43:51.764631 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.764596    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"]
Sep 10 15:43:51.773376 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:51.773340    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba5c7e-6f45d9db47-5qlq8"]
Sep 10 15:43:52.126435 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:52.126397    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14d79d08-e4f4-4cfd-b605-b2349ef1615e" path="/var/lib/kubelet/pods/14d79d08-e4f4-4cfd-b605-b2349ef1615e/volumes"
Sep 10 15:43:56.576203 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:43:56.576159    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"]
Sep 10 15:44:11.596673 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.596572    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv" podUID="21c98154-61fb-4c9b-98f9-bbf1f41c9ed1" containerName="main" containerID="cri-o://3500b7002a4b053aee814f5d37d5ad01d3ab5ae17ae8f456003afd3d6566c7e4" gracePeriod=45
Sep 10 15:44:11.851250 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.851184    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:44:11.932226 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.932193    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-dshm\") pod \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") "
Sep 10 15:44:11.932226 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.932230    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-model-cache\") pod \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") "
Sep 10 15:44:11.932475 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.932282    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-home\") pod \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") "
Sep 10 15:44:11.932475 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.932336    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpndt\" (UniqueName: \"kubernetes.io/projected/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-kube-api-access-gpndt\") pod \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") "
Sep 10 15:44:11.932475 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.932364    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-tmp-dir\") pod \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") "
Sep 10 15:44:11.932475 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.932413    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-tls-certs\") pod \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\" (UID: \"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1\") "
Sep 10 15:44:11.932685 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.932611    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-model-cache" (OuterVolumeSpecName: "model-cache") pod "21c98154-61fb-4c9b-98f9-bbf1f41c9ed1" (UID: "21c98154-61fb-4c9b-98f9-bbf1f41c9ed1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:44:11.932685 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.932596    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-home" (OuterVolumeSpecName: "home") pod "21c98154-61fb-4c9b-98f9-bbf1f41c9ed1" (UID: "21c98154-61fb-4c9b-98f9-bbf1f41c9ed1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:44:11.932685 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.932659    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21c98154-61fb-4c9b-98f9-bbf1f41c9ed1" (UID: "21c98154-61fb-4c9b-98f9-bbf1f41c9ed1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:44:11.934599 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.934575    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-dshm" (OuterVolumeSpecName: "dshm") pod "21c98154-61fb-4c9b-98f9-bbf1f41c9ed1" (UID: "21c98154-61fb-4c9b-98f9-bbf1f41c9ed1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:44:11.934710 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.934619    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21c98154-61fb-4c9b-98f9-bbf1f41c9ed1" (UID: "21c98154-61fb-4c9b-98f9-bbf1f41c9ed1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:44:11.934710 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:11.934652    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-kube-api-access-gpndt" (OuterVolumeSpecName: "kube-api-access-gpndt") pod "21c98154-61fb-4c9b-98f9-bbf1f41c9ed1" (UID: "21c98154-61fb-4c9b-98f9-bbf1f41c9ed1"). InnerVolumeSpecName "kube-api-access-gpndt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:44:12.033745 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.033703    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-dshm\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:44:12.033745 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.033748    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-model-cache\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:44:12.033973 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.033758    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-home\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:44:12.033973 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.033767    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpndt\" (UniqueName: \"kubernetes.io/projected/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-kube-api-access-gpndt\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:44:12.033973 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.033776    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-tmp-dir\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:44:12.033973 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.033785    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1-tls-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:44:12.550673 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.550583    2573 generic.go:356] "Generic (PLEG): container finished" podID="21c98154-61fb-4c9b-98f9-bbf1f41c9ed1" containerID="3500b7002a4b053aee814f5d37d5ad01d3ab5ae17ae8f456003afd3d6566c7e4" exitCode=0
Sep 10 15:44:12.550673 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.550648    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"
Sep 10 15:44:12.550673 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.550652    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv" event={"ID":"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1","Type":"ContainerDied","Data":"3500b7002a4b053aee814f5d37d5ad01d3ab5ae17ae8f456003afd3d6566c7e4"}
Sep 10 15:44:12.550954 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.550699    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv" event={"ID":"21c98154-61fb-4c9b-98f9-bbf1f41c9ed1","Type":"ContainerDied","Data":"80db679573cc759941123fd47965f8133ff514a8b6698cffad5a549a6b327bc0"}
Sep 10 15:44:12.550954 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.550723    2573 scope.go:117] "RemoveContainer" containerID="3500b7002a4b053aee814f5d37d5ad01d3ab5ae17ae8f456003afd3d6566c7e4"
Sep 10 15:44:12.559634 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.559611    2573 scope.go:117] "RemoveContainer" containerID="3500b7002a4b053aee814f5d37d5ad01d3ab5ae17ae8f456003afd3d6566c7e4"
Sep 10 15:44:12.559965 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:44:12.559939    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3500b7002a4b053aee814f5d37d5ad01d3ab5ae17ae8f456003afd3d6566c7e4\": container with ID starting with 3500b7002a4b053aee814f5d37d5ad01d3ab5ae17ae8f456003afd3d6566c7e4 not found: ID does not exist" containerID="3500b7002a4b053aee814f5d37d5ad01d3ab5ae17ae8f456003afd3d6566c7e4"
Sep 10 15:44:12.560070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.559971    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3500b7002a4b053aee814f5d37d5ad01d3ab5ae17ae8f456003afd3d6566c7e4"} err="failed to get container status \"3500b7002a4b053aee814f5d37d5ad01d3ab5ae17ae8f456003afd3d6566c7e4\": rpc error: code = NotFound desc = could not find container \"3500b7002a4b053aee814f5d37d5ad01d3ab5ae17ae8f456003afd3d6566c7e4\": container with ID starting with 3500b7002a4b053aee814f5d37d5ad01d3ab5ae17ae8f456003afd3d6566c7e4 not found: ID does not exist"
Sep 10 15:44:12.566164 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.566135    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"]
Sep 10 15:44:12.569116 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:12.569086    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-ba5c7e/ipp-route-ba5c7e-kserve-747f7c4879-rpsnv"]
Sep 10 15:44:14.126113 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:14.126079    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21c98154-61fb-4c9b-98f9-bbf1f41c9ed1" path="/var/lib/kubelet/pods/21c98154-61fb-4c9b-98f9-bbf1f41c9ed1/volumes"
Sep 10 15:44:51.102120 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.102080    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"]
Sep 10 15:44:51.102719 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.102383    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb" podUID="63af54d3-85f5-480c-b792-60619b27081e" containerName="payload-pre-processing" containerID="cri-o://f477c6514f15bdb32faa16c742fd66dd6a9384a1de8e3a80c3680702d2752206" gracePeriod=30
Sep 10 15:44:51.361230 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.361161    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"
Sep 10 15:44:51.477818 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.477752    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63af54d3-85f5-480c-b792-60619b27081e-plugins-config-volume\") pod \"63af54d3-85f5-480c-b792-60619b27081e\" (UID: \"63af54d3-85f5-480c-b792-60619b27081e\") "
Sep 10 15:44:51.477990 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.477863    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8qxk\" (UniqueName: \"kubernetes.io/projected/63af54d3-85f5-480c-b792-60619b27081e-kube-api-access-h8qxk\") pod \"63af54d3-85f5-480c-b792-60619b27081e\" (UID: \"63af54d3-85f5-480c-b792-60619b27081e\") "
Sep 10 15:44:51.478122 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.478096    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63af54d3-85f5-480c-b792-60619b27081e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63af54d3-85f5-480c-b792-60619b27081e" (UID: "63af54d3-85f5-480c-b792-60619b27081e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:44:51.480077 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.480048    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63af54d3-85f5-480c-b792-60619b27081e-kube-api-access-h8qxk" (OuterVolumeSpecName: "kube-api-access-h8qxk") pod "63af54d3-85f5-480c-b792-60619b27081e" (UID: "63af54d3-85f5-480c-b792-60619b27081e"). InnerVolumeSpecName "kube-api-access-h8qxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:44:51.579427 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.579389    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63af54d3-85f5-480c-b792-60619b27081e-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:44:51.579427 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.579419    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8qxk\" (UniqueName: \"kubernetes.io/projected/63af54d3-85f5-480c-b792-60619b27081e-kube-api-access-h8qxk\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:44:51.748107 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.748009    2573 generic.go:356] "Generic (PLEG): container finished" podID="63af54d3-85f5-480c-b792-60619b27081e" containerID="f477c6514f15bdb32faa16c742fd66dd6a9384a1de8e3a80c3680702d2752206" exitCode=0
Sep 10 15:44:51.748107 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.748062    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"
Sep 10 15:44:51.748107 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.748068    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb" event={"ID":"63af54d3-85f5-480c-b792-60619b27081e","Type":"ContainerDied","Data":"f477c6514f15bdb32faa16c742fd66dd6a9384a1de8e3a80c3680702d2752206"}
Sep 10 15:44:51.748385 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.748123    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb" event={"ID":"63af54d3-85f5-480c-b792-60619b27081e","Type":"ContainerDied","Data":"aa039d871ca77b3f319b782c4416dee0d86bb6e456b45cebc22244d4e2a1f1e5"}
Sep 10 15:44:51.748385 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.748152    2573 scope.go:117] "RemoveContainer" containerID="f477c6514f15bdb32faa16c742fd66dd6a9384a1de8e3a80c3680702d2752206"
Sep 10 15:44:51.757150 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.757126    2573 scope.go:117] "RemoveContainer" containerID="f477c6514f15bdb32faa16c742fd66dd6a9384a1de8e3a80c3680702d2752206"
Sep 10 15:44:51.757423 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:44:51.757403    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f477c6514f15bdb32faa16c742fd66dd6a9384a1de8e3a80c3680702d2752206\": container with ID starting with f477c6514f15bdb32faa16c742fd66dd6a9384a1de8e3a80c3680702d2752206 not found: ID does not exist" containerID="f477c6514f15bdb32faa16c742fd66dd6a9384a1de8e3a80c3680702d2752206"
Sep 10 15:44:51.757472 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.757439    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f477c6514f15bdb32faa16c742fd66dd6a9384a1de8e3a80c3680702d2752206"} err="failed to get container status \"f477c6514f15bdb32faa16c742fd66dd6a9384a1de8e3a80c3680702d2752206\": rpc error: code = NotFound desc = could not find container \"f477c6514f15bdb32faa16c742fd66dd6a9384a1de8e3a80c3680702d2752206\": container with ID starting with f477c6514f15bdb32faa16c742fd66dd6a9384a1de8e3a80c3680702d2752206 not found: ID does not exist"
Sep 10 15:44:51.767739 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.767718    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"]
Sep 10 15:44:51.771837 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:51.771811    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09b833-76465cf9bb-b2jkb"]
Sep 10 15:44:52.126731 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:44:52.126694    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63af54d3-85f5-480c-b792-60619b27081e" path="/var/lib/kubelet/pods/63af54d3-85f5-480c-b792-60619b27081e/volumes"
Sep 10 15:45:08.822061 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:08.822026    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"]
Sep 10 15:45:08.822664 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:08.822367    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f" podUID="9fd90bcf-76fa-42c2-bd3a-8a1834d05d00" containerName="maas-api" containerID="cri-o://d633dbc6a86aa860b00765bd6e36a9edeeb1b712c611c2b27178870a86e89aa4" gracePeriod=30
Sep 10 15:45:08.862345 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:08.862315    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"]
Sep 10 15:45:08.862582 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:08.862533    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv" podUID="c53f4421-3ed3-4cad-b985-11184c2447fb" containerName="payload-processing" containerID="cri-o://bbfc2435985a1dc3446d12f7fbd159a2542d73a0fc703b7051b5112105fbae8a" gracePeriod=30
Sep 10 15:45:09.113192 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.113162    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:45:09.143014 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.142989    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"
Sep 10 15:45:09.236727 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.236692    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-metrics-certs\") pod \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") "
Sep 10 15:45:09.236727 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.236733    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66swq\" (UniqueName: \"kubernetes.io/projected/c53f4421-3ed3-4cad-b985-11184c2447fb-kube-api-access-66swq\") pod \"c53f4421-3ed3-4cad-b985-11184c2447fb\" (UID: \"c53f4421-3ed3-4cad-b985-11184c2447fb\") "
Sep 10 15:45:09.237009 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.236763    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-openshift-service-ca\") pod \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") "
Sep 10 15:45:09.237009 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.236804    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls\") pod \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") "
Sep 10 15:45:09.237009 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.236958    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncgzt\" (UniqueName: \"kubernetes.io/projected/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-kube-api-access-ncgzt\") pod \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\" (UID: \"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00\") "
Sep 10 15:45:09.237176 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.237081    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53f4421-3ed3-4cad-b985-11184c2447fb-plugins-config-volume\") pod \"c53f4421-3ed3-4cad-b985-11184c2447fb\" (UID: \"c53f4421-3ed3-4cad-b985-11184c2447fb\") "
Sep 10 15:45:09.237176 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.237085    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9fd90bcf-76fa-42c2-bd3a-8a1834d05d00" (UID: "9fd90bcf-76fa-42c2-bd3a-8a1834d05d00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:45:09.237375 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.237352    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:09.237660 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.237467    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53f4421-3ed3-4cad-b985-11184c2447fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c53f4421-3ed3-4cad-b985-11184c2447fb" (UID: "c53f4421-3ed3-4cad-b985-11184c2447fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:45:09.239150 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.239119    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-kube-api-access-ncgzt" (OuterVolumeSpecName: "kube-api-access-ncgzt") pod "9fd90bcf-76fa-42c2-bd3a-8a1834d05d00" (UID: "9fd90bcf-76fa-42c2-bd3a-8a1834d05d00"). InnerVolumeSpecName "kube-api-access-ncgzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:45:09.239248 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.239166    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53f4421-3ed3-4cad-b985-11184c2447fb-kube-api-access-66swq" (OuterVolumeSpecName: "kube-api-access-66swq") pod "c53f4421-3ed3-4cad-b985-11184c2447fb" (UID: "c53f4421-3ed3-4cad-b985-11184c2447fb"). InnerVolumeSpecName "kube-api-access-66swq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:45:09.239533 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.239518    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9fd90bcf-76fa-42c2-bd3a-8a1834d05d00" (UID: "9fd90bcf-76fa-42c2-bd3a-8a1834d05d00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:45:09.239576 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.239535    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9fd90bcf-76fa-42c2-bd3a-8a1834d05d00" (UID: "9fd90bcf-76fa-42c2-bd3a-8a1834d05d00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:45:09.338105 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.338068    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncgzt\" (UniqueName: \"kubernetes.io/projected/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-kube-api-access-ncgzt\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:09.338105 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.338099    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53f4421-3ed3-4cad-b985-11184c2447fb-plugins-config-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:09.338105 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.338112    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:09.338348 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.338120    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66swq\" (UniqueName: \"kubernetes.io/projected/c53f4421-3ed3-4cad-b985-11184c2447fb-kube-api-access-66swq\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:09.338348 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.338129    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:09.836747 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.836711    2573 generic.go:356] "Generic (PLEG): container finished" podID="c53f4421-3ed3-4cad-b985-11184c2447fb" containerID="bbfc2435985a1dc3446d12f7fbd159a2542d73a0fc703b7051b5112105fbae8a" exitCode=0
Sep 10 15:45:09.837201 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.836758    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"
Sep 10 15:45:09.837201 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.836811    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv" event={"ID":"c53f4421-3ed3-4cad-b985-11184c2447fb","Type":"ContainerDied","Data":"bbfc2435985a1dc3446d12f7fbd159a2542d73a0fc703b7051b5112105fbae8a"}
Sep 10 15:45:09.837201 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.836868    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv" event={"ID":"c53f4421-3ed3-4cad-b985-11184c2447fb","Type":"ContainerDied","Data":"957035ada1314ac3721f318cd42b72812805198692f7545e79b9a7545c38022b"}
Sep 10 15:45:09.837201 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.836897    2573 scope.go:117] "RemoveContainer" containerID="bbfc2435985a1dc3446d12f7fbd159a2542d73a0fc703b7051b5112105fbae8a"
Sep 10 15:45:09.838650 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.838624    2573 generic.go:356] "Generic (PLEG): container finished" podID="9fd90bcf-76fa-42c2-bd3a-8a1834d05d00" containerID="d633dbc6a86aa860b00765bd6e36a9edeeb1b712c611c2b27178870a86e89aa4" exitCode=0
Sep 10 15:45:09.838781 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.838693    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f" event={"ID":"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00","Type":"ContainerDied","Data":"d633dbc6a86aa860b00765bd6e36a9edeeb1b712c611c2b27178870a86e89aa4"}
Sep 10 15:45:09.838781 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.838723    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"
Sep 10 15:45:09.838781 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.838739    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f" event={"ID":"9fd90bcf-76fa-42c2-bd3a-8a1834d05d00","Type":"ContainerDied","Data":"18b137a5f3d405f211d2a5cd536a6d156ebed11c7a7008643f0b21c1b24622ab"}
Sep 10 15:45:09.846436 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.846417    2573 scope.go:117] "RemoveContainer" containerID="bbfc2435985a1dc3446d12f7fbd159a2542d73a0fc703b7051b5112105fbae8a"
Sep 10 15:45:09.846755 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:45:09.846731    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbfc2435985a1dc3446d12f7fbd159a2542d73a0fc703b7051b5112105fbae8a\": container with ID starting with bbfc2435985a1dc3446d12f7fbd159a2542d73a0fc703b7051b5112105fbae8a not found: ID does not exist" containerID="bbfc2435985a1dc3446d12f7fbd159a2542d73a0fc703b7051b5112105fbae8a"
Sep 10 15:45:09.846877 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.846764    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbfc2435985a1dc3446d12f7fbd159a2542d73a0fc703b7051b5112105fbae8a"} err="failed to get container status \"bbfc2435985a1dc3446d12f7fbd159a2542d73a0fc703b7051b5112105fbae8a\": rpc error: code = NotFound desc = could not find container \"bbfc2435985a1dc3446d12f7fbd159a2542d73a0fc703b7051b5112105fbae8a\": container with ID starting with bbfc2435985a1dc3446d12f7fbd159a2542d73a0fc703b7051b5112105fbae8a not found: ID does not exist"
Sep 10 15:45:09.846877 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.846839    2573 scope.go:117] "RemoveContainer" containerID="d633dbc6a86aa860b00765bd6e36a9edeeb1b712c611c2b27178870a86e89aa4"
Sep 10 15:45:09.857175 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.857154    2573 scope.go:117] "RemoveContainer" containerID="d633dbc6a86aa860b00765bd6e36a9edeeb1b712c611c2b27178870a86e89aa4"
Sep 10 15:45:09.857519 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:45:09.857498    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d633dbc6a86aa860b00765bd6e36a9edeeb1b712c611c2b27178870a86e89aa4\": container with ID starting with d633dbc6a86aa860b00765bd6e36a9edeeb1b712c611c2b27178870a86e89aa4 not found: ID does not exist" containerID="d633dbc6a86aa860b00765bd6e36a9edeeb1b712c611c2b27178870a86e89aa4"
Sep 10 15:45:09.857611 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.857524    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d633dbc6a86aa860b00765bd6e36a9edeeb1b712c611c2b27178870a86e89aa4"} err="failed to get container status \"d633dbc6a86aa860b00765bd6e36a9edeeb1b712c611c2b27178870a86e89aa4\": rpc error: code = NotFound desc = could not find container \"d633dbc6a86aa860b00765bd6e36a9edeeb1b712c611c2b27178870a86e89aa4\": container with ID starting with d633dbc6a86aa860b00765bd6e36a9edeeb1b712c611c2b27178870a86e89aa4 not found: ID does not exist"
Sep 10 15:45:09.858429 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.858405    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"]
Sep 10 15:45:09.861846 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.861827    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-62d2f7-7cf58488df-sqfvv"]
Sep 10 15:45:09.870162 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.870138    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"]
Sep 10 15:45:09.872157 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:09.872134    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-62d2f7-6f577c6b69-br99f"]
Sep 10 15:45:10.125823 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:10.125727    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fd90bcf-76fa-42c2-bd3a-8a1834d05d00" path="/var/lib/kubelet/pods/9fd90bcf-76fa-42c2-bd3a-8a1834d05d00/volumes"
Sep 10 15:45:10.126135 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:10.126120    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c53f4421-3ed3-4cad-b985-11184c2447fb" path="/var/lib/kubelet/pods/c53f4421-3ed3-4cad-b985-11184c2447fb/volumes"
Sep 10 15:45:15.596014 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:15.595982    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"]
Sep 10 15:45:15.596614 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:15.596196    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerName="istio-proxy" containerID="cri-o://e47c6c4abf76a02bb727954b8d814d9a349df124fca3e9e9788d1d5a48f74d3b" gracePeriod=30
Sep 10 15:45:20.848520 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:20.848494    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:45:20.894467 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:20.894358    2573 generic.go:356] "Generic (PLEG): container finished" podID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerID="e47c6c4abf76a02bb727954b8d814d9a349df124fca3e9e9788d1d5a48f74d3b" exitCode=0
Sep 10 15:45:20.894467 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:20.894418    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" event={"ID":"d547ad4b-a726-422e-aaeb-a5293584e4ac","Type":"ContainerDied","Data":"e47c6c4abf76a02bb727954b8d814d9a349df124fca3e9e9788d1d5a48f74d3b"}
Sep 10 15:45:20.894720 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:20.894489    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7" event={"ID":"d547ad4b-a726-422e-aaeb-a5293584e4ac","Type":"ContainerDied","Data":"1b62e2314cca39df2ff0698b376424f178217fa36bbf16a78ce53aa5037ae3e2"}
Sep 10 15:45:20.894720 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:20.894423    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"
Sep 10 15:45:20.894720 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:20.894520    2573 scope.go:117] "RemoveContainer" containerID="e47c6c4abf76a02bb727954b8d814d9a349df124fca3e9e9788d1d5a48f74d3b"
Sep 10 15:45:20.904106 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:20.904072    2573 scope.go:117] "RemoveContainer" containerID="146b263e0470607ba52442e620822d608f6d944366fdb80b3d75d449aa21d8b6"
Sep 10 15:45:20.911760 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:20.911738    2573 scope.go:117] "RemoveContainer" containerID="e47c6c4abf76a02bb727954b8d814d9a349df124fca3e9e9788d1d5a48f74d3b"
Sep 10 15:45:20.912059 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:45:20.912036    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e47c6c4abf76a02bb727954b8d814d9a349df124fca3e9e9788d1d5a48f74d3b\": container with ID starting with e47c6c4abf76a02bb727954b8d814d9a349df124fca3e9e9788d1d5a48f74d3b not found: ID does not exist" containerID="e47c6c4abf76a02bb727954b8d814d9a349df124fca3e9e9788d1d5a48f74d3b"
Sep 10 15:45:20.912126 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:20.912071    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e47c6c4abf76a02bb727954b8d814d9a349df124fca3e9e9788d1d5a48f74d3b"} err="failed to get container status \"e47c6c4abf76a02bb727954b8d814d9a349df124fca3e9e9788d1d5a48f74d3b\": rpc error: code = NotFound desc = could not find container \"e47c6c4abf76a02bb727954b8d814d9a349df124fca3e9e9788d1d5a48f74d3b\": container with ID starting with e47c6c4abf76a02bb727954b8d814d9a349df124fca3e9e9788d1d5a48f74d3b not found: ID does not exist"
Sep 10 15:45:20.912126 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:20.912090    2573 scope.go:117] "RemoveContainer" containerID="146b263e0470607ba52442e620822d608f6d944366fdb80b3d75d449aa21d8b6"
Sep 10 15:45:20.912385 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:45:20.912367    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146b263e0470607ba52442e620822d608f6d944366fdb80b3d75d449aa21d8b6\": container with ID starting with 146b263e0470607ba52442e620822d608f6d944366fdb80b3d75d449aa21d8b6 not found: ID does not exist" containerID="146b263e0470607ba52442e620822d608f6d944366fdb80b3d75d449aa21d8b6"
Sep 10 15:45:20.912426 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:20.912391    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146b263e0470607ba52442e620822d608f6d944366fdb80b3d75d449aa21d8b6"} err="failed to get container status \"146b263e0470607ba52442e620822d608f6d944366fdb80b3d75d449aa21d8b6\": rpc error: code = NotFound desc = could not find container \"146b263e0470607ba52442e620822d608f6d944366fdb80b3d75d449aa21d8b6\": container with ID starting with 146b263e0470607ba52442e620822d608f6d944366fdb80b3d75d449aa21d8b6 not found: ID does not exist"
Sep 10 15:45:21.038473 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.038437    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-workload-certs\") pod \"d547ad4b-a726-422e-aaeb-a5293584e4ac\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") "
Sep 10 15:45:21.038473 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.038474    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-podinfo\") pod \"d547ad4b-a726-422e-aaeb-a5293584e4ac\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") "
Sep 10 15:45:21.038732 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.038501    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-workload-socket\") pod \"d547ad4b-a726-422e-aaeb-a5293584e4ac\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") "
Sep 10 15:45:21.038732 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.038523    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d547ad4b-a726-422e-aaeb-a5293584e4ac-istiod-ca-cert\") pod \"d547ad4b-a726-422e-aaeb-a5293584e4ac\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") "
Sep 10 15:45:21.038732 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.038593    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-759fj\" (UniqueName: \"kubernetes.io/projected/d547ad4b-a726-422e-aaeb-a5293584e4ac-kube-api-access-759fj\") pod \"d547ad4b-a726-422e-aaeb-a5293584e4ac\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") "
Sep 10 15:45:21.038732 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.038627    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-credential-socket\") pod \"d547ad4b-a726-422e-aaeb-a5293584e4ac\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") "
Sep 10 15:45:21.038732 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.038653    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-envoy\") pod \"d547ad4b-a726-422e-aaeb-a5293584e4ac\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") "
Sep 10 15:45:21.038732 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.038672    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-data\") pod \"d547ad4b-a726-422e-aaeb-a5293584e4ac\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") "
Sep 10 15:45:21.038732 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.038694    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-token\") pod \"d547ad4b-a726-422e-aaeb-a5293584e4ac\" (UID: \"d547ad4b-a726-422e-aaeb-a5293584e4ac\") "
Sep 10 15:45:21.039135 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.038763    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d547ad4b-a726-422e-aaeb-a5293584e4ac" (UID: "d547ad4b-a726-422e-aaeb-a5293584e4ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:21.039135 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.038897    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d547ad4b-a726-422e-aaeb-a5293584e4ac" (UID: "d547ad4b-a726-422e-aaeb-a5293584e4ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:21.039135 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.039006    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-workload-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:21.039135 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.039047    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-credential-socket\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:21.039135 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.039077    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d547ad4b-a726-422e-aaeb-a5293584e4ac" (UID: "d547ad4b-a726-422e-aaeb-a5293584e4ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:21.039135 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.039098    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "d547ad4b-a726-422e-aaeb-a5293584e4ac" (UID: "d547ad4b-a726-422e-aaeb-a5293584e4ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:21.039613 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.039364    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d547ad4b-a726-422e-aaeb-a5293584e4ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d547ad4b-a726-422e-aaeb-a5293584e4ac" (UID: "d547ad4b-a726-422e-aaeb-a5293584e4ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:45:21.040981 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.040946    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "d547ad4b-a726-422e-aaeb-a5293584e4ac" (UID: "d547ad4b-a726-422e-aaeb-a5293584e4ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:45:21.041101 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.040985    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d547ad4b-a726-422e-aaeb-a5293584e4ac" (UID: "d547ad4b-a726-422e-aaeb-a5293584e4ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:21.041101 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.041004    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d547ad4b-a726-422e-aaeb-a5293584e4ac-kube-api-access-759fj" (OuterVolumeSpecName: "kube-api-access-759fj") pod "d547ad4b-a726-422e-aaeb-a5293584e4ac" (UID: "d547ad4b-a726-422e-aaeb-a5293584e4ac"). InnerVolumeSpecName "kube-api-access-759fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:45:21.041101 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.041042    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d547ad4b-a726-422e-aaeb-a5293584e4ac" (UID: "d547ad4b-a726-422e-aaeb-a5293584e4ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:45:21.140148 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.140105    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-workload-socket\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:21.140148 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.140143    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d547ad4b-a726-422e-aaeb-a5293584e4ac-istiod-ca-cert\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:21.140148 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.140156    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-759fj\" (UniqueName: \"kubernetes.io/projected/d547ad4b-a726-422e-aaeb-a5293584e4ac-kube-api-access-759fj\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:21.140404 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.140171    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-envoy\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:21.140404 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.140182    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-data\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:21.140404 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.140193    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-token\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:21.140404 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.140202    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d547ad4b-a726-422e-aaeb-a5293584e4ac-istio-podinfo\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:21.214627 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.214596    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"]
Sep 10 15:45:21.217968 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:21.217942    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-62d2f7-openshift-default-658fbfcc9d-pbxd7"]
Sep 10 15:45:22.126448 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:22.126405    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" path="/var/lib/kubelet/pods/d547ad4b-a726-422e-aaeb-a5293584e4ac/volumes"
Sep 10 15:45:26.503207 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.501430    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"]
Sep 10 15:45:26.503207 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.501771    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm" podUID="37a07ee7-ef8b-46b2-9c4a-273bc91a63fb" containerName="maas-api" containerID="cri-o://e7011929cac828bab4745c249768efaf83e4448d376e781d6a338db3b6bb7dc6" gracePeriod=30
Sep 10 15:45:26.755420 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.755359    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:45:26.790176 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.790141    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-maas-api-tls\") pod \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") "
Sep 10 15:45:26.790541 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.790229    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-metrics-certs\") pod \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") "
Sep 10 15:45:26.790541 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.790263    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-openshift-service-ca\") pod \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") "
Sep 10 15:45:26.790541 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.790322    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76s7p\" (UniqueName: \"kubernetes.io/projected/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-kube-api-access-76s7p\") pod \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\" (UID: \"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb\") "
Sep 10 15:45:26.791459 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.791407    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "37a07ee7-ef8b-46b2-9c4a-273bc91a63fb" (UID: "37a07ee7-ef8b-46b2-9c4a-273bc91a63fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:45:26.794933 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.794889    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "37a07ee7-ef8b-46b2-9c4a-273bc91a63fb" (UID: "37a07ee7-ef8b-46b2-9c4a-273bc91a63fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:45:26.795174 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.795137    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "37a07ee7-ef8b-46b2-9c4a-273bc91a63fb" (UID: "37a07ee7-ef8b-46b2-9c4a-273bc91a63fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:45:26.795302 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.795174    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-kube-api-access-76s7p" (OuterVolumeSpecName: "kube-api-access-76s7p") pod "37a07ee7-ef8b-46b2-9c4a-273bc91a63fb" (UID: "37a07ee7-ef8b-46b2-9c4a-273bc91a63fb"). InnerVolumeSpecName "kube-api-access-76s7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:45:26.891756 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.891717    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76s7p\" (UniqueName: \"kubernetes.io/projected/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-kube-api-access-76s7p\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:26.891756 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.891752    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:26.891756 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.891761    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:26.892033 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.891769    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:26.924005 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.923970    2573 generic.go:356] "Generic (PLEG): container finished" podID="37a07ee7-ef8b-46b2-9c4a-273bc91a63fb" containerID="e7011929cac828bab4745c249768efaf83e4448d376e781d6a338db3b6bb7dc6" exitCode=0
Sep 10 15:45:26.924163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.924012    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"
Sep 10 15:45:26.924163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.924053    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm" event={"ID":"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb","Type":"ContainerDied","Data":"e7011929cac828bab4745c249768efaf83e4448d376e781d6a338db3b6bb7dc6"}
Sep 10 15:45:26.924163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.924114    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm" event={"ID":"37a07ee7-ef8b-46b2-9c4a-273bc91a63fb","Type":"ContainerDied","Data":"96ae830847327f56e432384edefb7c7a47a36f40951e1a0716f810d823fb74ae"}
Sep 10 15:45:26.924163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.924142    2573 scope.go:117] "RemoveContainer" containerID="e7011929cac828bab4745c249768efaf83e4448d376e781d6a338db3b6bb7dc6"
Sep 10 15:45:26.934126 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.934110    2573 scope.go:117] "RemoveContainer" containerID="e7011929cac828bab4745c249768efaf83e4448d376e781d6a338db3b6bb7dc6"
Sep 10 15:45:26.934411 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:45:26.934388    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7011929cac828bab4745c249768efaf83e4448d376e781d6a338db3b6bb7dc6\": container with ID starting with e7011929cac828bab4745c249768efaf83e4448d376e781d6a338db3b6bb7dc6 not found: ID does not exist" containerID="e7011929cac828bab4745c249768efaf83e4448d376e781d6a338db3b6bb7dc6"
Sep 10 15:45:26.934473 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.934424    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7011929cac828bab4745c249768efaf83e4448d376e781d6a338db3b6bb7dc6"} err="failed to get container status \"e7011929cac828bab4745c249768efaf83e4448d376e781d6a338db3b6bb7dc6\": rpc error: code = NotFound desc = could not find container \"e7011929cac828bab4745c249768efaf83e4448d376e781d6a338db3b6bb7dc6\": container with ID starting with e7011929cac828bab4745c249768efaf83e4448d376e781d6a338db3b6bb7dc6 not found: ID does not exist"
Sep 10 15:45:26.943359 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.943334    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"]
Sep 10 15:45:26.946939 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:26.946917    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fc3e54-59999556b7-57cmm"]
Sep 10 15:45:28.125465 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:28.125430    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37a07ee7-ef8b-46b2-9c4a-273bc91a63fb" path="/var/lib/kubelet/pods/37a07ee7-ef8b-46b2-9c4a-273bc91a63fb/volumes"
Sep 10 15:45:35.740336 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:35.740246    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57cc444cff-sjtz6"]
Sep 10 15:45:35.740784 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:35.740516    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57cc444cff-sjtz6" podUID="8de7ca77-478b-4bb1-a931-1382f4dd2796" containerName="manager" containerID="cri-o://b5c09fb824211392adb34211dd99b7baae22e34942e05407a8d07957787575a5" gracePeriod=30
Sep 10 15:45:35.968766 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:35.968728    2573 generic.go:356] "Generic (PLEG): container finished" podID="8de7ca77-478b-4bb1-a931-1382f4dd2796" containerID="b5c09fb824211392adb34211dd99b7baae22e34942e05407a8d07957787575a5" exitCode=0
Sep 10 15:45:35.968951 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:35.968834    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57cc444cff-sjtz6" event={"ID":"8de7ca77-478b-4bb1-a931-1382f4dd2796","Type":"ContainerDied","Data":"b5c09fb824211392adb34211dd99b7baae22e34942e05407a8d07957787575a5"}
Sep 10 15:45:35.993995 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:35.993930    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:45:36.066830 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.066776    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd5bm\" (UniqueName: \"kubernetes.io/projected/8de7ca77-478b-4bb1-a931-1382f4dd2796-kube-api-access-bd5bm\") pod \"8de7ca77-478b-4bb1-a931-1382f4dd2796\" (UID: \"8de7ca77-478b-4bb1-a931-1382f4dd2796\") "
Sep 10 15:45:36.067000 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.066877    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-webhook-cert\") pod \"8de7ca77-478b-4bb1-a931-1382f4dd2796\" (UID: \"8de7ca77-478b-4bb1-a931-1382f4dd2796\") "
Sep 10 15:45:36.067000 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.066910    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-metrics-certs\") pod \"8de7ca77-478b-4bb1-a931-1382f4dd2796\" (UID: \"8de7ca77-478b-4bb1-a931-1382f4dd2796\") "
Sep 10 15:45:36.068997 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.068963    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de7ca77-478b-4bb1-a931-1382f4dd2796-kube-api-access-bd5bm" (OuterVolumeSpecName: "kube-api-access-bd5bm") pod "8de7ca77-478b-4bb1-a931-1382f4dd2796" (UID: "8de7ca77-478b-4bb1-a931-1382f4dd2796"). InnerVolumeSpecName "kube-api-access-bd5bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:45:36.069124 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.069028    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8de7ca77-478b-4bb1-a931-1382f4dd2796" (UID: "8de7ca77-478b-4bb1-a931-1382f4dd2796"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:45:36.069124 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.069067    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8de7ca77-478b-4bb1-a931-1382f4dd2796" (UID: "8de7ca77-478b-4bb1-a931-1382f4dd2796"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:45:36.168507 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.168472    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd5bm\" (UniqueName: \"kubernetes.io/projected/8de7ca77-478b-4bb1-a931-1382f4dd2796-kube-api-access-bd5bm\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:36.168507 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.168507    2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-webhook-cert\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:36.168681 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.168522    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de7ca77-478b-4bb1-a931-1382f4dd2796-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:45:36.975145 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.975112    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57cc444cff-sjtz6"
Sep 10 15:45:36.975145 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.975138    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57cc444cff-sjtz6" event={"ID":"8de7ca77-478b-4bb1-a931-1382f4dd2796","Type":"ContainerDied","Data":"f75929116d6e1d2b5c20e29e8a738e5f6b1099877ec4ad750b850a39c7bcf5e3"}
Sep 10 15:45:36.975895 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.975214    2573 scope.go:117] "RemoveContainer" containerID="b5c09fb824211392adb34211dd99b7baae22e34942e05407a8d07957787575a5"
Sep 10 15:45:36.991413 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.991378    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57cc444cff-sjtz6"]
Sep 10 15:45:36.994161 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:36.994139    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57cc444cff-sjtz6"]
Sep 10 15:45:38.126296 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:45:38.126260    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8de7ca77-478b-4bb1-a931-1382f4dd2796" path="/var/lib/kubelet/pods/8de7ca77-478b-4bb1-a931-1382f4dd2796/volumes"
Sep 10 15:46:37.769108 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769068    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57cc444cff-q8bqv"]
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769409    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd90bcf-76fa-42c2-bd3a-8a1834d05d00" containerName="maas-api"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769421    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd90bcf-76fa-42c2-bd3a-8a1834d05d00" containerName="maas-api"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769429    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3506c9f-c807-4311-8520-6e9f9bb930d0" containerName="maas-api"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769435    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3506c9f-c807-4311-8520-6e9f9bb930d0" containerName="maas-api"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769444    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerName="istio-proxy"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769449    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerName="istio-proxy"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769455    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d79d08-e4f4-4cfd-b605-b2349ef1615e" containerName="payload-processing"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769460    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d79d08-e4f4-4cfd-b605-b2349ef1615e" containerName="payload-processing"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769474    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63af54d3-85f5-480c-b792-60619b27081e" containerName="payload-pre-processing"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769479    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="63af54d3-85f5-480c-b792-60619b27081e" containerName="payload-pre-processing"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769487    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerName="istio-proxy"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769492    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerName="istio-proxy"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769499    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de7ca77-478b-4bb1-a931-1382f4dd2796" containerName="manager"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769503    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de7ca77-478b-4bb1-a931-1382f4dd2796" containerName="manager"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769509    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37a07ee7-ef8b-46b2-9c4a-273bc91a63fb" containerName="maas-api"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769514    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37a07ee7-ef8b-46b2-9c4a-273bc91a63fb" containerName="maas-api"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769520    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c98154-61fb-4c9b-98f9-bbf1f41c9ed1" containerName="main"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769525    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c98154-61fb-4c9b-98f9-bbf1f41c9ed1" containerName="main"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769532    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c53f4421-3ed3-4cad-b985-11184c2447fb" containerName="payload-processing"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769537    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c53f4421-3ed3-4cad-b985-11184c2447fb" containerName="payload-processing"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769587    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="37a07ee7-ef8b-46b2-9c4a-273bc91a63fb" containerName="maas-api"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769592    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c53f4421-3ed3-4cad-b985-11184c2447fb" containerName="payload-processing"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769600    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerName="istio-proxy"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769606    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d547ad4b-a726-422e-aaeb-a5293584e4ac" containerName="istio-proxy"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769612    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3506c9f-c807-4311-8520-6e9f9bb930d0" containerName="maas-api"
Sep 10 15:46:37.769603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769617    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fd90bcf-76fa-42c2-bd3a-8a1834d05d00" containerName="maas-api"
Sep 10 15:46:37.770723 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769623    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c98154-61fb-4c9b-98f9-bbf1f41c9ed1" containerName="main"
Sep 10 15:46:37.770723 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769628    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="63af54d3-85f5-480c-b792-60619b27081e" containerName="payload-pre-processing"
Sep 10 15:46:37.770723 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769633    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="14d79d08-e4f4-4cfd-b605-b2349ef1615e" containerName="payload-processing"
Sep 10 15:46:37.770723 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.769638    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de7ca77-478b-4bb1-a931-1382f4dd2796" containerName="manager"
Sep 10 15:46:37.772673 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.772655    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:37.775458 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.775436    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 15:46:37.775596 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.775436    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 15:46:37.775596 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.775436    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 15:46:37.775596 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.775437    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gqnvd\""
Sep 10 15:46:37.782503 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.782480    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57cc444cff-q8bqv"]
Sep 10 15:46:37.864539 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.864494    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78v8p\" (UniqueName: \"kubernetes.io/projected/34ac33fa-98ec-40c2-b4ce-50229281052a-kube-api-access-78v8p\") pod \"maas-controller-57cc444cff-q8bqv\" (UID: \"34ac33fa-98ec-40c2-b4ce-50229281052a\") " pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:37.864539 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.864539    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34ac33fa-98ec-40c2-b4ce-50229281052a-webhook-cert\") pod \"maas-controller-57cc444cff-q8bqv\" (UID: \"34ac33fa-98ec-40c2-b4ce-50229281052a\") " pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:37.864784 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.864566    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ac33fa-98ec-40c2-b4ce-50229281052a-metrics-certs\") pod \"maas-controller-57cc444cff-q8bqv\" (UID: \"34ac33fa-98ec-40c2-b4ce-50229281052a\") " pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:37.965936 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.965880    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78v8p\" (UniqueName: \"kubernetes.io/projected/34ac33fa-98ec-40c2-b4ce-50229281052a-kube-api-access-78v8p\") pod \"maas-controller-57cc444cff-q8bqv\" (UID: \"34ac33fa-98ec-40c2-b4ce-50229281052a\") " pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:37.965936 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.965938    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34ac33fa-98ec-40c2-b4ce-50229281052a-webhook-cert\") pod \"maas-controller-57cc444cff-q8bqv\" (UID: \"34ac33fa-98ec-40c2-b4ce-50229281052a\") " pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:37.966169 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.965977    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ac33fa-98ec-40c2-b4ce-50229281052a-metrics-certs\") pod \"maas-controller-57cc444cff-q8bqv\" (UID: \"34ac33fa-98ec-40c2-b4ce-50229281052a\") " pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:37.968859 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.968831    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ac33fa-98ec-40c2-b4ce-50229281052a-metrics-certs\") pod \"maas-controller-57cc444cff-q8bqv\" (UID: \"34ac33fa-98ec-40c2-b4ce-50229281052a\") " pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:37.968996 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.968831    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34ac33fa-98ec-40c2-b4ce-50229281052a-webhook-cert\") pod \"maas-controller-57cc444cff-q8bqv\" (UID: \"34ac33fa-98ec-40c2-b4ce-50229281052a\") " pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:37.972916 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:37.972889    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78v8p\" (UniqueName: \"kubernetes.io/projected/34ac33fa-98ec-40c2-b4ce-50229281052a-kube-api-access-78v8p\") pod \"maas-controller-57cc444cff-q8bqv\" (UID: \"34ac33fa-98ec-40c2-b4ce-50229281052a\") " pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:38.085927 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:38.085893    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gqnvd\""
Sep 10 15:46:38.094516 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:38.094493    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:38.369230 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:38.368870    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57cc444cff-q8bqv"]
Sep 10 15:46:38.373663 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:46:38.373635    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ac33fa_98ec_40c2_b4ce_50229281052a.slice/crio-7599a74a84dab5904f68997db4d2fb9af3da7dce7a7c779651268a697c3daca8 WatchSource:0}: Error finding container 7599a74a84dab5904f68997db4d2fb9af3da7dce7a7c779651268a697c3daca8: Status 404 returned error can't find the container with id 7599a74a84dab5904f68997db4d2fb9af3da7dce7a7c779651268a697c3daca8
Sep 10 15:46:38.375100 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:38.375078    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:46:38.955727 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:38.955703    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 15:46:39.255696 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:39.255605    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57cc444cff-q8bqv" event={"ID":"34ac33fa-98ec-40c2-b4ce-50229281052a","Type":"ContainerStarted","Data":"841ddaba6e071d197e877a20a140feafb0f49a4f7468e3833cb77be82532cea0"}
Sep 10 15:46:39.255696 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:39.255664    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57cc444cff-q8bqv" event={"ID":"34ac33fa-98ec-40c2-b4ce-50229281052a","Type":"ContainerStarted","Data":"7599a74a84dab5904f68997db4d2fb9af3da7dce7a7c779651268a697c3daca8"}
Sep 10 15:46:39.255696 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:39.255689    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:39.272395 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:39.272339    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57cc444cff-q8bqv" podStartSLOduration=1.69413821 podStartE2EDuration="2.272322349s" totalImagesPullingTime="578.184139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:39.271193777 +0000 UTC m=+1861.686989598" watchObservedRunningTime="2026-09-10 15:46:39.272322349 +0000 UTC m=+1861.688118204"
Sep 10 15:46:50.262627 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:50.262585    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:59.546583 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:59.546499    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57cc444cff-q8bqv"]
Sep 10 15:46:59.547068 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:59.546718    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57cc444cff-q8bqv" podUID="34ac33fa-98ec-40c2-b4ce-50229281052a" containerName="manager" containerID="cri-o://841ddaba6e071d197e877a20a140feafb0f49a4f7468e3833cb77be82532cea0" gracePeriod=30
Sep 10 15:46:59.790550 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:59.790522    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:46:59.853867 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:59.853837    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ac33fa-98ec-40c2-b4ce-50229281052a-metrics-certs\") pod \"34ac33fa-98ec-40c2-b4ce-50229281052a\" (UID: \"34ac33fa-98ec-40c2-b4ce-50229281052a\") "
Sep 10 15:46:59.854053 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:59.853892    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34ac33fa-98ec-40c2-b4ce-50229281052a-webhook-cert\") pod \"34ac33fa-98ec-40c2-b4ce-50229281052a\" (UID: \"34ac33fa-98ec-40c2-b4ce-50229281052a\") "
Sep 10 15:46:59.854053 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:59.854000    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78v8p\" (UniqueName: \"kubernetes.io/projected/34ac33fa-98ec-40c2-b4ce-50229281052a-kube-api-access-78v8p\") pod \"34ac33fa-98ec-40c2-b4ce-50229281052a\" (UID: \"34ac33fa-98ec-40c2-b4ce-50229281052a\") "
Sep 10 15:46:59.856126 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:59.856090    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ac33fa-98ec-40c2-b4ce-50229281052a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "34ac33fa-98ec-40c2-b4ce-50229281052a" (UID: "34ac33fa-98ec-40c2-b4ce-50229281052a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:46:59.856126 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:59.856120    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ac33fa-98ec-40c2-b4ce-50229281052a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "34ac33fa-98ec-40c2-b4ce-50229281052a" (UID: "34ac33fa-98ec-40c2-b4ce-50229281052a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:46:59.856299 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:59.856176    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ac33fa-98ec-40c2-b4ce-50229281052a-kube-api-access-78v8p" (OuterVolumeSpecName: "kube-api-access-78v8p") pod "34ac33fa-98ec-40c2-b4ce-50229281052a" (UID: "34ac33fa-98ec-40c2-b4ce-50229281052a"). InnerVolumeSpecName "kube-api-access-78v8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:46:59.955245 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:59.955207    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34ac33fa-98ec-40c2-b4ce-50229281052a-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:46:59.955245 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:59.955238    2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34ac33fa-98ec-40c2-b4ce-50229281052a-webhook-cert\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:46:59.955245 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:46:59.955248    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78v8p\" (UniqueName: \"kubernetes.io/projected/34ac33fa-98ec-40c2-b4ce-50229281052a-kube-api-access-78v8p\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:47:00.350751 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:00.350714    2573 generic.go:356] "Generic (PLEG): container finished" podID="34ac33fa-98ec-40c2-b4ce-50229281052a" containerID="841ddaba6e071d197e877a20a140feafb0f49a4f7468e3833cb77be82532cea0" exitCode=0
Sep 10 15:47:00.350959 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:00.350765    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57cc444cff-q8bqv"
Sep 10 15:47:00.350959 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:00.350826    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57cc444cff-q8bqv" event={"ID":"34ac33fa-98ec-40c2-b4ce-50229281052a","Type":"ContainerDied","Data":"841ddaba6e071d197e877a20a140feafb0f49a4f7468e3833cb77be82532cea0"}
Sep 10 15:47:00.350959 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:00.350886    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57cc444cff-q8bqv" event={"ID":"34ac33fa-98ec-40c2-b4ce-50229281052a","Type":"ContainerDied","Data":"7599a74a84dab5904f68997db4d2fb9af3da7dce7a7c779651268a697c3daca8"}
Sep 10 15:47:00.350959 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:00.350915    2573 scope.go:117] "RemoveContainer" containerID="841ddaba6e071d197e877a20a140feafb0f49a4f7468e3833cb77be82532cea0"
Sep 10 15:47:00.359531 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:00.359512    2573 scope.go:117] "RemoveContainer" containerID="841ddaba6e071d197e877a20a140feafb0f49a4f7468e3833cb77be82532cea0"
Sep 10 15:47:00.359871 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:47:00.359847    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841ddaba6e071d197e877a20a140feafb0f49a4f7468e3833cb77be82532cea0\": container with ID starting with 841ddaba6e071d197e877a20a140feafb0f49a4f7468e3833cb77be82532cea0 not found: ID does not exist" containerID="841ddaba6e071d197e877a20a140feafb0f49a4f7468e3833cb77be82532cea0"
Sep 10 15:47:00.359928 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:00.359884    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841ddaba6e071d197e877a20a140feafb0f49a4f7468e3833cb77be82532cea0"} err="failed to get container status \"841ddaba6e071d197e877a20a140feafb0f49a4f7468e3833cb77be82532cea0\": rpc error: code = NotFound desc = could not find container \"841ddaba6e071d197e877a20a140feafb0f49a4f7468e3833cb77be82532cea0\": container with ID starting with 841ddaba6e071d197e877a20a140feafb0f49a4f7468e3833cb77be82532cea0 not found: ID does not exist"
Sep 10 15:47:00.364445 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:00.364421    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57cc444cff-q8bqv"]
Sep 10 15:47:00.371161 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:00.371131    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57cc444cff-q8bqv"]
Sep 10 15:47:01.109190 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.109143    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57cc444cff-xmhrj"]
Sep 10 15:47:01.109687 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.109597    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ac33fa-98ec-40c2-b4ce-50229281052a" containerName="manager"
Sep 10 15:47:01.109687 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.109617    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ac33fa-98ec-40c2-b4ce-50229281052a" containerName="manager"
Sep 10 15:47:01.109854 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.109724    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="34ac33fa-98ec-40c2-b4ce-50229281052a" containerName="manager"
Sep 10 15:47:01.113990 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.113968    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57cc444cff-xmhrj"
Sep 10 15:47:01.116896 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.116875    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 15:47:01.117022 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.116935    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gqnvd\""
Sep 10 15:47:01.117022 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.116956    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 15:47:01.117022 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.116956    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 15:47:01.120734 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.120713    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57cc444cff-xmhrj"]
Sep 10 15:47:01.266226 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.266179    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92f0ec48-b3a9-4803-b6ac-ca11501b055e-webhook-cert\") pod \"maas-controller-57cc444cff-xmhrj\" (UID: \"92f0ec48-b3a9-4803-b6ac-ca11501b055e\") " pod="opendatahub/maas-controller-57cc444cff-xmhrj"
Sep 10 15:47:01.266418 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.266320    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92f0ec48-b3a9-4803-b6ac-ca11501b055e-metrics-certs\") pod \"maas-controller-57cc444cff-xmhrj\" (UID: \"92f0ec48-b3a9-4803-b6ac-ca11501b055e\") " pod="opendatahub/maas-controller-57cc444cff-xmhrj"
Sep 10 15:47:01.266418 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.266348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-782np\" (UniqueName: \"kubernetes.io/projected/92f0ec48-b3a9-4803-b6ac-ca11501b055e-kube-api-access-782np\") pod \"maas-controller-57cc444cff-xmhrj\" (UID: \"92f0ec48-b3a9-4803-b6ac-ca11501b055e\") " pod="opendatahub/maas-controller-57cc444cff-xmhrj"
Sep 10 15:47:01.367349 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.367256    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92f0ec48-b3a9-4803-b6ac-ca11501b055e-metrics-certs\") pod \"maas-controller-57cc444cff-xmhrj\" (UID: \"92f0ec48-b3a9-4803-b6ac-ca11501b055e\") " pod="opendatahub/maas-controller-57cc444cff-xmhrj"
Sep 10 15:47:01.367349 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.367294    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-782np\" (UniqueName: \"kubernetes.io/projected/92f0ec48-b3a9-4803-b6ac-ca11501b055e-kube-api-access-782np\") pod \"maas-controller-57cc444cff-xmhrj\" (UID: \"92f0ec48-b3a9-4803-b6ac-ca11501b055e\") " pod="opendatahub/maas-controller-57cc444cff-xmhrj"
Sep 10 15:47:01.367349 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.367322    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92f0ec48-b3a9-4803-b6ac-ca11501b055e-webhook-cert\") pod \"maas-controller-57cc444cff-xmhrj\" (UID: \"92f0ec48-b3a9-4803-b6ac-ca11501b055e\") " pod="opendatahub/maas-controller-57cc444cff-xmhrj"
Sep 10 15:47:01.370222 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.370187    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92f0ec48-b3a9-4803-b6ac-ca11501b055e-webhook-cert\") pod \"maas-controller-57cc444cff-xmhrj\" (UID: \"92f0ec48-b3a9-4803-b6ac-ca11501b055e\") " pod="opendatahub/maas-controller-57cc444cff-xmhrj"
Sep 10 15:47:01.370377 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.370192    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92f0ec48-b3a9-4803-b6ac-ca11501b055e-metrics-certs\") pod \"maas-controller-57cc444cff-xmhrj\" (UID: \"92f0ec48-b3a9-4803-b6ac-ca11501b055e\") " pod="opendatahub/maas-controller-57cc444cff-xmhrj"
Sep 10 15:47:01.378176 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.378139    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-782np\" (UniqueName: \"kubernetes.io/projected/92f0ec48-b3a9-4803-b6ac-ca11501b055e-kube-api-access-782np\") pod \"maas-controller-57cc444cff-xmhrj\" (UID: \"92f0ec48-b3a9-4803-b6ac-ca11501b055e\") " pod="opendatahub/maas-controller-57cc444cff-xmhrj"
Sep 10 15:47:01.426149 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.426119    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57cc444cff-xmhrj"
Sep 10 15:47:01.699995 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:01.699909    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57cc444cff-xmhrj"]
Sep 10 15:47:01.702844 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:47:01.702815    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f0ec48_b3a9_4803_b6ac_ca11501b055e.slice/crio-24858e827f3550a03246fa96453d6af193da03973c85caf55a0ae927bfe9c0b6 WatchSource:0}: Error finding container 24858e827f3550a03246fa96453d6af193da03973c85caf55a0ae927bfe9c0b6: Status 404 returned error can't find the container with id 24858e827f3550a03246fa96453d6af193da03973c85caf55a0ae927bfe9c0b6
Sep 10 15:47:02.129929 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:02.127176    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34ac33fa-98ec-40c2-b4ce-50229281052a" path="/var/lib/kubelet/pods/34ac33fa-98ec-40c2-b4ce-50229281052a/volumes"
Sep 10 15:47:02.363084 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:02.362979    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57cc444cff-xmhrj" event={"ID":"92f0ec48-b3a9-4803-b6ac-ca11501b055e","Type":"ContainerStarted","Data":"94b5e124cd2fc08d7f6286c7d7c2fd98f075e65c56592169e8cf381e1274efc5"}
Sep 10 15:47:02.363084 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:02.363053    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57cc444cff-xmhrj"
Sep 10 15:47:02.363084 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:02.363079    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57cc444cff-xmhrj" event={"ID":"92f0ec48-b3a9-4803-b6ac-ca11501b055e","Type":"ContainerStarted","Data":"24858e827f3550a03246fa96453d6af193da03973c85caf55a0ae927bfe9c0b6"}
Sep 10 15:47:02.379512 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:02.379457    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57cc444cff-xmhrj" podStartSLOduration=0.995583225 podStartE2EDuration="1.379442674s" totalImagesPullingTime="383.859449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:02.37798831 +0000 UTC m=+1884.793784128" watchObservedRunningTime="2026-09-10 15:47:02.379442674 +0000 UTC m=+1884.795238490"
Sep 10 15:47:13.370657 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:47:13.370625    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57cc444cff-xmhrj"
Sep 10 15:48:18.365844 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.365776    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"]
Sep 10 15:48:18.366419 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.366035    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm" podUID="661edcdb-1c8f-42d6-a05a-fcf274343e9e" containerName="manager" containerID="cri-o://4e982c643f405ac78a3eaf136639878d2949884f22dfe4679c672b6a1f142e93" gracePeriod=10
Sep 10 15:48:18.610344 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.610317    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"
Sep 10 15:48:18.645837 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.645742    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zssnv\" (UniqueName: \"kubernetes.io/projected/661edcdb-1c8f-42d6-a05a-fcf274343e9e-kube-api-access-zssnv\") pod \"661edcdb-1c8f-42d6-a05a-fcf274343e9e\" (UID: \"661edcdb-1c8f-42d6-a05a-fcf274343e9e\") "
Sep 10 15:48:18.645987 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.645920    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/661edcdb-1c8f-42d6-a05a-fcf274343e9e-extensions-socket-volume\") pod \"661edcdb-1c8f-42d6-a05a-fcf274343e9e\" (UID: \"661edcdb-1c8f-42d6-a05a-fcf274343e9e\") "
Sep 10 15:48:18.646288 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.646263    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661edcdb-1c8f-42d6-a05a-fcf274343e9e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "661edcdb-1c8f-42d6-a05a-fcf274343e9e" (UID: "661edcdb-1c8f-42d6-a05a-fcf274343e9e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:48:18.648175 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.648150    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661edcdb-1c8f-42d6-a05a-fcf274343e9e-kube-api-access-zssnv" (OuterVolumeSpecName: "kube-api-access-zssnv") pod "661edcdb-1c8f-42d6-a05a-fcf274343e9e" (UID: "661edcdb-1c8f-42d6-a05a-fcf274343e9e"). InnerVolumeSpecName "kube-api-access-zssnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:48:18.709728 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.709691    2573 generic.go:356] "Generic (PLEG): container finished" podID="661edcdb-1c8f-42d6-a05a-fcf274343e9e" containerID="4e982c643f405ac78a3eaf136639878d2949884f22dfe4679c672b6a1f142e93" exitCode=0
Sep 10 15:48:18.709953 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.709744    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"
Sep 10 15:48:18.709953 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.709746    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm" event={"ID":"661edcdb-1c8f-42d6-a05a-fcf274343e9e","Type":"ContainerDied","Data":"4e982c643f405ac78a3eaf136639878d2949884f22dfe4679c672b6a1f142e93"}
Sep 10 15:48:18.709953 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.709889    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm" event={"ID":"661edcdb-1c8f-42d6-a05a-fcf274343e9e","Type":"ContainerDied","Data":"131f1399da7f8d49bb3c58e7e2a9f96dfe264a7c5ccd5b6af2c82a1b52f70587"}
Sep 10 15:48:18.709953 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.709920    2573 scope.go:117] "RemoveContainer" containerID="4e982c643f405ac78a3eaf136639878d2949884f22dfe4679c672b6a1f142e93"
Sep 10 15:48:18.719261 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.719244    2573 scope.go:117] "RemoveContainer" containerID="4e982c643f405ac78a3eaf136639878d2949884f22dfe4679c672b6a1f142e93"
Sep 10 15:48:18.719550 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:48:18.719523    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e982c643f405ac78a3eaf136639878d2949884f22dfe4679c672b6a1f142e93\": container with ID starting with 4e982c643f405ac78a3eaf136639878d2949884f22dfe4679c672b6a1f142e93 not found: ID does not exist" containerID="4e982c643f405ac78a3eaf136639878d2949884f22dfe4679c672b6a1f142e93"
Sep 10 15:48:18.719649 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.719556    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e982c643f405ac78a3eaf136639878d2949884f22dfe4679c672b6a1f142e93"} err="failed to get container status \"4e982c643f405ac78a3eaf136639878d2949884f22dfe4679c672b6a1f142e93\": rpc error: code = NotFound desc = could not find container \"4e982c643f405ac78a3eaf136639878d2949884f22dfe4679c672b6a1f142e93\": container with ID starting with 4e982c643f405ac78a3eaf136639878d2949884f22dfe4679c672b6a1f142e93 not found: ID does not exist"
Sep 10 15:48:18.729657 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.729634    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"]
Sep 10 15:48:18.734669 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.734644    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bqdfm"]
Sep 10 15:48:18.746889 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.746860    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/661edcdb-1c8f-42d6-a05a-fcf274343e9e-extensions-socket-volume\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:48:18.746889 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:18.746885    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zssnv\" (UniqueName: \"kubernetes.io/projected/661edcdb-1c8f-42d6-a05a-fcf274343e9e-kube-api-access-zssnv\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:48:20.125474 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:48:20.125439    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="661edcdb-1c8f-42d6-a05a-fcf274343e9e" path="/var/lib/kubelet/pods/661edcdb-1c8f-42d6-a05a-fcf274343e9e/volumes"
Sep 10 15:49:19.025306 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.025271    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47"]
Sep 10 15:49:19.025826 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.025598    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661edcdb-1c8f-42d6-a05a-fcf274343e9e" containerName="manager"
Sep 10 15:49:19.025826 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.025608    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="661edcdb-1c8f-42d6-a05a-fcf274343e9e" containerName="manager"
Sep 10 15:49:19.025826 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.025662    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="661edcdb-1c8f-42d6-a05a-fcf274343e9e" containerName="manager"
Sep 10 15:49:19.028295 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.028278    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47"
Sep 10 15:49:19.030462 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.030438    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-g5nj9\""
Sep 10 15:49:19.030978 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.030962    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd0b3bce-8651-40c9-a4c8-1b1cefde3878-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2ft47\" (UID: \"dd0b3bce-8651-40c9-a4c8-1b1cefde3878\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47"
Sep 10 15:49:19.031057 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.030992    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcqb2\" (UniqueName: \"kubernetes.io/projected/dd0b3bce-8651-40c9-a4c8-1b1cefde3878-kube-api-access-wcqb2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2ft47\" (UID: \"dd0b3bce-8651-40c9-a4c8-1b1cefde3878\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47"
Sep 10 15:49:19.040939 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.040903    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47"]
Sep 10 15:49:19.132207 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.132169    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd0b3bce-8651-40c9-a4c8-1b1cefde3878-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2ft47\" (UID: \"dd0b3bce-8651-40c9-a4c8-1b1cefde3878\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47"
Sep 10 15:49:19.132391 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.132219    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcqb2\" (UniqueName: \"kubernetes.io/projected/dd0b3bce-8651-40c9-a4c8-1b1cefde3878-kube-api-access-wcqb2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2ft47\" (UID: \"dd0b3bce-8651-40c9-a4c8-1b1cefde3878\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47"
Sep 10 15:49:19.132621 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.132595    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd0b3bce-8651-40c9-a4c8-1b1cefde3878-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2ft47\" (UID: \"dd0b3bce-8651-40c9-a4c8-1b1cefde3878\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47"
Sep 10 15:49:19.139796 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.139772    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcqb2\" (UniqueName: \"kubernetes.io/projected/dd0b3bce-8651-40c9-a4c8-1b1cefde3878-kube-api-access-wcqb2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2ft47\" (UID: \"dd0b3bce-8651-40c9-a4c8-1b1cefde3878\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47"
Sep 10 15:49:19.345308 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.345272    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47"
Sep 10 15:49:19.620569 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.620539    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47"]
Sep 10 15:49:19.623835 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:49:19.623806    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0b3bce_8651_40c9_a4c8_1b1cefde3878.slice/crio-5cdecc9b1c7d6870d56e84a862e62176f1ae605d756f191e54cb7df5b18b4209 WatchSource:0}: Error finding container 5cdecc9b1c7d6870d56e84a862e62176f1ae605d756f191e54cb7df5b18b4209: Status 404 returned error can't find the container with id 5cdecc9b1c7d6870d56e84a862e62176f1ae605d756f191e54cb7df5b18b4209
Sep 10 15:49:19.991749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.991665    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47" event={"ID":"dd0b3bce-8651-40c9-a4c8-1b1cefde3878","Type":"ContainerStarted","Data":"6639e196911d76ef63546cd4a115f64996e2214901661e43c5c1de1598917ebb"}
Sep 10 15:49:19.991749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.991726    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47" event={"ID":"dd0b3bce-8651-40c9-a4c8-1b1cefde3878","Type":"ContainerStarted","Data":"5cdecc9b1c7d6870d56e84a862e62176f1ae605d756f191e54cb7df5b18b4209"}
Sep 10 15:49:19.991749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:19.991752    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47"
Sep 10 15:49:20.011266 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:20.011206    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47" podStartSLOduration=1.011190007 podStartE2EDuration="1.011190007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:20.009654424 +0000 UTC m=+2022.425450242" watchObservedRunningTime="2026-09-10 15:49:20.011190007 +0000 UTC m=+2022.426985826"
Sep 10 15:49:30.999229 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:49:30.999198    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2ft47"
Sep 10 15:50:04.127221 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.127143    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"]
Sep 10 15:50:04.130507 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.130486    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.132419 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.132396    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d03fd470-kserve-self-signed-certs\""
Sep 10 15:50:04.135195 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.135175    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"]
Sep 10 15:50:04.212078 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.212040    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-dshm\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.212249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.212104    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6679105c-a4ef-4b97-8c50-142f3ce16e1c-tls-certs\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.212249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.212212    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-model-cache\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.212378 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.212265    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9d7b\" (UniqueName: \"kubernetes.io/projected/6679105c-a4ef-4b97-8c50-142f3ce16e1c-kube-api-access-s9d7b\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.212378 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.212307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-home\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.212378 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.212333    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-tmp-dir\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.313121 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.313081    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-dshm\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.313337 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.313144    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6679105c-a4ef-4b97-8c50-142f3ce16e1c-tls-certs\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.313337 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.313176    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-model-cache\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.313337 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.313208    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9d7b\" (UniqueName: \"kubernetes.io/projected/6679105c-a4ef-4b97-8c50-142f3ce16e1c-kube-api-access-s9d7b\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.313337 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.313232    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-home\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.313558 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.313353    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-tmp-dir\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.313643 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.313619    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-model-cache\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.313702 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.313648    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-tmp-dir\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.313702 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.313685    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-home\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.315605 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.315579    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-dshm\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.315868 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.315849    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6679105c-a4ef-4b97-8c50-142f3ce16e1c-tls-certs\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.320733 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.320707    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9d7b\" (UniqueName: \"kubernetes.io/projected/6679105c-a4ef-4b97-8c50-142f3ce16e1c-kube-api-access-s9d7b\") pod \"e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") " pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.442152 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.442063    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:04.718339 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:04.718261    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"]
Sep 10 15:50:04.721693 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:50:04.721666    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6679105c_a4ef_4b97_8c50_142f3ce16e1c.slice/crio-3d02b515cb3560dc752f792a86623b1e13732fcaf92a504cf6b8925ef7186e73 WatchSource:0}: Error finding container 3d02b515cb3560dc752f792a86623b1e13732fcaf92a504cf6b8925ef7186e73: Status 404 returned error can't find the container with id 3d02b515cb3560dc752f792a86623b1e13732fcaf92a504cf6b8925ef7186e73
Sep 10 15:50:05.199866 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:05.199823    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249" event={"ID":"6679105c-a4ef-4b97-8c50-142f3ce16e1c","Type":"ContainerStarted","Data":"b53c5371e627faf2149e985c1b584b9f710085d21b6b9aedc365141b41880c44"}
Sep 10 15:50:05.199866 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:05.199866    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249" event={"ID":"6679105c-a4ef-4b97-8c50-142f3ce16e1c","Type":"ContainerStarted","Data":"3d02b515cb3560dc752f792a86623b1e13732fcaf92a504cf6b8925ef7186e73"}
Sep 10 15:50:05.220299 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:05.220245    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249" podStartSLOduration=1.220230498 podStartE2EDuration="1.220230498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:05.21910388 +0000 UTC m=+2067.634899699" watchObservedRunningTime="2026-09-10 15:50:05.220230498 +0000 UTC m=+2067.636026316"
Sep 10 15:50:14.443190 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:14.443150    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:14.443190 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:14.443199    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:14.456044 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:14.456015    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:15.255486 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:15.255453    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:50:29.533173 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.533130    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"]
Sep 10 15:50:29.536451 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.536433    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.538703 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.538675    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d03fd470-kserve-self-signed-certs\""
Sep 10 15:50:29.546427 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.546400    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"]
Sep 10 15:50:29.613603 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.613561    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-model-cache\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.613817 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.613658    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr494\" (UniqueName: \"kubernetes.io/projected/b2e1507f-c171-49bb-b79f-6ccce39e6df5-kube-api-access-rr494\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.613817 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.613691    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-dshm\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.613817 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.613721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-tmp-dir\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.613970 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.613827    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2e1507f-c171-49bb-b79f-6ccce39e6df5-tls-certs\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.613970 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.613868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-home\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.715056 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.715014    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr494\" (UniqueName: \"kubernetes.io/projected/b2e1507f-c171-49bb-b79f-6ccce39e6df5-kube-api-access-rr494\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.715056 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.715063    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-dshm\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.715321 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.715183    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-tmp-dir\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.715321 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.715222    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2e1507f-c171-49bb-b79f-6ccce39e6df5-tls-certs\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.715321 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.715253    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-home\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.715478 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.715331    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-model-cache\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.715619 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.715592    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-tmp-dir\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.715749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.715644    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-home\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.715749 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.715674    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-model-cache\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.717580 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.717561    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-dshm\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.717714 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.717697    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2e1507f-c171-49bb-b79f-6ccce39e6df5-tls-certs\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.721673 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.721651    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr494\" (UniqueName: \"kubernetes.io/projected/b2e1507f-c171-49bb-b79f-6ccce39e6df5-kube-api-access-rr494\") pod \"e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") " pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:29.846429 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:29.846385    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:30.115052 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:30.114973    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"]
Sep 10 15:50:30.121520 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:50:30.121463    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2e1507f_c171_49bb_b79f_6ccce39e6df5.slice/crio-2ef3899462c12edc00788aac80a460d5b7ad379d97a51fae308550f261271243 WatchSource:0}: Error finding container 2ef3899462c12edc00788aac80a460d5b7ad379d97a51fae308550f261271243: Status 404 returned error can't find the container with id 2ef3899462c12edc00788aac80a460d5b7ad379d97a51fae308550f261271243
Sep 10 15:50:30.312418 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:30.312377    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4" event={"ID":"b2e1507f-c171-49bb-b79f-6ccce39e6df5","Type":"ContainerStarted","Data":"39f2cf068672791ef4bf6e042b1c7601994bbd09560e540f419dd75df0128a9d"}
Sep 10 15:50:30.312418 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:30.312424    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4" event={"ID":"b2e1507f-c171-49bb-b79f-6ccce39e6df5","Type":"ContainerStarted","Data":"2ef3899462c12edc00788aac80a460d5b7ad379d97a51fae308550f261271243"}
Sep 10 15:50:30.328544 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:30.328493    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4" podStartSLOduration=1.328478702 podStartE2EDuration="1.328478702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:30.326590843 +0000 UTC m=+2092.742386662" watchObservedRunningTime="2026-09-10 15:50:30.328478702 +0000 UTC m=+2092.744274521"
Sep 10 15:50:39.846852 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:39.846750    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:39.847390 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:39.846867    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:39.859301 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:39.859274    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:40.372738 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:40.372704    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:50:54.082756 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:54.082721    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"]
Sep 10 15:50:55.308048 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:50:55.308011    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"]
Sep 10 15:51:09.103230 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.103183    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249" podUID="6679105c-a4ef-4b97-8c50-142f3ce16e1c" containerName="main" containerID="cri-o://b53c5371e627faf2149e985c1b584b9f710085d21b6b9aedc365141b41880c44" gracePeriod=45
Sep 10 15:51:09.348437 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.348409    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:51:09.471914 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.471803    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-dshm\") pod \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") "
Sep 10 15:51:09.471914 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.471844    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-tmp-dir\") pod \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") "
Sep 10 15:51:09.471914 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.471884    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9d7b\" (UniqueName: \"kubernetes.io/projected/6679105c-a4ef-4b97-8c50-142f3ce16e1c-kube-api-access-s9d7b\") pod \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") "
Sep 10 15:51:09.472216 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.471924    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-model-cache\") pod \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") "
Sep 10 15:51:09.472216 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.471965    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6679105c-a4ef-4b97-8c50-142f3ce16e1c-tls-certs\") pod \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") "
Sep 10 15:51:09.472216 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.471987    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-home\") pod \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\" (UID: \"6679105c-a4ef-4b97-8c50-142f3ce16e1c\") "
Sep 10 15:51:09.472216 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.472058    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6679105c-a4ef-4b97-8c50-142f3ce16e1c" (UID: "6679105c-a4ef-4b97-8c50-142f3ce16e1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:51:09.472432 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.472238    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-home" (OuterVolumeSpecName: "home") pod "6679105c-a4ef-4b97-8c50-142f3ce16e1c" (UID: "6679105c-a4ef-4b97-8c50-142f3ce16e1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:51:09.472432 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.472249    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "6679105c-a4ef-4b97-8c50-142f3ce16e1c" (UID: "6679105c-a4ef-4b97-8c50-142f3ce16e1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:51:09.472432 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.472284    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-tmp-dir\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:09.474278 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.474248    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-dshm" (OuterVolumeSpecName: "dshm") pod "6679105c-a4ef-4b97-8c50-142f3ce16e1c" (UID: "6679105c-a4ef-4b97-8c50-142f3ce16e1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:51:09.474410 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.474297    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6679105c-a4ef-4b97-8c50-142f3ce16e1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6679105c-a4ef-4b97-8c50-142f3ce16e1c" (UID: "6679105c-a4ef-4b97-8c50-142f3ce16e1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:51:09.474410 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.474316    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6679105c-a4ef-4b97-8c50-142f3ce16e1c-kube-api-access-s9d7b" (OuterVolumeSpecName: "kube-api-access-s9d7b") pod "6679105c-a4ef-4b97-8c50-142f3ce16e1c" (UID: "6679105c-a4ef-4b97-8c50-142f3ce16e1c"). InnerVolumeSpecName "kube-api-access-s9d7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:51:09.499385 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.499349    2573 generic.go:356] "Generic (PLEG): container finished" podID="6679105c-a4ef-4b97-8c50-142f3ce16e1c" containerID="b53c5371e627faf2149e985c1b584b9f710085d21b6b9aedc365141b41880c44" exitCode=0
Sep 10 15:51:09.499524 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.499391    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"
Sep 10 15:51:09.499524 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.499433    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249" event={"ID":"6679105c-a4ef-4b97-8c50-142f3ce16e1c","Type":"ContainerDied","Data":"b53c5371e627faf2149e985c1b584b9f710085d21b6b9aedc365141b41880c44"}
Sep 10 15:51:09.499524 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.499480    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249" event={"ID":"6679105c-a4ef-4b97-8c50-142f3ce16e1c","Type":"ContainerDied","Data":"3d02b515cb3560dc752f792a86623b1e13732fcaf92a504cf6b8925ef7186e73"}
Sep 10 15:51:09.499524 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.499500    2573 scope.go:117] "RemoveContainer" containerID="b53c5371e627faf2149e985c1b584b9f710085d21b6b9aedc365141b41880c44"
Sep 10 15:51:09.508482 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.508465    2573 scope.go:117] "RemoveContainer" containerID="b53c5371e627faf2149e985c1b584b9f710085d21b6b9aedc365141b41880c44"
Sep 10 15:51:09.508716 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:51:09.508699    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53c5371e627faf2149e985c1b584b9f710085d21b6b9aedc365141b41880c44\": container with ID starting with b53c5371e627faf2149e985c1b584b9f710085d21b6b9aedc365141b41880c44 not found: ID does not exist" containerID="b53c5371e627faf2149e985c1b584b9f710085d21b6b9aedc365141b41880c44"
Sep 10 15:51:09.508766 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.508725    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53c5371e627faf2149e985c1b584b9f710085d21b6b9aedc365141b41880c44"} err="failed to get container status \"b53c5371e627faf2149e985c1b584b9f710085d21b6b9aedc365141b41880c44\": rpc error: code = NotFound desc = could not find container \"b53c5371e627faf2149e985c1b584b9f710085d21b6b9aedc365141b41880c44\": container with ID starting with b53c5371e627faf2149e985c1b584b9f710085d21b6b9aedc365141b41880c44 not found: ID does not exist"
Sep 10 15:51:09.518766 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.518745    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"]
Sep 10 15:51:09.520733 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.520706    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d03fd470-kserve-8dbf5f759-qv249"]
Sep 10 15:51:09.573124 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.573077    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6679105c-a4ef-4b97-8c50-142f3ce16e1c-tls-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:09.573124 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.573117    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-home\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:09.573124 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.573129    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-dshm\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:09.573352 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.573142    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9d7b\" (UniqueName: \"kubernetes.io/projected/6679105c-a4ef-4b97-8c50-142f3ce16e1c-kube-api-access-s9d7b\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:09.573352 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:09.573161    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6679105c-a4ef-4b97-8c50-142f3ce16e1c-model-cache\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:10.125405 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.125369    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6679105c-a4ef-4b97-8c50-142f3ce16e1c" path="/var/lib/kubelet/pods/6679105c-a4ef-4b97-8c50-142f3ce16e1c/volumes"
Sep 10 15:51:10.329446 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.329406    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4" podUID="b2e1507f-c171-49bb-b79f-6ccce39e6df5" containerName="main" containerID="cri-o://39f2cf068672791ef4bf6e042b1c7601994bbd09560e540f419dd75df0128a9d" gracePeriod=45
Sep 10 15:51:10.362167 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.362123    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4" podUID="b2e1507f-c171-49bb-b79f-6ccce39e6df5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 10 15:51:10.507840 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.507780    2573 generic.go:356] "Generic (PLEG): container finished" podID="b2e1507f-c171-49bb-b79f-6ccce39e6df5" containerID="39f2cf068672791ef4bf6e042b1c7601994bbd09560e540f419dd75df0128a9d" exitCode=0
Sep 10 15:51:10.507840 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.507822    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4" event={"ID":"b2e1507f-c171-49bb-b79f-6ccce39e6df5","Type":"ContainerDied","Data":"39f2cf068672791ef4bf6e042b1c7601994bbd09560e540f419dd75df0128a9d"}
Sep 10 15:51:10.585518 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.585493    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:51:10.681249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.681208    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-dshm\") pod \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") "
Sep 10 15:51:10.681249 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.681253    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-model-cache\") pod \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") "
Sep 10 15:51:10.681496 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.681271    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2e1507f-c171-49bb-b79f-6ccce39e6df5-tls-certs\") pod \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") "
Sep 10 15:51:10.681496 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.681349    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-home\") pod \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") "
Sep 10 15:51:10.681496 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.681371    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr494\" (UniqueName: \"kubernetes.io/projected/b2e1507f-c171-49bb-b79f-6ccce39e6df5-kube-api-access-rr494\") pod \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") "
Sep 10 15:51:10.681496 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.681393    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-tmp-dir\") pod \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\" (UID: \"b2e1507f-c171-49bb-b79f-6ccce39e6df5\") "
Sep 10 15:51:10.681834 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.681563    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2e1507f-c171-49bb-b79f-6ccce39e6df5" (UID: "b2e1507f-c171-49bb-b79f-6ccce39e6df5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:51:10.681834 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.681605    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-home" (OuterVolumeSpecName: "home") pod "b2e1507f-c171-49bb-b79f-6ccce39e6df5" (UID: "b2e1507f-c171-49bb-b79f-6ccce39e6df5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:51:10.681834 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.681721    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2e1507f-c171-49bb-b79f-6ccce39e6df5" (UID: "b2e1507f-c171-49bb-b79f-6ccce39e6df5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:51:10.683607 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.683585    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2e1507f-c171-49bb-b79f-6ccce39e6df5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2e1507f-c171-49bb-b79f-6ccce39e6df5" (UID: "b2e1507f-c171-49bb-b79f-6ccce39e6df5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:51:10.683688 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.683622    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-dshm" (OuterVolumeSpecName: "dshm") pod "b2e1507f-c171-49bb-b79f-6ccce39e6df5" (UID: "b2e1507f-c171-49bb-b79f-6ccce39e6df5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:51:10.683731 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.683704    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e1507f-c171-49bb-b79f-6ccce39e6df5-kube-api-access-rr494" (OuterVolumeSpecName: "kube-api-access-rr494") pod "b2e1507f-c171-49bb-b79f-6ccce39e6df5" (UID: "b2e1507f-c171-49bb-b79f-6ccce39e6df5"). InnerVolumeSpecName "kube-api-access-rr494". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:51:10.782097 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.782015    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-home\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:10.782097 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.782045    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr494\" (UniqueName: \"kubernetes.io/projected/b2e1507f-c171-49bb-b79f-6ccce39e6df5-kube-api-access-rr494\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:10.782097 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.782057    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-tmp-dir\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:10.782097 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.782065    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-dshm\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:10.782097 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.782074    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2e1507f-c171-49bb-b79f-6ccce39e6df5-model-cache\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:10.782097 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:10.782081    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2e1507f-c171-49bb-b79f-6ccce39e6df5-tls-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:11.515074 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:11.515040    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"
Sep 10 15:51:11.515074 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:11.515054    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4" event={"ID":"b2e1507f-c171-49bb-b79f-6ccce39e6df5","Type":"ContainerDied","Data":"2ef3899462c12edc00788aac80a460d5b7ad379d97a51fae308550f261271243"}
Sep 10 15:51:11.515551 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:11.515100    2573 scope.go:117] "RemoveContainer" containerID="39f2cf068672791ef4bf6e042b1c7601994bbd09560e540f419dd75df0128a9d"
Sep 10 15:51:11.534531 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:11.534499    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"]
Sep 10 15:51:11.538360 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:11.538336    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d03fd470-kserve-66cdc76c95-x56w4"]
Sep 10 15:51:12.125414 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:12.125381    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2e1507f-c171-49bb-b79f-6ccce39e6df5" path="/var/lib/kubelet/pods/b2e1507f-c171-49bb-b79f-6ccce39e6df5/volumes"
Sep 10 15:51:17.445959 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.445927    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"]
Sep 10 15:51:17.446375 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.446246    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6679105c-a4ef-4b97-8c50-142f3ce16e1c" containerName="main"
Sep 10 15:51:17.446375 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.446257    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6679105c-a4ef-4b97-8c50-142f3ce16e1c" containerName="main"
Sep 10 15:51:17.446375 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.446274    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2e1507f-c171-49bb-b79f-6ccce39e6df5" containerName="main"
Sep 10 15:51:17.446375 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.446280    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2e1507f-c171-49bb-b79f-6ccce39e6df5" containerName="main"
Sep 10 15:51:17.446375 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.446335    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6679105c-a4ef-4b97-8c50-142f3ce16e1c" containerName="main"
Sep 10 15:51:17.446375 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.446343    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2e1507f-c171-49bb-b79f-6ccce39e6df5" containerName="main"
Sep 10 15:51:17.450550 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.450529    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:17.460060 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.460031    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"]
Sep 10 15:51:17.641070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.641033    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq896\" (UniqueName: \"kubernetes.io/projected/798cf16f-07f3-4885-ac92-69ccb2a7ae6b-kube-api-access-jq896\") pod \"maas-api-684c4cfcb5-h959h\" (UID: \"798cf16f-07f3-4885-ac92-69ccb2a7ae6b\") " pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:17.641070 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.641070    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798cf16f-07f3-4885-ac92-69ccb2a7ae6b-metrics-certs\") pod \"maas-api-684c4cfcb5-h959h\" (UID: \"798cf16f-07f3-4885-ac92-69ccb2a7ae6b\") " pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:17.641314 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.641133    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/798cf16f-07f3-4885-ac92-69ccb2a7ae6b-maas-api-tls\") pod \"maas-api-684c4cfcb5-h959h\" (UID: \"798cf16f-07f3-4885-ac92-69ccb2a7ae6b\") " pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:17.641314 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.641169    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/798cf16f-07f3-4885-ac92-69ccb2a7ae6b-openshift-service-ca\") pod \"maas-api-684c4cfcb5-h959h\" (UID: \"798cf16f-07f3-4885-ac92-69ccb2a7ae6b\") " pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:17.742326 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.742233    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/798cf16f-07f3-4885-ac92-69ccb2a7ae6b-openshift-service-ca\") pod \"maas-api-684c4cfcb5-h959h\" (UID: \"798cf16f-07f3-4885-ac92-69ccb2a7ae6b\") " pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:17.742326 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.742290    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq896\" (UniqueName: \"kubernetes.io/projected/798cf16f-07f3-4885-ac92-69ccb2a7ae6b-kube-api-access-jq896\") pod \"maas-api-684c4cfcb5-h959h\" (UID: \"798cf16f-07f3-4885-ac92-69ccb2a7ae6b\") " pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:17.742326 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.742318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798cf16f-07f3-4885-ac92-69ccb2a7ae6b-metrics-certs\") pod \"maas-api-684c4cfcb5-h959h\" (UID: \"798cf16f-07f3-4885-ac92-69ccb2a7ae6b\") " pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:17.742631 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.742384    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/798cf16f-07f3-4885-ac92-69ccb2a7ae6b-maas-api-tls\") pod \"maas-api-684c4cfcb5-h959h\" (UID: \"798cf16f-07f3-4885-ac92-69ccb2a7ae6b\") " pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:17.742950 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.742926    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/798cf16f-07f3-4885-ac92-69ccb2a7ae6b-openshift-service-ca\") pod \"maas-api-684c4cfcb5-h959h\" (UID: \"798cf16f-07f3-4885-ac92-69ccb2a7ae6b\") " pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:17.745012 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.744990    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798cf16f-07f3-4885-ac92-69ccb2a7ae6b-metrics-certs\") pod \"maas-api-684c4cfcb5-h959h\" (UID: \"798cf16f-07f3-4885-ac92-69ccb2a7ae6b\") " pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:17.745129 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.745033    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/798cf16f-07f3-4885-ac92-69ccb2a7ae6b-maas-api-tls\") pod \"maas-api-684c4cfcb5-h959h\" (UID: \"798cf16f-07f3-4885-ac92-69ccb2a7ae6b\") " pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:17.752158 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.752136    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq896\" (UniqueName: \"kubernetes.io/projected/798cf16f-07f3-4885-ac92-69ccb2a7ae6b-kube-api-access-jq896\") pod \"maas-api-684c4cfcb5-h959h\" (UID: \"798cf16f-07f3-4885-ac92-69ccb2a7ae6b\") " pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:17.761969 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:17.761946    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:18.038437 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:18.038359    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"]
Sep 10 15:51:18.041093 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:51:18.041063    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798cf16f_07f3_4885_ac92_69ccb2a7ae6b.slice/crio-468e5f9dcad8e7e4db0874d996fec178f022e266ad8d8c01c01469bc4e04d929 WatchSource:0}: Error finding container 468e5f9dcad8e7e4db0874d996fec178f022e266ad8d8c01c01469bc4e04d929: Status 404 returned error can't find the container with id 468e5f9dcad8e7e4db0874d996fec178f022e266ad8d8c01c01469bc4e04d929
Sep 10 15:51:18.547643 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:18.547603    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h" event={"ID":"798cf16f-07f3-4885-ac92-69ccb2a7ae6b","Type":"ContainerStarted","Data":"468e5f9dcad8e7e4db0874d996fec178f022e266ad8d8c01c01469bc4e04d929"}
Sep 10 15:51:19.553516 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:19.553481    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h" event={"ID":"798cf16f-07f3-4885-ac92-69ccb2a7ae6b","Type":"ContainerStarted","Data":"27adc273ebe9b7e6ef640058a41cb4b480eaf28a585a0ab452698cdc465ca3d6"}
Sep 10 15:51:19.554044 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:19.553594    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:19.571587 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:19.571529    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h" podStartSLOduration=2.037748805 podStartE2EDuration="2.571508454s" totalImagesPullingTime="533.759649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:19.568337639 +0000 UTC m=+2141.984133458" watchObservedRunningTime="2026-09-10 15:51:19.571508454 +0000 UTC m=+2141.987304274"
Sep 10 15:51:25.563049 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:25.563013    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-684c4cfcb5-h959h"
Sep 10 15:51:33.014977 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:33.014940    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"]
Sep 10 15:51:33.015395 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:33.015172    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm" podUID="8bf98897-ed7a-4933-8a73-2b3a3bef2d46" containerName="maas-api" containerID="cri-o://3bac7b601772fdd4f60f99ff2ac62f10f79ee15e2001ba9256447b167e15e2e2" gracePeriod=30
Sep 10 15:51:34.272498 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.272470    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:51:34.385343 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.385307    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-metrics-certs\") pod \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") "
Sep 10 15:51:34.385536 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.385372    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-maas-api-tls\") pod \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") "
Sep 10 15:51:34.385536 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.385418    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgr4w\" (UniqueName: \"kubernetes.io/projected/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-kube-api-access-dgr4w\") pod \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") "
Sep 10 15:51:34.385536 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.385444    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-openshift-service-ca\") pod \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\" (UID: \"8bf98897-ed7a-4933-8a73-2b3a3bef2d46\") "
Sep 10 15:51:34.385841 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.385810    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8bf98897-ed7a-4933-8a73-2b3a3bef2d46" (UID: "8bf98897-ed7a-4933-8a73-2b3a3bef2d46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:51:34.387663 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.387637    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8bf98897-ed7a-4933-8a73-2b3a3bef2d46" (UID: "8bf98897-ed7a-4933-8a73-2b3a3bef2d46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:51:34.387724 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.387657    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8bf98897-ed7a-4933-8a73-2b3a3bef2d46" (UID: "8bf98897-ed7a-4933-8a73-2b3a3bef2d46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:51:34.387771 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.387740    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-kube-api-access-dgr4w" (OuterVolumeSpecName: "kube-api-access-dgr4w") pod "8bf98897-ed7a-4933-8a73-2b3a3bef2d46" (UID: "8bf98897-ed7a-4933-8a73-2b3a3bef2d46"). InnerVolumeSpecName "kube-api-access-dgr4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:51:34.486843 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.486771    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgr4w\" (UniqueName: \"kubernetes.io/projected/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-kube-api-access-dgr4w\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:34.486843 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.486832    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-openshift-service-ca\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:34.486843 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.486843    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-metrics-certs\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:34.486843 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.486852    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf98897-ed7a-4933-8a73-2b3a3bef2d46-maas-api-tls\") on node \"ip-10-0-134-115.ec2.internal\" DevicePath \"\""
Sep 10 15:51:34.630379 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.630342    2573 generic.go:356] "Generic (PLEG): container finished" podID="8bf98897-ed7a-4933-8a73-2b3a3bef2d46" containerID="3bac7b601772fdd4f60f99ff2ac62f10f79ee15e2001ba9256447b167e15e2e2" exitCode=0
Sep 10 15:51:34.630582 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.630394    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"
Sep 10 15:51:34.630582 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.630439    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm" event={"ID":"8bf98897-ed7a-4933-8a73-2b3a3bef2d46","Type":"ContainerDied","Data":"3bac7b601772fdd4f60f99ff2ac62f10f79ee15e2001ba9256447b167e15e2e2"}
Sep 10 15:51:34.630582 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.630500    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-788d559bf-sjxxm" event={"ID":"8bf98897-ed7a-4933-8a73-2b3a3bef2d46","Type":"ContainerDied","Data":"ebb651c72de6801db4bda2f20b95ea7afeacc230ee84815b2a6c92643c71364d"}
Sep 10 15:51:34.630582 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.630524    2573 scope.go:117] "RemoveContainer" containerID="3bac7b601772fdd4f60f99ff2ac62f10f79ee15e2001ba9256447b167e15e2e2"
Sep 10 15:51:34.639718 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.639702    2573 scope.go:117] "RemoveContainer" containerID="3bac7b601772fdd4f60f99ff2ac62f10f79ee15e2001ba9256447b167e15e2e2"
Sep 10 15:51:34.639988 ip-10-0-134-115 kubenswrapper[2573]: E0910 15:51:34.639968    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bac7b601772fdd4f60f99ff2ac62f10f79ee15e2001ba9256447b167e15e2e2\": container with ID starting with 3bac7b601772fdd4f60f99ff2ac62f10f79ee15e2001ba9256447b167e15e2e2 not found: ID does not exist" containerID="3bac7b601772fdd4f60f99ff2ac62f10f79ee15e2001ba9256447b167e15e2e2"
Sep 10 15:51:34.640047 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.640003    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bac7b601772fdd4f60f99ff2ac62f10f79ee15e2001ba9256447b167e15e2e2"} err="failed to get container status \"3bac7b601772fdd4f60f99ff2ac62f10f79ee15e2001ba9256447b167e15e2e2\": rpc error: code = NotFound desc = could not find container \"3bac7b601772fdd4f60f99ff2ac62f10f79ee15e2001ba9256447b167e15e2e2\": container with ID starting with 3bac7b601772fdd4f60f99ff2ac62f10f79ee15e2001ba9256447b167e15e2e2 not found: ID does not exist"
Sep 10 15:51:34.649784 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.649759    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"]
Sep 10 15:51:34.654327 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:34.654306    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-788d559bf-sjxxm"]
Sep 10 15:51:36.125174 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:36.125122    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf98897-ed7a-4933-8a73-2b3a3bef2d46" path="/var/lib/kubelet/pods/8bf98897-ed7a-4933-8a73-2b3a3bef2d46/volumes"
Sep 10 15:51:48.204132 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:48.204098    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b6846c95c-6w464_46084e47-e4cb-472d-af64-1de691f0eca0/authorino/0.log"
Sep 10 15:51:53.335385 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:53.335338    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-tc64f_59318b3f-2caa-4029-907d-63beb057d67e/manager/0.log"
Sep 10 15:51:53.445283 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:53.445229    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zzt29_d2d57895-c3bd-4629-baf0-b80543c9f60f/manager/0.log"
Sep 10 15:51:53.568304 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:53.568270    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57cc444cff-xmhrj_92f0ec48-b3a9-4803-b6ac-ca11501b055e/manager/0.log"
Sep 10 15:51:54.155155 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:54.155125    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76f847f9cd-mtl9p_65312fa5-0192-4130-81c8-2e9b54329d00/copy-manifests/0.log"
Sep 10 15:51:54.167646 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:54.167619    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76f847f9cd-mtl9p_65312fa5-0192-4130-81c8-2e9b54329d00/manager/0.log"
Sep 10 15:51:54.484827 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:54.484685    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-684c4cfcb5-h959h_798cf16f-07f3-4885-ac92-69ccb2a7ae6b/maas-api/0.log"
Sep 10 15:51:54.925546 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:54.925515    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-x8npx_a314adf2-281b-42d4-8c30-d2d0e00d7e4f/postgres/0.log"
Sep 10 15:51:55.663163 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:55.663131    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b6846c95c-6w464_46084e47-e4cb-472d-af64-1de691f0eca0/authorino/0.log"
Sep 10 15:51:55.784217 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:55.784182    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-jtnw6_d5604c15-0aca-435f-ae29-db31db0eb6ea/manager/0.log"
Sep 10 15:51:55.889604 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:55.889555    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-5tz85_c34331c8-06d5-4b13-8901-36325d5c2a6e/manager/0.log"
Sep 10 15:51:56.135606 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:56.135569    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-2ft47_dd0b3bce-8651-40c9-a4c8-1b1cefde3878/manager/0.log"
Sep 10 15:51:56.367806 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:56.367762    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-62jls_87213ea5-dbc7-46f0-abfc-613d537ad421/manager/0.log"
Sep 10 15:51:56.935739 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:56.935711    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84bd548dcf-gt6gm_181199ba-a7a1-4031-a5ac-8b8eb120b694/kube-auth-proxy/0.log"
Sep 10 15:51:57.388509 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:57.388474    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kxctv_4e9b368b-9061-4e9d-be30-4b59ada419c8/payload-processing/0.log"
Sep 10 15:51:58.160322 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:51:58.160294    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-clkjf_a2a4ddb4-eb74-4bc1-84c7-b57ea8a5a9d4/main/0.log"
Sep 10 15:52:03.898646 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:03.898605    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8zzjt/must-gather-fz4fk"]
Sep 10 15:52:03.899151 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:03.899128    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf98897-ed7a-4933-8a73-2b3a3bef2d46" containerName="maas-api"
Sep 10 15:52:03.899151 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:03.899147    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf98897-ed7a-4933-8a73-2b3a3bef2d46" containerName="maas-api"
Sep 10 15:52:03.899279 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:03.899233    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf98897-ed7a-4933-8a73-2b3a3bef2d46" containerName="maas-api"
Sep 10 15:52:03.902415 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:03.902392    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8zzjt/must-gather-fz4fk"
Sep 10 15:52:03.904579 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:03.904559    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8zzjt\"/\"kube-root-ca.crt\""
Sep 10 15:52:03.904701 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:03.904663    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8zzjt\"/\"default-dockercfg-kqxjj\""
Sep 10 15:52:03.904805 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:03.904775    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8zzjt\"/\"openshift-service-ca.crt\""
Sep 10 15:52:03.911990 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:03.911969    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8zzjt/must-gather-fz4fk"]
Sep 10 15:52:04.044388 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:04.044350    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/de60b29a-0d67-4fd8-af2a-2ac50a630e1a-must-gather-output\") pod \"must-gather-fz4fk\" (UID: \"de60b29a-0d67-4fd8-af2a-2ac50a630e1a\") " pod="openshift-must-gather-8zzjt/must-gather-fz4fk"
Sep 10 15:52:04.044561 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:04.044400    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xhd\" (UniqueName: \"kubernetes.io/projected/de60b29a-0d67-4fd8-af2a-2ac50a630e1a-kube-api-access-m7xhd\") pod \"must-gather-fz4fk\" (UID: \"de60b29a-0d67-4fd8-af2a-2ac50a630e1a\") " pod="openshift-must-gather-8zzjt/must-gather-fz4fk"
Sep 10 15:52:04.145057 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:04.145027    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xhd\" (UniqueName: \"kubernetes.io/projected/de60b29a-0d67-4fd8-af2a-2ac50a630e1a-kube-api-access-m7xhd\") pod \"must-gather-fz4fk\" (UID: \"de60b29a-0d67-4fd8-af2a-2ac50a630e1a\") " pod="openshift-must-gather-8zzjt/must-gather-fz4fk"
Sep 10 15:52:04.145255 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:04.145118    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/de60b29a-0d67-4fd8-af2a-2ac50a630e1a-must-gather-output\") pod \"must-gather-fz4fk\" (UID: \"de60b29a-0d67-4fd8-af2a-2ac50a630e1a\") " pod="openshift-must-gather-8zzjt/must-gather-fz4fk"
Sep 10 15:52:04.145442 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:04.145421    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/de60b29a-0d67-4fd8-af2a-2ac50a630e1a-must-gather-output\") pod \"must-gather-fz4fk\" (UID: \"de60b29a-0d67-4fd8-af2a-2ac50a630e1a\") " pod="openshift-must-gather-8zzjt/must-gather-fz4fk"
Sep 10 15:52:04.171303 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:04.171239    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xhd\" (UniqueName: \"kubernetes.io/projected/de60b29a-0d67-4fd8-af2a-2ac50a630e1a-kube-api-access-m7xhd\") pod \"must-gather-fz4fk\" (UID: \"de60b29a-0d67-4fd8-af2a-2ac50a630e1a\") " pod="openshift-must-gather-8zzjt/must-gather-fz4fk"
Sep 10 15:52:04.212008 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:04.211973    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8zzjt/must-gather-fz4fk"
Sep 10 15:52:04.482231 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:04.482200    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8zzjt/must-gather-fz4fk"]
Sep 10 15:52:04.484821 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:52:04.484777    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde60b29a_0d67_4fd8_af2a_2ac50a630e1a.slice/crio-a02cb4fa031db0bed67a3ee60674929b2e9e91d2276be310679d11a744789d77 WatchSource:0}: Error finding container a02cb4fa031db0bed67a3ee60674929b2e9e91d2276be310679d11a744789d77: Status 404 returned error can't find the container with id a02cb4fa031db0bed67a3ee60674929b2e9e91d2276be310679d11a744789d77
Sep 10 15:52:04.487250 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:04.487229    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:52:04.774472 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:04.774374    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8zzjt/must-gather-fz4fk" event={"ID":"de60b29a-0d67-4fd8-af2a-2ac50a630e1a","Type":"ContainerStarted","Data":"a02cb4fa031db0bed67a3ee60674929b2e9e91d2276be310679d11a744789d77"}
Sep 10 15:52:05.782779 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:05.782740    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8zzjt/must-gather-fz4fk" event={"ID":"de60b29a-0d67-4fd8-af2a-2ac50a630e1a","Type":"ContainerStarted","Data":"3e5a6c0e1b7c1c0899a70ece93b31ca7d2f688c9f136fd842d2f1cf6148106f7"}
Sep 10 15:52:05.783188 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:05.782821    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8zzjt/must-gather-fz4fk" event={"ID":"de60b29a-0d67-4fd8-af2a-2ac50a630e1a","Type":"ContainerStarted","Data":"9504b6d128739128f9ed5e2f9b8fb262e57f64bf3bd912a7698d78f653de0df5"}
Sep 10 15:52:05.804071 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:05.804020    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8zzjt/must-gather-fz4fk" podStartSLOduration=1.992436308 podStartE2EDuration="2.804003908s" totalImagesPullingTime="811.5676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:05.802540291 +0000 UTC m=+2188.218336193" watchObservedRunningTime="2026-09-10 15:52:05.804003908 +0000 UTC m=+2188.219799726"
Sep 10 15:52:06.922463 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:06.922419    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wkpgh_3f858dfb-38ba-44b7-8b7d-b38cfb655428/global-pull-secret-syncer/0.log"
Sep 10 15:52:07.072435 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:07.072386    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zc2lh_216864e7-bd81-4a5a-a0d7-f243edea4857/konnectivity-agent/0.log"
Sep 10 15:52:07.096712 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:07.096686    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-115.ec2.internal_17327a0c112603d3f5deb1d673c13162/haproxy/0.log"
Sep 10 15:52:11.541814 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:11.530456    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b6846c95c-6w464_46084e47-e4cb-472d-af64-1de691f0eca0/authorino/0.log"
Sep 10 15:52:11.562755 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:11.562623    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-jtnw6_d5604c15-0aca-435f-ae29-db31db0eb6ea/manager/0.log"
Sep 10 15:52:11.587861 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:11.587836    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-5tz85_c34331c8-06d5-4b13-8901-36325d5c2a6e/manager/0.log"
Sep 10 15:52:11.668183 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:11.668152    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-2ft47_dd0b3bce-8651-40c9-a4c8-1b1cefde3878/manager/0.log"
Sep 10 15:52:11.813036 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:11.812920    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-62jls_87213ea5-dbc7-46f0-abfc-613d537ad421/manager/0.log"
Sep 10 15:52:13.734862 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:13.734819    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kwgtl_91a736cd-69d5-4a64-9eda-566b8cc61889/node-exporter/0.log"
Sep 10 15:52:13.757008 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:13.756931    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kwgtl_91a736cd-69d5-4a64-9eda-566b8cc61889/kube-rbac-proxy/0.log"
Sep 10 15:52:13.787523 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:13.787478    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kwgtl_91a736cd-69d5-4a64-9eda-566b8cc61889/init-textfile/0.log"
Sep 10 15:52:17.081537 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.081499    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"]
Sep 10 15:52:17.090366 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.090335    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.097300 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.097259    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"]
Sep 10 15:52:17.164484 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.164446    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a31f2147-ce39-4421-81a6-abf4afa94484-podres\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.164776 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.164742    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a31f2147-ce39-4421-81a6-abf4afa94484-proc\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.165035 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.165019    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a31f2147-ce39-4421-81a6-abf4afa94484-sys\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.165231 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.165195    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk6xt\" (UniqueName: \"kubernetes.io/projected/a31f2147-ce39-4421-81a6-abf4afa94484-kube-api-access-gk6xt\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.165473 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.165444    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a31f2147-ce39-4421-81a6-abf4afa94484-lib-modules\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.266689 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.266645    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a31f2147-ce39-4421-81a6-abf4afa94484-proc\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.266943 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.266914    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a31f2147-ce39-4421-81a6-abf4afa94484-proc\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.267145 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.266986    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a31f2147-ce39-4421-81a6-abf4afa94484-sys\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.267352 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.267316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk6xt\" (UniqueName: \"kubernetes.io/projected/a31f2147-ce39-4421-81a6-abf4afa94484-kube-api-access-gk6xt\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.267549 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.267516    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a31f2147-ce39-4421-81a6-abf4afa94484-lib-modules\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.267703 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.267687    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a31f2147-ce39-4421-81a6-abf4afa94484-podres\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.268024 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.268005    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a31f2147-ce39-4421-81a6-abf4afa94484-podres\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.268416 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.268345    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a31f2147-ce39-4421-81a6-abf4afa94484-lib-modules\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.268563 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.267064    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a31f2147-ce39-4421-81a6-abf4afa94484-sys\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.278033 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.278004    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk6xt\" (UniqueName: \"kubernetes.io/projected/a31f2147-ce39-4421-81a6-abf4afa94484-kube-api-access-gk6xt\") pod \"perf-node-gather-daemonset-zxghh\" (UID: \"a31f2147-ce39-4421-81a6-abf4afa94484\") " pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.407580 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.407486    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:17.712627 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.712584    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"]
Sep 10 15:52:17.715143 ip-10-0-134-115 kubenswrapper[2573]: W0910 15:52:17.715099    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda31f2147_ce39_4421_81a6_abf4afa94484.slice/crio-7bf27e3a39434cde229306ec4604b2b31bb28add5dccc6cdefb7b42c4d6feb1e WatchSource:0}: Error finding container 7bf27e3a39434cde229306ec4604b2b31bb28add5dccc6cdefb7b42c4d6feb1e: Status 404 returned error can't find the container with id 7bf27e3a39434cde229306ec4604b2b31bb28add5dccc6cdefb7b42c4d6feb1e
Sep 10 15:52:17.865440 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.864488    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh" event={"ID":"a31f2147-ce39-4421-81a6-abf4afa94484","Type":"ContainerStarted","Data":"f2199f8127d73bb7dbfa9c51dec80b239abdbc3803d55dff8173534f34923cff"}
Sep 10 15:52:17.865440 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.864550    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh" event={"ID":"a31f2147-ce39-4421-81a6-abf4afa94484","Type":"ContainerStarted","Data":"7bf27e3a39434cde229306ec4604b2b31bb28add5dccc6cdefb7b42c4d6feb1e"}
Sep 10 15:52:17.865440 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:17.865354    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:19.364290 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:19.364261    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wmlgv_d6a33951-8222-4e14-9630-2b4b11140aaf/dns/0.log"
Sep 10 15:52:19.386034 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:19.386007    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wmlgv_d6a33951-8222-4e14-9630-2b4b11140aaf/kube-rbac-proxy/0.log"
Sep 10 15:52:19.485703 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:19.485665    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kqvj8_04e0b284-688f-4bd7-a282-f9052bb11bf4/dns-node-resolver/0.log"
Sep 10 15:52:20.143523 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:20.143496    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xsmjp_b7ab4301-eaa0-4db1-9a8d-a2ae36a8808d/node-ca/0.log"
Sep 10 15:52:21.440221 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:21.440193    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84bd548dcf-gt6gm_181199ba-a7a1-4031-a5ac-8b8eb120b694/kube-auth-proxy/0.log"
Sep 10 15:52:21.582119 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:21.582088    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kxctv_4e9b368b-9061-4e9d-be30-4b59ada419c8/payload-processing/0.log"
Sep 10 15:52:22.573277 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:22.573218    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z8tnd_07b0c4f8-0d00-40cc-a8b6-eb84f9546b3a/serve-healthcheck-canary/0.log"
Sep 10 15:52:23.238537 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:23.238510    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dl4c4_b12f58a1-a40c-4024-8411-2334e8d0244f/kube-rbac-proxy/0.log"
Sep 10 15:52:23.265352 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:23.265321    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dl4c4_b12f58a1-a40c-4024-8411-2334e8d0244f/exporter/0.log"
Sep 10 15:52:23.291890 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:23.291855    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dl4c4_b12f58a1-a40c-4024-8411-2334e8d0244f/extractor/0.log"
Sep 10 15:52:24.895305 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:24.895268    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh"
Sep 10 15:52:24.918361 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:24.918300    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8zzjt/perf-node-gather-daemonset-zxghh" podStartSLOduration=7.918278766 podStartE2EDuration="7.918278766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:17.883772538 +0000 UTC m=+2200.299568351" watchObservedRunningTime="2026-09-10 15:52:24.918278766 +0000 UTC m=+2207.334074586"
Sep 10 15:52:25.734055 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:25.734018    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-tc64f_59318b3f-2caa-4029-907d-63beb057d67e/manager/0.log"
Sep 10 15:52:25.981246 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:25.980975    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zzt29_d2d57895-c3bd-4629-baf0-b80543c9f60f/manager/0.log"
Sep 10 15:52:26.040321 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:26.040286    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57cc444cff-xmhrj_92f0ec48-b3a9-4803-b6ac-ca11501b055e/manager/0.log"
Sep 10 15:52:26.354820 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:26.354754    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76f847f9cd-mtl9p_65312fa5-0192-4130-81c8-2e9b54329d00/manager/0.log"
Sep 10 15:52:26.379072 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:26.379040    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76f847f9cd-mtl9p_65312fa5-0192-4130-81c8-2e9b54329d00/copy-manifests/0.log"
Sep 10 15:52:27.835425 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:27.835393    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-6x82w_dc41b164-0531-49a0-8f6a-eb7da8e60c54/openshift-lws-operator/0.log"
Sep 10 15:52:28.091695 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:28.091616    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84bd548dcf-gt6gm_181199ba-a7a1-4031-a5ac-8b8eb120b694/kube-auth-proxy/0.log"
Sep 10 15:52:28.204584 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:28.204541    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kxctv_4e9b368b-9061-4e9d-be30-4b59ada419c8/payload-processing/0.log"
Sep 10 15:52:29.994955 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:29.994916    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84bd548dcf-gt6gm_181199ba-a7a1-4031-a5ac-8b8eb120b694/kube-auth-proxy/0.log"
Sep 10 15:52:30.104913 ip-10-0-134-115 kubenswrapper[2573]: I0910 15:52:30.104880    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-kxctv_4e9b368b-9061-4e9d-be30-4b59ada419c8/payload-processing/0.log"