Sep 07 12:59:13.477927 ip-10-0-128-114 systemd[1]: Starting Kubernetes Kubelet...
Sep 07 12:59:13.897421 ip-10-0-128-114 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 12:59:13.897421 ip-10-0-128-114 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 07 12:59:13.897421 ip-10-0-128-114 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 12:59:13.897421 ip-10-0-128-114 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 07 12:59:13.897421 ip-10-0-128-114 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 12:59:13.898122 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.897516    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 07 12:59:13.906710 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906682    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 12:59:13.906710 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906700    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 12:59:13.906710 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906705    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 12:59:13.906710 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906711    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 12:59:13.906710 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906715    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 12:59:13.906710 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906719    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906724    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906728    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906732    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906736    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906740    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906744    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906747    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906750    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906754    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906758    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906761    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906786    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906792    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906797    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906801    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906805    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906809    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906813    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906816    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 12:59:13.907102 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906820    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906824    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906828    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906832    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906835    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906840    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906844    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906849    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906853    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906857    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906862    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906866    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906870    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906875    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906879    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906884    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906888    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906892    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906897    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906901    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 12:59:13.907950 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906905    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906910    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906914    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906918    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906925    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906932    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906938    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906942    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906946    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906950    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906956    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906960    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906967    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906974    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906981    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906986    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906992    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.906996    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 12:59:13.908836 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907000    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907004    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907008    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907012    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907017    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907021    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907026    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907031    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907035    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907039    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907043    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907047    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907052    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907056    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907062    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907068    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907073    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907078    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907082    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907087    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 12:59:13.909715 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907093    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907097    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907101    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907105    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907109    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907113    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907117    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907122    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907126    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907130    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907135    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907139    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907144    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907148    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907153    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.907157    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908340    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908353    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908361    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908368    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 12:59:13.910559 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908373    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908378    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908383    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908388    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908394    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908398    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908403    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908408    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908412    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908417    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908421    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908426    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908431    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908437    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908441    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908445    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908449    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908455    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908461    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 12:59:13.911437 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908467    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908472    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908477    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908482    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908486    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908491    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908495    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908499    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908503    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908507    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908511    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908518    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908523    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908527    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908531    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908536    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908540    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908544    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908548    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908552    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 12:59:13.912042 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908556    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908560    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908564    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908577    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908582    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908586    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908592    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908599    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908604    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908608    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908629    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908634    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908638    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908642    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908647    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908651    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908656    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908660    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908666    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908671    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 12:59:13.912673 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908675    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908680    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908684    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908688    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908692    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908696    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908700    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908704    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908708    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908713    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908717    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908721    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908726    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908730    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908734    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908738    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908743    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908746    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908750    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908755    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 12:59:13.913203 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908760    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908764    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908768    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908773    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908777    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908782    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908786    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908790    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908795    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908803    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908807    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908811    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908815    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908820    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908824    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.908828    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909457    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909467    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909473    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909478    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 12:59:13.913768 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909483    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909487    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909491    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909495    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909500    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909504    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909508    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909512    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909517    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909521    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909525    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909529    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909534    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909538    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909543    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909548    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909552    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909556    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909561    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909565    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 12:59:13.914425 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909570    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909575    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909579    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909583    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909587    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909592    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909597    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909601    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909605    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909628    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909634    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909638    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909642    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909647    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909651    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909656    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909660    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909664    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909668    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909672    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 12:59:13.914949 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909680    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909684    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909689    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909693    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909698    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909703    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909709    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909713    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909717    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909722    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909725    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909731    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909736    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909740    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909745    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909750    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909754    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909758    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909762    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909766    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 12:59:13.915557 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909771    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909775    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909779    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909783    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909787    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909791    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909795    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909799    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909804    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909808    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909813    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909817    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909823    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909828    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909836    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909842    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909846    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909850    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909855    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 12:59:13.916068 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909860    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909864    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909868    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909872    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909876    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909881    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909885    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909889    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909893    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909898    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909902    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909907    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909911    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909915    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909919    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.909923    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910025    2567 flags.go:64] FLAG: --address="0.0.0.0"
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910081    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910092    2567 flags.go:64] FLAG: --anonymous-auth="true"
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910099    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910105    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 07 12:59:13.916585 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910111    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910118    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910125    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910130    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910137    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910144    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910149    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910154    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910160    2567 flags.go:64] FLAG: --cgroup-root=""
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910164    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910169    2567 flags.go:64] FLAG: --client-ca-file=""
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910175    2567 flags.go:64] FLAG: --cloud-provider="external"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910180    2567 flags.go:64] FLAG: --cluster-dns="[]"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910186    2567 flags.go:64] FLAG: --cluster-domain=""
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910191    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910196    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910201    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910206    2567 flags.go:64] FLAG: --container-log-max-files="5"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910212    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910217    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910222    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910227    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910232    2567 flags.go:64] FLAG: --contention-profiling="false"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910237    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 07 12:59:13.917150 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910242    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910247    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910252    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910258    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910263    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910267    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910272    2567 flags.go:64] FLAG: --enable-load-reader="false"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910277    2567 flags.go:64] FLAG: --enable-server="true"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910281    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910289    2567 flags.go:64] FLAG: --event-burst="100"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910294    2567 flags.go:64] FLAG: --event-qps="50"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910299    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910305    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910312    2567 flags.go:64] FLAG: --eviction-hard=""
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910318    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910323    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910328    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910333    2567 flags.go:64] FLAG: --eviction-soft=""
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910338    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910343    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910348    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910353    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910358    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910362    2567 flags.go:64] FLAG: --fail-swap-on="true"
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910367    2567 flags.go:64] FLAG: --feature-gates=""
Sep 07 12:59:13.917777 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910377    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910382    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910387    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910392    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910397    2567 flags.go:64] FLAG: --healthz-port="10248"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910402    2567 flags.go:64] FLAG: --help="false"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910406    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-128-114.ec2.internal"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910412    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910417    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910421    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910427    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910432    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910437    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910442    2567 flags.go:64] FLAG: --image-service-endpoint=""
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910446    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910451    2567 flags.go:64] FLAG: --kube-api-burst="100"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910456    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910461    2567 flags.go:64] FLAG: --kube-api-qps="50"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910466    2567 flags.go:64] FLAG: --kube-reserved=""
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910474    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910480    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910485    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910489    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910494    2567 flags.go:64] FLAG: --lock-file=""
Sep 07 12:59:13.918446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910499    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910504    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910509    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910517    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910523    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910528    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910533    2567 flags.go:64] FLAG: --logging-format="text"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910537    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910543    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910548    2567 flags.go:64] FLAG: --manifest-url=""
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910553    2567 flags.go:64] FLAG: --manifest-url-header=""
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910560    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910565    2567 flags.go:64] FLAG: --max-open-files="1000000"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910571    2567 flags.go:64] FLAG: --max-pods="110"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910575    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910580    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910585    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910590    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910596    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910601    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910606    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910634    2567 flags.go:64] FLAG: --node-status-max-images="50"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910639    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910644    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 07 12:59:13.919093 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910648    2567 flags.go:64] FLAG: --pod-cidr=""
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910653    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910662    2567 flags.go:64] FLAG: --pod-manifest-path=""
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910669    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910674    2567 flags.go:64] FLAG: --pods-per-core="0"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910680    2567 flags.go:64] FLAG: --port="10250"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910685    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910690    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07fae7e10a92f8946"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910695    2567 flags.go:64] FLAG: --qos-reserved=""
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910700    2567 flags.go:64] FLAG: --read-only-port="10255"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910705    2567 flags.go:64] FLAG: --register-node="true"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910710    2567 flags.go:64] FLAG: --register-with-taints=""
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910716    2567 flags.go:64] FLAG: --registry-burst="10"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910722    2567 flags.go:64] FLAG: --registry-qps="5"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910727    2567 flags.go:64] FLAG: --reserved-cpus=""
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910732    2567 flags.go:64] FLAG: --reserved-memory=""
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910739    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910744    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910749    2567 flags.go:64] FLAG: --rotate-certificates="false"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910754    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910758    2567 flags.go:64] FLAG: --runonce="false"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910763    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910768    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910773    2567 flags.go:64] FLAG: --seccomp-default="false"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910778    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 07 12:59:13.919703 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910783    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910788    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910793    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910797    2567 flags.go:64] FLAG: --storage-driver-password="root"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910802    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910807    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910811    2567 flags.go:64] FLAG: --storage-driver-user="root"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910816    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910821    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910826    2567 flags.go:64] FLAG: --system-cgroups=""
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910832    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910841    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910847    2567 flags.go:64] FLAG: --tls-cert-file=""
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910852    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910858    2567 flags.go:64] FLAG: --tls-min-version=""
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910862    2567 flags.go:64] FLAG: --tls-private-key-file=""
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910867    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910872    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910877    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910883    2567 flags.go:64] FLAG: --v="2"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910889    2567 flags.go:64] FLAG: --version="false"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910896    2567 flags.go:64] FLAG: --vmodule=""
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910902    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.910908    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911088    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 12:59:13.920340 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911095    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911101    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911105    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911110    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911114    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911118    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911123    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911127    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911132    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911136    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911140    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911144    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911148    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911153    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911157    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911162    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911166    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911172    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911176    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911182    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 12:59:13.920981 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911186    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911190    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911197    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911203    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911207    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911212    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911217    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911221    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911225    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911231    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911236    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911242    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911249    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911256    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911262    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911266    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911270    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911275    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911280    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 12:59:13.921524 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911285    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911289    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911293    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911297    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911301    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911305    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911310    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911314    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911318    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911322    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911328    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911332    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911338    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911342    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911346    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911350    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911354    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911358    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911362    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911367    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 12:59:13.922009 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911371    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911375    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911379    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911384    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911388    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911392    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911396    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911400    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911404    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911409    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911413    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911417    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911421    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911425    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911429    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911433    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911437    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911441    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911445    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 12:59:13.922504 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911449    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911453    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911457    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911463    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911467    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911474    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911478    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911482    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911486    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911490    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911495    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911499    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911503    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911507    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911512    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911516    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911520    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911525    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911529    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 12:59:13.922994 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.911533    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.912271    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.919901    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.919916    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.920749    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.920763    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920820    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920826    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920830    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920833    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920836    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920839    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920841    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920844    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920847    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 12:59:13.923467 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920850    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920853    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920855    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920858    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920861    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920863    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920866    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920868    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920871    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920874    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920876    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920879    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920882    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920884    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920886    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920890    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920894    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920896    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920899    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920901    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 12:59:13.923867 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920905    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920908    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920910    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920913    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920916    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920918    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920921    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920924    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920926    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920929    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920932    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920934    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920937    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920941    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920945    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920948    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920951    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920953    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920956    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 12:59:13.924344 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920959    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920961    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920964    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920966    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920969    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920972    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920974    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920977    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920980    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920983    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920986    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920988    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920991    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920995    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.920998    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921000    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921003    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921005    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921008    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921011    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 12:59:13.924874 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921013    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921016    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921018    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921021    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921025    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921028    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921032    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921035    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921037    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921040    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921043    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921045    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921048    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921050    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921052    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921055    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921057    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921061    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921065    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 12:59:13.925368 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921067    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921070    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921072    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921076    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921079    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921081    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921084    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921087    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921090    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921092    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921095    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921097    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.921102    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921223    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921227    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 12:59:13.925855 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921230    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921233    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921236    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921238    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921241    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921244    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921246    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921249    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921251    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921254    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921256    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921259    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921263    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921267    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921270    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921273    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921276    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921279    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921282    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 12:59:13.926224 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921284    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921287    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921290    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921293    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921297    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921299    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921302    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921304    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921307    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921309    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921312    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921315    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921317    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921320    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921323    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921325    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921327    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921330    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921333    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921335    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 12:59:13.926757 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921338    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921340    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921343    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921346    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921349    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921351    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921354    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921357    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921359    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921362    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921364    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921367    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921370    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921372    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921375    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921378    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921381    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921384    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921386    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 12:59:13.927247 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921389    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921392    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921395    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921397    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921399    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921402    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921405    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921407    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921410    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921412    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921415    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921418    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921420    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921423    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921425    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921428    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921431    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921434    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921437    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921439    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 12:59:13.927743 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921443    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921446    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921449    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921452    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921456    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921460    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921463    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921465    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921468    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921471    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921474    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921476    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921479    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921481    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921484    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921487    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921489    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921491    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 12:59:13.928230 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:13.921494    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 12:59:13.928746 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.921499    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 12:59:13.928746 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.921595    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 07 12:59:13.928746 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.923976    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 07 12:59:13.928746 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.925526    2567 server.go:1019] "Starting client certificate rotation"
Sep 07 12:59:13.928746 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.925638    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 12:59:13.928746 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.925668    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 12:59:13.928746 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.928509    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 12:59:13.931976 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.931958    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 12:59:13.944048 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.944029    2567 log.go:25] "Validated CRI v1 runtime API"
Sep 07 12:59:13.947075 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.947056    2567 log.go:25] "Validated CRI v1 image API"
Sep 07 12:59:13.947880 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.947865    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 07 12:59:13.951274 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.951244    2567 fs.go:135] Filesystem UUIDs: map[3f4b200c-ee48-4c7a-b87f-595029997584:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 95b42830-4aa0-4afc-b553-fac3f6499386:/dev/nvme0n1p4]
Sep 07 12:59:13.951356 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.951272    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 07 12:59:13.956588 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.956461    2567 manager.go:216] Machine: {Timestamp:2026-09-07 12:59:13.955013373 +0000 UTC m=+0.361757362 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3081224 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c3c21dc2311f9ecf2eb8b720311c4 SystemUUID:ec2c3c21-dc23-11f9-ecf2-eb8b720311c4 BootID:8a4def7d-47c2-4b1b-b150-e04892cec1d0 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6c:9f:79:1a:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6c:9f:79:1a:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:8b:bc:63:2a:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 07 12:59:13.956588 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.956577    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 07 12:59:13.956757 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.956702    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 07 12:59:13.960022 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.959993    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 07 12:59:13.960187 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.960026    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-114.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 07 12:59:13.960488 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.960473    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 12:59:13.960921 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.960907    2567 topology_manager.go:138] "Creating topology manager with none policy"
Sep 07 12:59:13.960986 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.960928    2567 container_manager_linux.go:309] "Creating device plugin manager"
Sep 07 12:59:13.960986 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.960940    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 12:59:13.961835 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.961823    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 12:59:13.963749 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.963736    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 07 12:59:13.968887 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.968873    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 12:59:13.969027 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.969016    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 07 12:59:13.969084 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.969062    2567 kubelet.go:486] "Attempting to sync node with API server"
Sep 07 12:59:13.969084 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.969081    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 07 12:59:13.969175 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.969101    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 07 12:59:13.969175 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.969114    2567 kubelet.go:398] "Adding apiserver pod source"
Sep 07 12:59:13.969175 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.969127    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 07 12:59:13.969525 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.969503    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 12:59:13.969598 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.969526    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 12:59:13.972539 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.972523    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 07 12:59:13.975438 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.975424    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 07 12:59:13.975500 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.975454    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 07 12:59:13.975624 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.975598    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 07 12:59:13.975664 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.975633    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 07 12:59:13.975664 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.975639    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 07 12:59:13.975664 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.975645    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 07 12:59:13.975664 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.975653    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 07 12:59:13.975664 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.975658    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 07 12:59:13.976293 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.976285    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 07 12:59:13.976293 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.976293    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 07 12:59:13.976353 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.976300    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 07 12:59:13.976353 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.976305    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 07 12:59:13.976353 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.976314    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 07 12:59:13.976353 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.976321    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 07 12:59:13.977187 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.977177    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 07 12:59:13.977187 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.977187    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 07 12:59:13.980938 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.980915    2567 server.go:1268] "Started kubelet"
Sep 07 12:59:13.981051 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.980983    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 07 12:59:13.981112 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.981051    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 07 12:59:13.981165 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.981152    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 07 12:59:13.981853 ip-10-0-128-114 systemd[1]: Started Kubernetes Kubelet.
Sep 07 12:59:13.983185 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.983169    2567 server.go:310] "Adding debug handlers to kubelet server"
Sep 07 12:59:13.987416 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.987397    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 07 12:59:13.988011 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.987982    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-114.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 07 12:59:13.988104 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:13.988042    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-114.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 07 12:59:13.988489 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:13.988451    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 07 12:59:13.989840 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:13.989818    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 07 12:59:13.991453 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:13.990360    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-114.ec2.internal.18d30b59063d396f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-114.ec2.internal,UID:ip-10-0-128-114.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-114.ec2.internal,},FirstTimestamp:2026-09-07 12:59:13.980889455 +0000 UTC m=+0.387633423,LastTimestamp:2026-09-07 12:59:13.980889455 +0000 UTC m=+0.387633423,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-114.ec2.internal,}"
Sep 07 12:59:13.993328 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.993304    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 07 12:59:13.993328 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.993323    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 07 12:59:13.993493 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:13.993463    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:13.994130 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.994108    2567 factory.go:55] Registering systemd factory
Sep 07 12:59:13.994130 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.994128    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 07 12:59:13.994243 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.994131    2567 factory.go:223] Registration of the systemd container factory successfully
Sep 07 12:59:13.994243 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.994146    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 07 12:59:13.994243 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.994151    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 07 12:59:13.994375 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.994260    2567 reconstruct.go:97] "Volume reconstruction finished"
Sep 07 12:59:13.994375 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.994268    2567 reconciler.go:29] "Reconciler: start to sync state"
Sep 07 12:59:13.994375 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.994349    2567 factory.go:153] Registering CRI-O factory
Sep 07 12:59:13.994375 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.994364    2567 factory.go:223] Registration of the crio container factory successfully
Sep 07 12:59:13.994535 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.994417    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 07 12:59:13.994535 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.994436    2567 factory.go:103] Registering Raw factory
Sep 07 12:59:13.994535 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.994453    2567 manager.go:1196] Started watching for new ooms in manager
Sep 07 12:59:13.995507 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:13.995489    2567 manager.go:349] Starting recovery of all containers
Sep 07 12:59:14.004579 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.004539    2567 manager.go:354] Recovery completed
Sep 07 12:59:14.006230 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.006087    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-114.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 07 12:59:14.006230 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.006133    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 07 12:59:14.008846 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.008832    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:59:14.010311 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.010291    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:59:14.010368 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.010320    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:59:14.010368 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.010330    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:59:14.011075 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.011057    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 07 12:59:14.011075 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.011071    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 07 12:59:14.011181 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.011089    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 12:59:14.011181 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.011094    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:59:14.011836 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.011822    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:59:14.011888 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.011848    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:59:14.011888 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.011859    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:59:14.013082 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.013070    2567 policy_none.go:49] "None policy: Start"
Sep 07 12:59:14.013126 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.013085    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 07 12:59:14.013126 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.013095    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 07 12:59:14.014889 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.014878    2567 policy_none.go:47] "Start"
Sep 07 12:59:14.025810 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.025742    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-114.ec2.internal.18d30b5907fe2c1c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-114.ec2.internal,UID:ip-10-0-128-114.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-114.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-114.ec2.internal,},FirstTimestamp:2026-09-07 12:59:14.010311708 +0000 UTC m=+0.417055675,LastTimestamp:2026-09-07 12:59:14.010311708 +0000 UTC m=+0.417055675,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-114.ec2.internal,}"
Sep 07 12:59:14.039695 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.038358    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-114.ec2.internal.18d30b5907fe5f18  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-114.ec2.internal,UID:ip-10-0-128-114.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-114.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-114.ec2.internal,},FirstTimestamp:2026-09-07 12:59:14.01032476 +0000 UTC m=+0.417068731,LastTimestamp:2026-09-07 12:59:14.01032476 +0000 UTC m=+0.417068731,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-114.ec2.internal,}"
Sep 07 12:59:14.041272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.041255    2567 manager.go:337] "Starting Device Plugin manager"
Sep 07 12:59:14.041358 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.041298    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 07 12:59:14.041358 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.041312    2567 server.go:87] "Starting device plugin registration server"
Sep 07 12:59:14.041696 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.041681    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 07 12:59:14.069273 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.041697    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 07 12:59:14.069273 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.041860    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 07 12:59:14.069273 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.041927    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 07 12:59:14.069273 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.041936    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 07 12:59:14.069273 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.042570    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 07 12:59:14.069273 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.042668    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:14.069273 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.049810    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-114.ec2.internal.18d30b5907fe80e7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-114.ec2.internal,UID:ip-10-0-128-114.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-114.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-114.ec2.internal,},FirstTimestamp:2026-09-07 12:59:14.010333415 +0000 UTC m=+0.417077382,LastTimestamp:2026-09-07 12:59:14.010333415 +0000 UTC m=+0.417077382,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-114.ec2.internal,}"
Sep 07 12:59:14.069273 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.060004    2567 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-114.ec2.internal.18d30b5907fe2c1c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-114.ec2.internal.18d30b5907fe2c1c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-114.ec2.internal,UID:ip-10-0-128-114.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-114.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-114.ec2.internal,},FirstTimestamp:2026-09-07 12:59:14.010311708 +0000 UTC m=+0.417055675,LastTimestamp:2026-09-07 12:59:14.011837549 +0000 UTC m=+0.418581520,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-114.ec2.internal,}"
Sep 07 12:59:14.072780 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.072716    2567 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-114.ec2.internal.18d30b5907fe5f18\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-114.ec2.internal.18d30b5907fe5f18  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-114.ec2.internal,UID:ip-10-0-128-114.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-114.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-114.ec2.internal,},FirstTimestamp:2026-09-07 12:59:14.01032476 +0000 UTC m=+0.417068731,LastTimestamp:2026-09-07 12:59:14.011853952 +0000 UTC m=+0.418597920,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-114.ec2.internal,}"
Sep 07 12:59:14.083276 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.083216    2567 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-114.ec2.internal.18d30b5907fe80e7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-114.ec2.internal.18d30b5907fe80e7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-114.ec2.internal,UID:ip-10-0-128-114.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-114.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-114.ec2.internal,},FirstTimestamp:2026-09-07 12:59:14.010333415 +0000 UTC m=+0.417077382,LastTimestamp:2026-09-07 12:59:14.011862472 +0000 UTC m=+0.418606441,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-114.ec2.internal,}"
Sep 07 12:59:14.087364 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.087342    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2p845"
Sep 07 12:59:14.088974 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.088950    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 07 12:59:14.090143 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.090129    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 07 12:59:14.090219 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.090183    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 07 12:59:14.090219 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.090208    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 07 12:59:14.090314 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.090299    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 07 12:59:14.094329 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.094300    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 07 12:59:14.094414 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.094352    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-114.ec2.internal.18d30b590a04043c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-114.ec2.internal,UID:ip-10-0-128-114.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-128-114.ec2.internal,},FirstTimestamp:2026-09-07 12:59:14.044249148 +0000 UTC m=+0.450993107,LastTimestamp:2026-09-07 12:59:14.044249148 +0000 UTC m=+0.450993107,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-114.ec2.internal,}"
Sep 07 12:59:14.094814 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.094796    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2p845"
Sep 07 12:59:14.142232 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.142208    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:59:14.143131 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.143114    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:59:14.143226 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.143149    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:59:14.143226 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.143163    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:59:14.143226 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.143192    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.156473 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.156407    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.156473 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.156429    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-114.ec2.internal\": node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:14.191141 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.191117    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-114.ec2.internal"]
Sep 07 12:59:14.191236 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.191187    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:59:14.191489 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.191468    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:14.192106 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.192092    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:59:14.192170 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.192122    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:59:14.192170 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.192138    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:59:14.194421 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.194403    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/121d79ea4b4f92683e7694220da50ad1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal\" (UID: \"121d79ea4b4f92683e7694220da50ad1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.194472 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.194432    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/121d79ea4b4f92683e7694220da50ad1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal\" (UID: \"121d79ea4b4f92683e7694220da50ad1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.194514 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.194501    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:59:14.194621 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.194595    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.194667 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.194641    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:59:14.195267 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.195252    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:59:14.195330 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.195280    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:59:14.195330 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.195292    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:59:14.196370 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.196355    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:59:14.196437 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.196380    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:59:14.196437 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.196393    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:59:14.198457 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.198443    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.198524 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.198467    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:59:14.199151 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.199136    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:59:14.199213 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.199166    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:59:14.199213 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.199180    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:59:14.218228 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.218210    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-114.ec2.internal\" not found" node="ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.222459 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.222444    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-114.ec2.internal\" not found" node="ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.292428 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.292393    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:14.295394 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.295374    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/121d79ea4b4f92683e7694220da50ad1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal\" (UID: \"121d79ea4b4f92683e7694220da50ad1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.295484 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.295406    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/121d79ea4b4f92683e7694220da50ad1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal\" (UID: \"121d79ea4b4f92683e7694220da50ad1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.295484 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.295434    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/68fb4e41d01603c60ad3ed3f0a6da717-config\") pod \"kube-apiserver-proxy-ip-10-0-128-114.ec2.internal\" (UID: \"68fb4e41d01603c60ad3ed3f0a6da717\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.295555 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.295498    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/121d79ea4b4f92683e7694220da50ad1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal\" (UID: \"121d79ea4b4f92683e7694220da50ad1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.295555 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.295502    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/121d79ea4b4f92683e7694220da50ad1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal\" (UID: \"121d79ea4b4f92683e7694220da50ad1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.393296 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.393270    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:14.396496 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.396483    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/68fb4e41d01603c60ad3ed3f0a6da717-config\") pod \"kube-apiserver-proxy-ip-10-0-128-114.ec2.internal\" (UID: \"68fb4e41d01603c60ad3ed3f0a6da717\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.396545 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.396535    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/68fb4e41d01603c60ad3ed3f0a6da717-config\") pod \"kube-apiserver-proxy-ip-10-0-128-114.ec2.internal\" (UID: \"68fb4e41d01603c60ad3ed3f0a6da717\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.493650 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.493556    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:14.520950 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.520930    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.524704 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:14.524686    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:14.594377 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:14.594342    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:16.673227 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:16.673199    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:16.773660 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:16.773631    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:16.874269 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:16.874234    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:16.903548 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:16.903518    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 07 12:59:16.974673 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:16.974638    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:17.074842 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:17.074814    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:17.075953 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.075905    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 12:54:16 +0000 UTC" deadline="2028-03-10 17:41:52.836743715 +0000 UTC"
Sep 07 12:59:17.075953 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.075952    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13204h42m35.760797137s"
Sep 07 12:59:17.175461 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:17.175430    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:17.194434 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:17.194405    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fb4e41d01603c60ad3ed3f0a6da717.slice/crio-24c045087ee7095a24a06e00875436ed922694fd3d3f5f28e91f7d91147942ad WatchSource:0}: Error finding container 24c045087ee7095a24a06e00875436ed922694fd3d3f5f28e91f7d91147942ad: Status 404 returned error can't find the container with id 24c045087ee7095a24a06e00875436ed922694fd3d3f5f28e91f7d91147942ad
Sep 07 12:59:17.195423 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:17.195399    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121d79ea4b4f92683e7694220da50ad1.slice/crio-76386fdd82dd7e3f206bde44bea2cac366ba4632feb70cb90fadde1066fa7c99 WatchSource:0}: Error finding container 76386fdd82dd7e3f206bde44bea2cac366ba4632feb70cb90fadde1066fa7c99: Status 404 returned error can't find the container with id 76386fdd82dd7e3f206bde44bea2cac366ba4632feb70cb90fadde1066fa7c99
Sep 07 12:59:17.198589 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.198569    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 12:59:17.200235 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.200217    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:59:17.201043 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.201026    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:59:17.201137 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.201054    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:59:17.201137 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.201082    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:59:17.275856 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:17.275825    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:17.280961 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.280941    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 12:59:17.376909 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:17.376838    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:17.440108 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.440085    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 12:59:17.477361 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:17.477325    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-114.ec2.internal\" not found"
Sep 07 12:59:17.519679 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.519657    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 12:59:17.572542 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.572512    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:17.588395 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.588368    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 12:59:17.592340 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.592324    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 12:59:17.593154 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.593138    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-114.ec2.internal"
Sep 07 12:59:17.604414 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.604396    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 12:59:17.948692 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.948660    2567 apiserver.go:52] "Watching apiserver"
Sep 07 12:59:17.958889 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.958856    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 07 12:59:17.959369 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.959339    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6","openshift-image-registry/node-ca-kdkj7","openshift-multus/multus-additional-cni-plugins-dtj6g","openshift-multus/network-metrics-daemon-tsqvk","openshift-network-diagnostics/network-check-target-p7mj7","openshift-cluster-node-tuning-operator/tuned-z98gm","openshift-dns/node-resolver-sqnx4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal","openshift-multus/multus-6xmwx","openshift-network-operator/iptables-alerter-2hsq7","openshift-ovn-kubernetes/ovnkube-node-4vwcx","kube-system/global-pull-secret-syncer-njk59","kube-system/konnectivity-agent-fkppx","kube-system/kube-apiserver-proxy-ip-10-0-128-114.ec2.internal"]
Sep 07 12:59:17.961896 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.961867    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sqnx4"
Sep 07 12:59:17.963970 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.963947    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fkppx"
Sep 07 12:59:17.965455 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.965439    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 07 12:59:17.965560 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.965505    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 07 12:59:17.965560 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.965551    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kgmtr\""
Sep 07 12:59:17.966247 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.966222    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:17.968442 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.968418    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:17.968541 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:17.968495    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:17.969189 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.968738    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 07 12:59:17.969189 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.968774    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 07 12:59:17.969189 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.968743    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-df9ht\""
Sep 07 12:59:17.969189 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.968922    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 07 12:59:17.969189 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.968929    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 07 12:59:17.969189 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.969015    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 07 12:59:17.969189 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.969090    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dthxs\""
Sep 07 12:59:17.969189 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.969170    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 07 12:59:17.969189 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.969191    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 07 12:59:17.970684 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.970667    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:17.970762 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:17.970735    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:17.972467 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.972446    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 07 12:59:17.972898 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.972878    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.975164 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.975142    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.976577 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.976561    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 07 12:59:17.976677 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.976646    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 07 12:59:17.977474 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.977455    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:17.978170 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.978044    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 07 12:59:17.978170 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.978140    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-td2kc\""
Sep 07 12:59:17.978170 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.978168    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s2pjd\""
Sep 07 12:59:17.979669 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.979653    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kdkj7"
Sep 07 12:59:17.981366 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.981347    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 07 12:59:17.981461 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.981354    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 07 12:59:17.981509 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.981483    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h7547\""
Sep 07 12:59:17.981921 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.981902    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 07 12:59:17.981921 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.981914    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2hsq7"
Sep 07 12:59:17.982234 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982215    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-var-lib-kubelet\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.982334 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982239    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-sys\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.982334 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982257    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-lib-modules\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.982334 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982270    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-var-lib-kubelet\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.982334 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982284    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-host\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.982334 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982326    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-run-k8s-cni-cncf-io\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.982577 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982350    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-etc-kubernetes\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.982577 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982367    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9t8\" (UniqueName: \"kubernetes.io/projected/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-kube-api-access-ft9t8\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:17.982577 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982382    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-run\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.982577 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982402    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a700e69-d961-4293-81be-98b17240902b-tmp\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.982577 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982417    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47b9aa4a-12ea-4917-94b3-eef425da8b28-multus-daemon-config\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.982577 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982431    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-run-multus-certs\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.982577 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982447    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4a700e69-d961-4293-81be-98b17240902b-etc-tuned\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.982577 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982468    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-sysconfig\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.982577 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982502    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-systemd\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.982577 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982541    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcgvj\" (UniqueName: \"kubernetes.io/projected/4a700e69-d961-4293-81be-98b17240902b-kube-api-access-jcgvj\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.982577 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982564    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-cnibin\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982584    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-var-lib-cni-multus\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982605    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-hostroot\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982624    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-multus-conf-dir\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982643    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47b9aa4a-12ea-4917-94b3-eef425da8b28-cni-binary-copy\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982666    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-run-netns\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982686    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa91b78f-8f6e-41e5-83df-322e9825cec5-agent-certs\") pod \"konnectivity-agent-fkppx\" (UID: \"fa91b78f-8f6e-41e5-83df-322e9825cec5\") " pod="kube-system/konnectivity-agent-fkppx"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982707    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/813ca344-4b86-40c6-b18c-f30a83c23504-system-cni-dir\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982731    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/813ca344-4b86-40c6-b18c-f30a83c23504-cnibin\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982753    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/813ca344-4b86-40c6-b18c-f30a83c23504-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982804    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1b14f3-6875-4d07-859d-ada2d3bb1f0f-tmp-dir\") pod \"node-resolver-sqnx4\" (UID: \"8f1b14f3-6875-4d07-859d-ada2d3bb1f0f\") " pod="openshift-dns/node-resolver-sqnx4"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982823    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkqzq\" (UniqueName: \"kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq\") pod \"network-check-target-p7mj7\" (UID: \"a2da8de4-8df5-4b96-9449-1c8d346c5fec\") " pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982836    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-multus-cni-dir\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982850    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-multus-socket-dir-parent\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982886    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa91b78f-8f6e-41e5-83df-322e9825cec5-konnectivity-ca\") pod \"konnectivity-agent-fkppx\" (UID: \"fa91b78f-8f6e-41e5-83df-322e9825cec5\") " pod="kube-system/konnectivity-agent-fkppx"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982925    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/813ca344-4b86-40c6-b18c-f30a83c23504-cni-binary-copy\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:17.983111 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982946    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j85gx\" (UniqueName: \"kubernetes.io/projected/813ca344-4b86-40c6-b18c-f30a83c23504-kube-api-access-j85gx\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.982975    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx7gp\" (UniqueName: \"kubernetes.io/projected/8f1b14f3-6875-4d07-859d-ada2d3bb1f0f-kube-api-access-dx7gp\") pod \"node-resolver-sqnx4\" (UID: \"8f1b14f3-6875-4d07-859d-ada2d3bb1f0f\") " pod="openshift-dns/node-resolver-sqnx4"
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983023    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-modprobe-d\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983052    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-sysctl-d\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983099    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983103    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983106    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-system-cni-dir\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983197    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6fmg\" (UniqueName: \"kubernetes.io/projected/47b9aa4a-12ea-4917-94b3-eef425da8b28-kube-api-access-m6fmg\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983245    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/813ca344-4b86-40c6-b18c-f30a83c23504-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983267    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/813ca344-4b86-40c6-b18c-f30a83c23504-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983289    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983311    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f1b14f3-6875-4d07-859d-ada2d3bb1f0f-hosts-file\") pod \"node-resolver-sqnx4\" (UID: \"8f1b14f3-6875-4d07-859d-ada2d3bb1f0f\") " pod="openshift-dns/node-resolver-sqnx4"
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983358    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-kubernetes\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983362    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983376    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-sysctl-conf\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983394    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-var-lib-cni-bin\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:17.983812 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.983457    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lbqzv\""
Sep 07 12:59:17.984621 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.984603    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:17.986365 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.985684    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 07 12:59:17.986365 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.985758    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 07 12:59:17.986611 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.986504    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 07 12:59:17.986895 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.986877    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:17.986974 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:17.986950    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:17.987592 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.987573    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6trkj\""
Sep 07 12:59:17.988402 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.988382    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 07 12:59:17.988488 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.988469    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 07 12:59:17.988543 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.988524    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 07 12:59:17.988594 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.988524    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 07 12:59:17.988660 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.988640    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 07 12:59:17.989097 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.989081    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 07 12:59:17.989416 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.989395    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p9c22\""
Sep 07 12:59:17.994321 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:17.994303    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 12:59:18.069445 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.069411    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x8xkb"
Sep 07 12:59:18.073385 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.073358    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 07 12:59:18.074031 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.073971    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal" event={"ID":"121d79ea4b4f92683e7694220da50ad1","Type":"ContainerStarted","Data":"76386fdd82dd7e3f206bde44bea2cac366ba4632feb70cb90fadde1066fa7c99"}
Sep 07 12:59:18.075317 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.075292    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-114.ec2.internal" event={"ID":"68fb4e41d01603c60ad3ed3f0a6da717","Type":"ContainerStarted","Data":"24c045087ee7095a24a06e00875436ed922694fd3d3f5f28e91f7d91147942ad"}
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084109    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x8xkb"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084155    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-var-lib-cni-bin\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084191    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-lib-modules\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084217    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-run-systemd\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084235    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-var-lib-cni-bin\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084243    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-kubelet-config\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084273    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-dbus\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084298    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-run-k8s-cni-cncf-io\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084320    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9t8\" (UniqueName: \"kubernetes.io/projected/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-kube-api-access-ft9t8\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084341    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-run-multus-certs\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084364    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4a700e69-d961-4293-81be-98b17240902b-etc-tuned\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084376    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-lib-modules\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084379    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-run-k8s-cni-cncf-io\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084387    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-slash\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084416    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-run-multus-certs\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084424    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-var-lib-openvswitch\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.084439 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084451    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25a5233b-4688-4609-abb4-94ad4eeecd29-iptables-alerter-script\") pod \"iptables-alerter-2hsq7\" (UID: \"25a5233b-4688-4609-abb4-94ad4eeecd29\") " pod="openshift-network-operator/iptables-alerter-2hsq7"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084474    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-node-log\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084491    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-ovnkube-script-lib\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084510    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-cnibin\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084526    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-var-lib-cni-multus\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084546    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-env-overrides\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084568    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47b9aa4a-12ea-4917-94b3-eef425da8b28-cni-binary-copy\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084614    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa91b78f-8f6e-41e5-83df-322e9825cec5-agent-certs\") pod \"konnectivity-agent-fkppx\" (UID: \"fa91b78f-8f6e-41e5-83df-322e9825cec5\") " pod="kube-system/konnectivity-agent-fkppx"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084632    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-cnibin\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084695    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-var-lib-cni-multus\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/813ca344-4b86-40c6-b18c-f30a83c23504-system-cni-dir\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084808    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/813ca344-4b86-40c6-b18c-f30a83c23504-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084815    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/813ca344-4b86-40c6-b18c-f30a83c23504-system-cni-dir\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084846    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1b14f3-6875-4d07-859d-ada2d3bb1f0f-tmp-dir\") pod \"node-resolver-sqnx4\" (UID: \"8f1b14f3-6875-4d07-859d-ada2d3bb1f0f\") " pod="openshift-dns/node-resolver-sqnx4"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084873    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqzq\" (UniqueName: \"kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq\") pod \"network-check-target-p7mj7\" (UID: \"a2da8de4-8df5-4b96-9449-1c8d346c5fec\") " pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084897    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-host\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084920    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-multus-socket-dir-parent\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.085224 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084943    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa91b78f-8f6e-41e5-83df-322e9825cec5-konnectivity-ca\") pod \"konnectivity-agent-fkppx\" (UID: \"fa91b78f-8f6e-41e5-83df-322e9825cec5\") " pod="kube-system/konnectivity-agent-fkppx"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.084965    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/813ca344-4b86-40c6-b18c-f30a83c23504-cni-binary-copy\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085032    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx7gp\" (UniqueName: \"kubernetes.io/projected/8f1b14f3-6875-4d07-859d-ada2d3bb1f0f-kube-api-access-dx7gp\") pod \"node-resolver-sqnx4\" (UID: \"8f1b14f3-6875-4d07-859d-ada2d3bb1f0f\") " pod="openshift-dns/node-resolver-sqnx4"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085079    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-sysctl-conf\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085107    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-log-socket\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085151    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hb9g\" (UniqueName: \"kubernetes.io/projected/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-kube-api-access-2hb9g\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085193    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1b14f3-6875-4d07-859d-ada2d3bb1f0f-tmp-dir\") pod \"node-resolver-sqnx4\" (UID: \"8f1b14f3-6875-4d07-859d-ada2d3bb1f0f\") " pod="openshift-dns/node-resolver-sqnx4"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085266    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47b9aa4a-12ea-4917-94b3-eef425da8b28-cni-binary-copy\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085318    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/813ca344-4b86-40c6-b18c-f30a83c23504-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085348    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-registration-dir\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085356    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-multus-socket-dir-parent\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085376    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-var-lib-kubelet\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085402    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-sys\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085427    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-var-lib-kubelet\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085426    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/813ca344-4b86-40c6-b18c-f30a83c23504-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085480    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085498    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-var-lib-kubelet\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.086018 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085539    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-var-lib-kubelet\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085560    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-systemd-units\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085576    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-cni-netd\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085599    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-sysctl-conf\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085612    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea2d8b03-e721-4102-846e-ee7d41d13e96-serviceca\") pod \"node-ca-kdkj7\" (UID: \"ea2d8b03-e721-4102-846e-ee7d41d13e96\") " pod="openshift-image-registry/node-ca-kdkj7"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085665    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/813ca344-4b86-40c6-b18c-f30a83c23504-cni-binary-copy\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085674    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-host\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085699    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-sys\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085728    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-etc-kubernetes\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085757    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-run\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085764    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-etc-kubernetes\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085781    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a700e69-d961-4293-81be-98b17240902b-tmp\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085809    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dgtp\" (UniqueName: \"kubernetes.io/projected/6b681468-002e-4500-b222-49da180d381c-kube-api-access-6dgtp\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085833    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-run-netns\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085856    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-run-openvswitch\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085881    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47b9aa4a-12ea-4917-94b3-eef425da8b28-multus-daemon-config\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085907    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-ovnkube-config\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.086737 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085930    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea2d8b03-e721-4102-846e-ee7d41d13e96-host\") pod \"node-ca-kdkj7\" (UID: \"ea2d8b03-e721-4102-846e-ee7d41d13e96\") " pod="openshift-image-registry/node-ca-kdkj7"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085954    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25a5233b-4688-4609-abb4-94ad4eeecd29-host-slash\") pod \"iptables-alerter-2hsq7\" (UID: \"25a5233b-4688-4609-abb4-94ad4eeecd29\") " pod="openshift-network-operator/iptables-alerter-2hsq7"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.085978    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5pv8\" (UniqueName: \"kubernetes.io/projected/25a5233b-4688-4609-abb4-94ad4eeecd29-kube-api-access-s5pv8\") pod \"iptables-alerter-2hsq7\" (UID: \"25a5233b-4688-4609-abb4-94ad4eeecd29\") " pod="openshift-network-operator/iptables-alerter-2hsq7"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086004    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-sysconfig\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086008    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/813ca344-4b86-40c6-b18c-f30a83c23504-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086083    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-sysconfig\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086124    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-run\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086310    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa91b78f-8f6e-41e5-83df-322e9825cec5-konnectivity-ca\") pod \"konnectivity-agent-fkppx\" (UID: \"fa91b78f-8f6e-41e5-83df-322e9825cec5\") " pod="kube-system/konnectivity-agent-fkppx"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086363    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-systemd\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086401    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgvj\" (UniqueName: \"kubernetes.io/projected/4a700e69-d961-4293-81be-98b17240902b-kube-api-access-jcgvj\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086428    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-kubelet\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086447    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-hostroot\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086460    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47b9aa4a-12ea-4917-94b3-eef425da8b28-multus-daemon-config\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086503    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-multus-conf-dir\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086533    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086558    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-etc-selinux\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086563    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-hostroot\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.087582 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086582    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-run-ovn\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086535    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-systemd\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086627    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-run-netns\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086636    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-multus-conf-dir\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086650    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/813ca344-4b86-40c6-b18c-f30a83c23504-cnibin\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086678    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-modprobe-d\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086693    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-cni-bin\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086696    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/813ca344-4b86-40c6-b18c-f30a83c23504-cnibin\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086742    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-host-run-netns\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086765    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-modprobe-d\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086786    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086831    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086874    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmbgl\" (UniqueName: \"kubernetes.io/projected/ea2d8b03-e721-4102-846e-ee7d41d13e96-kube-api-access-pmbgl\") pod \"node-ca-kdkj7\" (UID: \"ea2d8b03-e721-4102-846e-ee7d41d13e96\") " pod="openshift-image-registry/node-ca-kdkj7"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086929    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-socket-dir\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.086962    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-multus-cni-dir\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087021    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j85gx\" (UniqueName: \"kubernetes.io/projected/813ca344-4b86-40c6-b18c-f30a83c23504-kube-api-access-j85gx\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087077    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-sysctl-d\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.088398 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087088    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-multus-cni-dir\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087201    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-etc-openvswitch\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087240    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-sysctl-d\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087280    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087311    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-device-dir\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087342    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-system-cni-dir\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087367    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6fmg\" (UniqueName: \"kubernetes.io/projected/47b9aa4a-12ea-4917-94b3-eef425da8b28-kube-api-access-m6fmg\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087392    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/813ca344-4b86-40c6-b18c-f30a83c23504-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087547    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/813ca344-4b86-40c6-b18c-f30a83c23504-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087561    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47b9aa4a-12ea-4917-94b3-eef425da8b28-system-cni-dir\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087582    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087609    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f1b14f3-6875-4d07-859d-ada2d3bb1f0f-hosts-file\") pod \"node-resolver-sqnx4\" (UID: \"8f1b14f3-6875-4d07-859d-ada2d3bb1f0f\") " pod="openshift-dns/node-resolver-sqnx4"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087632    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-kubernetes\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.087656    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087657    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-ovn-node-metrics-cert\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.087720    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs podName:f79dcf5c-ddc9-47c8-b61e-96fd4401c89f nodeName:}" failed. No retries permitted until 2026-09-07 12:59:18.587699579 +0000 UTC m=+3.015997854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs") pod "network-metrics-daemon-tsqvk" (UID: "f79dcf5c-ddc9-47c8-b61e-96fd4401c89f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087714    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f1b14f3-6875-4d07-859d-ada2d3bb1f0f-hosts-file\") pod \"node-resolver-sqnx4\" (UID: \"8f1b14f3-6875-4d07-859d-ada2d3bb1f0f\") " pod="openshift-dns/node-resolver-sqnx4"
Sep 07 12:59:18.089259 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087777    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-sys-fs\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.089988 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.087778    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a700e69-d961-4293-81be-98b17240902b-etc-kubernetes\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.089988 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.089406    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4a700e69-d961-4293-81be-98b17240902b-etc-tuned\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.089988 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.089695    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a700e69-d961-4293-81be-98b17240902b-tmp\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.090131 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.090013    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa91b78f-8f6e-41e5-83df-322e9825cec5-agent-certs\") pod \"konnectivity-agent-fkppx\" (UID: \"fa91b78f-8f6e-41e5-83df-322e9825cec5\") " pod="kube-system/konnectivity-agent-fkppx"
Sep 07 12:59:18.106102 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.106071    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx7gp\" (UniqueName: \"kubernetes.io/projected/8f1b14f3-6875-4d07-859d-ada2d3bb1f0f-kube-api-access-dx7gp\") pod \"node-resolver-sqnx4\" (UID: \"8f1b14f3-6875-4d07-859d-ada2d3bb1f0f\") " pod="openshift-dns/node-resolver-sqnx4"
Sep 07 12:59:18.107031 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.106997    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6fmg\" (UniqueName: \"kubernetes.io/projected/47b9aa4a-12ea-4917-94b3-eef425da8b28-kube-api-access-m6fmg\") pod \"multus-6xmwx\" (UID: \"47b9aa4a-12ea-4917-94b3-eef425da8b28\") " pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.109147 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.109119    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgvj\" (UniqueName: \"kubernetes.io/projected/4a700e69-d961-4293-81be-98b17240902b-kube-api-access-jcgvj\") pod \"tuned-z98gm\" (UID: \"4a700e69-d961-4293-81be-98b17240902b\") " pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.110524 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.110499    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:59:18.110613 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.110530    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:59:18.110613 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.110547    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dkqzq for pod openshift-network-diagnostics/network-check-target-p7mj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:18.110723 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.110614    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq podName:a2da8de4-8df5-4b96-9449-1c8d346c5fec nodeName:}" failed. No retries permitted until 2026-09-07 12:59:18.610594043 +0000 UTC m=+3.038892316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dkqzq" (UniqueName: "kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq") pod "network-check-target-p7mj7" (UID: "a2da8de4-8df5-4b96-9449-1c8d346c5fec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:18.111310 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.111283    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j85gx\" (UniqueName: \"kubernetes.io/projected/813ca344-4b86-40c6-b18c-f30a83c23504-kube-api-access-j85gx\") pod \"multus-additional-cni-plugins-dtj6g\" (UID: \"813ca344-4b86-40c6-b18c-f30a83c23504\") " pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.113166 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.113143    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9t8\" (UniqueName: \"kubernetes.io/projected/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-kube-api-access-ft9t8\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:18.188090 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188048    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-log-socket\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188090 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188092    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hb9g\" (UniqueName: \"kubernetes.io/projected/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-kube-api-access-2hb9g\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188337 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188117    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-registration-dir\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.188337 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188138    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-systemd-units\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188337 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188160    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-cni-netd\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188337 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188185    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea2d8b03-e721-4102-846e-ee7d41d13e96-serviceca\") pod \"node-ca-kdkj7\" (UID: \"ea2d8b03-e721-4102-846e-ee7d41d13e96\") " pod="openshift-image-registry/node-ca-kdkj7"
Sep 07 12:59:18.188337 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188201    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-log-socket\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188337 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188221    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dgtp\" (UniqueName: \"kubernetes.io/projected/6b681468-002e-4500-b222-49da180d381c-kube-api-access-6dgtp\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.188337 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188221    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-registration-dir\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.188337 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188264    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-cni-netd\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188337 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188306    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-systemd-units\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188406    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-run-netns\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188433    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-run-openvswitch\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188483    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-ovnkube-config\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188510    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea2d8b03-e721-4102-846e-ee7d41d13e96-host\") pod \"node-ca-kdkj7\" (UID: \"ea2d8b03-e721-4102-846e-ee7d41d13e96\") " pod="openshift-image-registry/node-ca-kdkj7"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188513    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-run-netns\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188487    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-run-openvswitch\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188548    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25a5233b-4688-4609-abb4-94ad4eeecd29-host-slash\") pod \"iptables-alerter-2hsq7\" (UID: \"25a5233b-4688-4609-abb4-94ad4eeecd29\") " pod="openshift-network-operator/iptables-alerter-2hsq7"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188588    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea2d8b03-e721-4102-846e-ee7d41d13e96-serviceca\") pod \"node-ca-kdkj7\" (UID: \"ea2d8b03-e721-4102-846e-ee7d41d13e96\") " pod="openshift-image-registry/node-ca-kdkj7"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188597    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25a5233b-4688-4609-abb4-94ad4eeecd29-host-slash\") pod \"iptables-alerter-2hsq7\" (UID: \"25a5233b-4688-4609-abb4-94ad4eeecd29\") " pod="openshift-network-operator/iptables-alerter-2hsq7"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188552    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea2d8b03-e721-4102-846e-ee7d41d13e96-host\") pod \"node-ca-kdkj7\" (UID: \"ea2d8b03-e721-4102-846e-ee7d41d13e96\") " pod="openshift-image-registry/node-ca-kdkj7"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188582    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5pv8\" (UniqueName: \"kubernetes.io/projected/25a5233b-4688-4609-abb4-94ad4eeecd29-kube-api-access-s5pv8\") pod \"iptables-alerter-2hsq7\" (UID: \"25a5233b-4688-4609-abb4-94ad4eeecd29\") " pod="openshift-network-operator/iptables-alerter-2hsq7"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188659    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-kubelet\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188699    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188724    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-etc-selinux\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188734    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-kubelet\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188748    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-run-ovn\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188787    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.188800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188790    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-run-ovn\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188795    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-cni-bin\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188833    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-etc-selinux\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188830    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188864    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188875    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188895    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmbgl\" (UniqueName: \"kubernetes.io/projected/ea2d8b03-e721-4102-846e-ee7d41d13e96-kube-api-access-pmbgl\") pod \"node-ca-kdkj7\" (UID: \"ea2d8b03-e721-4102-846e-ee7d41d13e96\") " pod="openshift-image-registry/node-ca-kdkj7"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188910    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-cni-bin\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188920    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-socket-dir\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188949    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-etc-openvswitch\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188975    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.188999    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-device-dir\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189018    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-etc-openvswitch\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189042    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-ovn-node-metrics-cert\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189077    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-socket-dir\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189083    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189099    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-sys-fs\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.189625 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189136    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-run-systemd\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189149    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-sys-fs\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189152    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6b681468-002e-4500-b222-49da180d381c-device-dir\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189052    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-ovnkube-config\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189169    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-kubelet-config\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189176    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-run-systemd\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.188977    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189232    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-kubelet-config\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189194    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-dbus\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189248    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-dbus\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.189255    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret podName:d1dbf089-2e69-485b-9e8b-eb30c69e15aa nodeName:}" failed. No retries permitted until 2026-09-07 12:59:18.689236437 +0000 UTC m=+3.117534703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret") pod "global-pull-secret-syncer-njk59" (UID: "d1dbf089-2e69-485b-9e8b-eb30c69e15aa") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189303    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-slash\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189333    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-var-lib-openvswitch\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189360    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25a5233b-4688-4609-abb4-94ad4eeecd29-iptables-alerter-script\") pod \"iptables-alerter-2hsq7\" (UID: \"25a5233b-4688-4609-abb4-94ad4eeecd29\") " pod="openshift-network-operator/iptables-alerter-2hsq7"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189386    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-node-log\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189397    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-var-lib-openvswitch\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189363    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-host-slash\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.190272 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189418    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-ovnkube-script-lib\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.190923 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-env-overrides\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.190923 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189515    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-node-log\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.190923 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189844    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-ovnkube-script-lib\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.190923 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189864    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-env-overrides\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.190923 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.189955    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25a5233b-4688-4609-abb4-94ad4eeecd29-iptables-alerter-script\") pod \"iptables-alerter-2hsq7\" (UID: \"25a5233b-4688-4609-abb4-94ad4eeecd29\") " pod="openshift-network-operator/iptables-alerter-2hsq7"
Sep 07 12:59:18.191696 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.191670    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-ovn-node-metrics-cert\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.201222 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.201127    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hb9g\" (UniqueName: \"kubernetes.io/projected/3e4e1f64-dbc9-4677-a251-b03a65ae3ddb-kube-api-access-2hb9g\") pod \"ovnkube-node-4vwcx\" (UID: \"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.210800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.210776    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dgtp\" (UniqueName: \"kubernetes.io/projected/6b681468-002e-4500-b222-49da180d381c-kube-api-access-6dgtp\") pod \"aws-ebs-csi-driver-node-g7qx6\" (UID: \"6b681468-002e-4500-b222-49da180d381c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.216302 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.216270    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5pv8\" (UniqueName: \"kubernetes.io/projected/25a5233b-4688-4609-abb4-94ad4eeecd29-kube-api-access-s5pv8\") pod \"iptables-alerter-2hsq7\" (UID: \"25a5233b-4688-4609-abb4-94ad4eeecd29\") " pod="openshift-network-operator/iptables-alerter-2hsq7"
Sep 07 12:59:18.219921 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.219899    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmbgl\" (UniqueName: \"kubernetes.io/projected/ea2d8b03-e721-4102-846e-ee7d41d13e96-kube-api-access-pmbgl\") pod \"node-ca-kdkj7\" (UID: \"ea2d8b03-e721-4102-846e-ee7d41d13e96\") " pod="openshift-image-registry/node-ca-kdkj7"
Sep 07 12:59:18.273073 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.273025    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sqnx4"
Sep 07 12:59:18.283107 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.282266    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fkppx"
Sep 07 12:59:18.285439 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:18.285391    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1b14f3_6875_4d07_859d_ada2d3bb1f0f.slice/crio-2af510b4814606b3c84ce694a8177b06e73d644ddf7f40e6f63f8a39e9313371 WatchSource:0}: Error finding container 2af510b4814606b3c84ce694a8177b06e73d644ddf7f40e6f63f8a39e9313371: Status 404 returned error can't find the container with id 2af510b4814606b3c84ce694a8177b06e73d644ddf7f40e6f63f8a39e9313371
Sep 07 12:59:18.290569 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.290542    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dtj6g"
Sep 07 12:59:18.291218 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:18.291195    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa91b78f_8f6e_41e5_83df_322e9825cec5.slice/crio-ef1cfa1f768c1e4166bc654b7931a4c9eedd3d328e6d13b867e445b0ca6d19d4 WatchSource:0}: Error finding container ef1cfa1f768c1e4166bc654b7931a4c9eedd3d328e6d13b867e445b0ca6d19d4: Status 404 returned error can't find the container with id ef1cfa1f768c1e4166bc654b7931a4c9eedd3d328e6d13b867e445b0ca6d19d4
Sep 07 12:59:18.295429 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.295402    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z98gm"
Sep 07 12:59:18.300899 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:18.300875    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813ca344_4b86_40c6_b18c_f30a83c23504.slice/crio-964983f1b58fd6cbb26a7c669625bd0b8d109932d978079fa1bb96fc526db831 WatchSource:0}: Error finding container 964983f1b58fd6cbb26a7c669625bd0b8d109932d978079fa1bb96fc526db831: Status 404 returned error can't find the container with id 964983f1b58fd6cbb26a7c669625bd0b8d109932d978079fa1bb96fc526db831
Sep 07 12:59:18.304310 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.304287    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6xmwx"
Sep 07 12:59:18.304911 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:18.304890    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a700e69_d961_4293_81be_98b17240902b.slice/crio-f99181ca8d27c8fdf23614692d369b33605ef313760904d9ca8a317050ac73a2 WatchSource:0}: Error finding container f99181ca8d27c8fdf23614692d369b33605ef313760904d9ca8a317050ac73a2: Status 404 returned error can't find the container with id f99181ca8d27c8fdf23614692d369b33605ef313760904d9ca8a317050ac73a2
Sep 07 12:59:18.309267 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.309249    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6"
Sep 07 12:59:18.313939 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:18.313913    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b9aa4a_12ea_4917_94b3_eef425da8b28.slice/crio-000b620b9b5051e417347cce4d9dd125850b029e5cbdc5df0631ea2c5a2fc11a WatchSource:0}: Error finding container 000b620b9b5051e417347cce4d9dd125850b029e5cbdc5df0631ea2c5a2fc11a: Status 404 returned error can't find the container with id 000b620b9b5051e417347cce4d9dd125850b029e5cbdc5df0631ea2c5a2fc11a
Sep 07 12:59:18.317261 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.317239    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kdkj7"
Sep 07 12:59:18.319702 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:18.319683    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b681468_002e_4500_b222_49da180d381c.slice/crio-8feffe1134445147d79fa807456e3b033a315606c9ee5e96d2f6cd6de76d1760 WatchSource:0}: Error finding container 8feffe1134445147d79fa807456e3b033a315606c9ee5e96d2f6cd6de76d1760: Status 404 returned error can't find the container with id 8feffe1134445147d79fa807456e3b033a315606c9ee5e96d2f6cd6de76d1760
Sep 07 12:59:18.324145 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.324124    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2hsq7"
Sep 07 12:59:18.328286 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:18.328264    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2d8b03_e721_4102_846e_ee7d41d13e96.slice/crio-3f933663ad9256875b93ee212076e1d4a235c1ebd16515a09fa178c1833e6f82 WatchSource:0}: Error finding container 3f933663ad9256875b93ee212076e1d4a235c1ebd16515a09fa178c1833e6f82: Status 404 returned error can't find the container with id 3f933663ad9256875b93ee212076e1d4a235c1ebd16515a09fa178c1833e6f82
Sep 07 12:59:18.329569 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.329553    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:18.333900 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:18.333864    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a5233b_4688_4609_abb4_94ad4eeecd29.slice/crio-c85e684b6797537326aefeda7c764909775212cc5d379c2b8c62e904979b74b6 WatchSource:0}: Error finding container c85e684b6797537326aefeda7c764909775212cc5d379c2b8c62e904979b74b6: Status 404 returned error can't find the container with id c85e684b6797537326aefeda7c764909775212cc5d379c2b8c62e904979b74b6
Sep 07 12:59:18.339518 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:18.339488    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4e1f64_dbc9_4677_a251_b03a65ae3ddb.slice/crio-239f65c523d7d9cb88362778316490c3c8f1559afb0566c12c947e499e1b4401 WatchSource:0}: Error finding container 239f65c523d7d9cb88362778316490c3c8f1559afb0566c12c947e499e1b4401: Status 404 returned error can't find the container with id 239f65c523d7d9cb88362778316490c3c8f1559afb0566c12c947e499e1b4401
Sep 07 12:59:18.592291 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.592251    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:18.592485 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.592417    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:18.592549 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.592490    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs podName:f79dcf5c-ddc9-47c8-b61e-96fd4401c89f nodeName:}" failed. No retries permitted until 2026-09-07 12:59:19.592471078 +0000 UTC m=+4.020769328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs") pod "network-metrics-daemon-tsqvk" (UID: "f79dcf5c-ddc9-47c8-b61e-96fd4401c89f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:18.693292 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.693250    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:18.693477 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:18.693340    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqzq\" (UniqueName: \"kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq\") pod \"network-check-target-p7mj7\" (UID: \"a2da8de4-8df5-4b96-9449-1c8d346c5fec\") " pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:18.693477 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.693395    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:18.693477 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.693460    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:59:18.693477 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.693466    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret podName:d1dbf089-2e69-485b-9e8b-eb30c69e15aa nodeName:}" failed. No retries permitted until 2026-09-07 12:59:19.693447435 +0000 UTC m=+4.121745705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret") pod "global-pull-secret-syncer-njk59" (UID: "d1dbf089-2e69-485b-9e8b-eb30c69e15aa") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:18.693477 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.693475    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:59:18.693726 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.693488    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dkqzq for pod openshift-network-diagnostics/network-check-target-p7mj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:18.693726 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:18.693529    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq podName:a2da8de4-8df5-4b96-9449-1c8d346c5fec nodeName:}" failed. No retries permitted until 2026-09-07 12:59:19.693518314 +0000 UTC m=+4.121816580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkqzq" (UniqueName: "kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq") pod "network-check-target-p7mj7" (UID: "a2da8de4-8df5-4b96-9449-1c8d346c5fec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:19.072090 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.068912    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:19.072090 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:19.069044    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:19.087277 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.087241    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx" event={"ID":"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb","Type":"ContainerStarted","Data":"239f65c523d7d9cb88362778316490c3c8f1559afb0566c12c947e499e1b4401"}
Sep 07 12:59:19.088595 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.088267    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 12:54:18 +0000 UTC" deadline="2028-03-19 06:45:09.929730634 +0000 UTC"
Sep 07 12:59:19.088595 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.088296    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13409h45m50.841438327s"
Sep 07 12:59:19.089712 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.089637    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2hsq7" event={"ID":"25a5233b-4688-4609-abb4-94ad4eeecd29","Type":"ContainerStarted","Data":"c85e684b6797537326aefeda7c764909775212cc5d379c2b8c62e904979b74b6"}
Sep 07 12:59:19.094675 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.094651    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z98gm" event={"ID":"4a700e69-d961-4293-81be-98b17240902b","Type":"ContainerStarted","Data":"f99181ca8d27c8fdf23614692d369b33605ef313760904d9ca8a317050ac73a2"}
Sep 07 12:59:19.100626 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.100600    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sqnx4" event={"ID":"8f1b14f3-6875-4d07-859d-ada2d3bb1f0f","Type":"ContainerStarted","Data":"2af510b4814606b3c84ce694a8177b06e73d644ddf7f40e6f63f8a39e9313371"}
Sep 07 12:59:19.103354 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.103329    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kdkj7" event={"ID":"ea2d8b03-e721-4102-846e-ee7d41d13e96","Type":"ContainerStarted","Data":"3f933663ad9256875b93ee212076e1d4a235c1ebd16515a09fa178c1833e6f82"}
Sep 07 12:59:19.104929 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.104906    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6" event={"ID":"6b681468-002e-4500-b222-49da180d381c","Type":"ContainerStarted","Data":"8feffe1134445147d79fa807456e3b033a315606c9ee5e96d2f6cd6de76d1760"}
Sep 07 12:59:19.108313 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.108290    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6xmwx" event={"ID":"47b9aa4a-12ea-4917-94b3-eef425da8b28","Type":"ContainerStarted","Data":"000b620b9b5051e417347cce4d9dd125850b029e5cbdc5df0631ea2c5a2fc11a"}
Sep 07 12:59:19.121648 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.121615    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtj6g" event={"ID":"813ca344-4b86-40c6-b18c-f30a83c23504","Type":"ContainerStarted","Data":"964983f1b58fd6cbb26a7c669625bd0b8d109932d978079fa1bb96fc526db831"}
Sep 07 12:59:19.126741 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.126713    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fkppx" event={"ID":"fa91b78f-8f6e-41e5-83df-322e9825cec5","Type":"ContainerStarted","Data":"ef1cfa1f768c1e4166bc654b7931a4c9eedd3d328e6d13b867e445b0ca6d19d4"}
Sep 07 12:59:19.601348 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.601315    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:19.601572 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:19.601512    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:19.601639 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:19.601576    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs podName:f79dcf5c-ddc9-47c8-b61e-96fd4401c89f nodeName:}" failed. No retries permitted until 2026-09-07 12:59:21.601558061 +0000 UTC m=+6.029856315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs") pod "network-metrics-daemon-tsqvk" (UID: "f79dcf5c-ddc9-47c8-b61e-96fd4401c89f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:19.702250 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.702199    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:19.702385 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:19.702279    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqzq\" (UniqueName: \"kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq\") pod \"network-check-target-p7mj7\" (UID: \"a2da8de4-8df5-4b96-9449-1c8d346c5fec\") " pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:19.702385 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:19.702343    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:19.702491 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:19.702387    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:59:19.702491 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:19.702401    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:59:19.702491 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:19.702406    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret podName:d1dbf089-2e69-485b-9e8b-eb30c69e15aa nodeName:}" failed. No retries permitted until 2026-09-07 12:59:21.702388387 +0000 UTC m=+6.130686644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret") pod "global-pull-secret-syncer-njk59" (UID: "d1dbf089-2e69-485b-9e8b-eb30c69e15aa") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:19.702491 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:19.702429    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dkqzq for pod openshift-network-diagnostics/network-check-target-p7mj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:19.702491 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:19.702483    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq podName:a2da8de4-8df5-4b96-9449-1c8d346c5fec nodeName:}" failed. No retries permitted until 2026-09-07 12:59:21.70247083 +0000 UTC m=+6.130769098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkqzq" (UniqueName: "kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq") pod "network-check-target-p7mj7" (UID: "a2da8de4-8df5-4b96-9449-1c8d346c5fec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:20.070684 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:20.070190    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:20.070684 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:20.070321    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:20.071231 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:20.071028    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:20.071231 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:20.071155    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:20.089171 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:20.089123    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 12:54:18 +0000 UTC" deadline="2028-05-29 18:28:49.68156195 +0000 UTC"
Sep 07 12:59:20.089171 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:20.089162    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15125h29m29.592404526s"
Sep 07 12:59:20.175016 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:20.174306    2567 generic.go:356] "Generic (PLEG): container finished" podID="121d79ea4b4f92683e7694220da50ad1" containerID="83fde9e14c347976086099709dab04de283f2508097a8fa4e344da3ec6d9cea7" exitCode=0
Sep 07 12:59:20.175016 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:20.174370    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal" event={"ID":"121d79ea4b4f92683e7694220da50ad1","Type":"ContainerDied","Data":"83fde9e14c347976086099709dab04de283f2508097a8fa4e344da3ec6d9cea7"}
Sep 07 12:59:21.069099 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:21.069054    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:21.069290 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:21.069195    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:21.198215 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:21.198117    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal" event={"ID":"121d79ea4b4f92683e7694220da50ad1","Type":"ContainerStarted","Data":"d0fb8a43cb585353d3db1c2c10da913140693caa5191eba277e2f54922a37ef9"}
Sep 07 12:59:21.621429 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:21.621323    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:21.621607 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:21.621518    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:21.621607 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:21.621580    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs podName:f79dcf5c-ddc9-47c8-b61e-96fd4401c89f nodeName:}" failed. No retries permitted until 2026-09-07 12:59:25.621563208 +0000 UTC m=+10.049861465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs") pod "network-metrics-daemon-tsqvk" (UID: "f79dcf5c-ddc9-47c8-b61e-96fd4401c89f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:21.722541 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:21.722501    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:21.722729 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:21.722588    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqzq\" (UniqueName: \"kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq\") pod \"network-check-target-p7mj7\" (UID: \"a2da8de4-8df5-4b96-9449-1c8d346c5fec\") " pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:21.722729 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:21.722671    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:21.722846 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:21.722731    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:59:21.722846 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:21.722747    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:59:21.722846 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:21.722749    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret podName:d1dbf089-2e69-485b-9e8b-eb30c69e15aa nodeName:}" failed. No retries permitted until 2026-09-07 12:59:25.722727898 +0000 UTC m=+10.151026178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret") pod "global-pull-secret-syncer-njk59" (UID: "d1dbf089-2e69-485b-9e8b-eb30c69e15aa") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:21.722846 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:21.722760    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dkqzq for pod openshift-network-diagnostics/network-check-target-p7mj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:21.722846 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:21.722811    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq podName:a2da8de4-8df5-4b96-9449-1c8d346c5fec nodeName:}" failed. No retries permitted until 2026-09-07 12:59:25.722795387 +0000 UTC m=+10.151093652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkqzq" (UniqueName: "kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq") pod "network-check-target-p7mj7" (UID: "a2da8de4-8df5-4b96-9449-1c8d346c5fec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:22.071498 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:22.071464    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:22.071689 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:22.071609    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:22.071689 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:22.071466    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:22.071809 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:22.071731    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:23.069162 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:23.069125    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:23.069655 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:23.069269    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:24.069891 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:24.069456    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:24.069891 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:24.069605    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:24.069891 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:24.069659    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:24.069891 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:24.069764    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:25.070342 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:25.069790    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:25.070342 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:25.069920    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:25.657330 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:25.656713    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:25.657330 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:25.656899    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:25.657330 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:25.656979    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs podName:f79dcf5c-ddc9-47c8-b61e-96fd4401c89f nodeName:}" failed. No retries permitted until 2026-09-07 12:59:33.656958489 +0000 UTC m=+18.085256766 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs") pod "network-metrics-daemon-tsqvk" (UID: "f79dcf5c-ddc9-47c8-b61e-96fd4401c89f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:25.757348 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:25.757307    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqzq\" (UniqueName: \"kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq\") pod \"network-check-target-p7mj7\" (UID: \"a2da8de4-8df5-4b96-9449-1c8d346c5fec\") " pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:25.757531 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:25.757389    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:25.757531 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:25.757489    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:59:25.757531 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:25.757514    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:59:25.757531 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:25.757527    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dkqzq for pod openshift-network-diagnostics/network-check-target-p7mj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:25.757745 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:25.757495    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:25.757745 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:25.757581    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq podName:a2da8de4-8df5-4b96-9449-1c8d346c5fec nodeName:}" failed. No retries permitted until 2026-09-07 12:59:33.757561584 +0000 UTC m=+18.185859850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkqzq" (UniqueName: "kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq") pod "network-check-target-p7mj7" (UID: "a2da8de4-8df5-4b96-9449-1c8d346c5fec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:25.757745 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:25.757605    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret podName:d1dbf089-2e69-485b-9e8b-eb30c69e15aa nodeName:}" failed. No retries permitted until 2026-09-07 12:59:33.75759318 +0000 UTC m=+18.185891446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret") pod "global-pull-secret-syncer-njk59" (UID: "d1dbf089-2e69-485b-9e8b-eb30c69e15aa") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:26.069983 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:26.069943    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:26.071246 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:26.070741    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:26.071246 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:26.070862    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:26.071246 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:26.071122    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:27.069518 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:27.069485    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:27.069676 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:27.069622    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:28.069904 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:28.069874    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:28.070402 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:28.070021    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:28.070402 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:28.069875    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:28.070529 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:28.070455    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:29.069216 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:29.069182    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:29.069474 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:29.069306    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:30.069824 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:30.069783    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:30.070301 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:30.069797    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:30.070301 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:30.069926    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:30.070301 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:30.070006    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:31.069178 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:31.069137    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:31.069342 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:31.069272    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:32.069161 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:32.069119    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:32.069161 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:32.069141    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:32.069635 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:32.069257    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:32.069635 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:32.069397    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:33.069834 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:33.069798    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:33.070262 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:33.069901    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:33.722677 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:33.722439    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:33.722884 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:33.722756    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:33.722884 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:33.722852    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs podName:f79dcf5c-ddc9-47c8-b61e-96fd4401c89f nodeName:}" failed. No retries permitted until 2026-09-07 12:59:49.722832114 +0000 UTC m=+34.151130384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs") pod "network-metrics-daemon-tsqvk" (UID: "f79dcf5c-ddc9-47c8-b61e-96fd4401c89f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:33.823072 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:33.823016    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqzq\" (UniqueName: \"kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq\") pod \"network-check-target-p7mj7\" (UID: \"a2da8de4-8df5-4b96-9449-1c8d346c5fec\") " pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:33.823247 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:33.823118    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:33.823247 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:33.823211    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:59:33.823247 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:33.823226    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:33.823247 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:33.823236    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:59:33.823247 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:33.823246    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dkqzq for pod openshift-network-diagnostics/network-check-target-p7mj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:33.823416 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:33.823281    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret podName:d1dbf089-2e69-485b-9e8b-eb30c69e15aa nodeName:}" failed. No retries permitted until 2026-09-07 12:59:49.823266277 +0000 UTC m=+34.251564534 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret") pod "global-pull-secret-syncer-njk59" (UID: "d1dbf089-2e69-485b-9e8b-eb30c69e15aa") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:33.823416 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:33.823296    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq podName:a2da8de4-8df5-4b96-9449-1c8d346c5fec nodeName:}" failed. No retries permitted until 2026-09-07 12:59:49.823290568 +0000 UTC m=+34.251588819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkqzq" (UniqueName: "kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq") pod "network-check-target-p7mj7" (UID: "a2da8de4-8df5-4b96-9449-1c8d346c5fec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:34.069742 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:34.069706    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:34.069916 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:34.069706    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:34.069916 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:34.069833    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:34.069916 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:34.069905    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:35.069470 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:35.069218    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:35.069658 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:35.069632    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:36.069675 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:36.069641    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:36.070136 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:36.069762    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:36.070136 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:36.069818    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:36.070136 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:36.069934    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:37.069191 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:37.069165    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:37.069389 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:37.069296    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:37.248825 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:37.247773    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z98gm" event={"ID":"4a700e69-d961-4293-81be-98b17240902b","Type":"ContainerStarted","Data":"69bc2d135ffb3e2b5e01c10cfcbfcf9c2089a4c08ddae9830817654cb01b5f31"}
Sep 07 12:59:37.250993 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:37.250528    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-114.ec2.internal" event={"ID":"68fb4e41d01603c60ad3ed3f0a6da717","Type":"ContainerStarted","Data":"049d69ed5ecd917f50cac15a8ede0dfaf5fd0d184d0c0b8fc3ff016b04169abc"}
Sep 07 12:59:37.260307 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:37.260269    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6xmwx" event={"ID":"47b9aa4a-12ea-4917-94b3-eef425da8b28","Type":"ContainerStarted","Data":"8418e270af4d99dd711cfada197efe323cd176f1a6c966d9a4fe1c662532733b"}
Sep 07 12:59:37.264266 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:37.263529    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtj6g" event={"ID":"813ca344-4b86-40c6-b18c-f30a83c23504","Type":"ContainerStarted","Data":"31f9009285a55152d2c7aca4bdefee650f8580f96c70e409737b8fc8e7bd3b5e"}
Sep 07 12:59:37.272612 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:37.271274    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z98gm" podStartSLOduration=2.520201485 podStartE2EDuration="21.271252484s" totalImagesPullingTime="18.751050999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:37.271162143 +0000 UTC m=+21.699460438" watchObservedRunningTime="2026-09-07 12:59:37.271252484 +0000 UTC m=+21.699550760"
Sep 07 12:59:37.272612 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:37.271494    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-114.ec2.internal" podStartSLOduration=20.271483081 podStartE2EDuration="20.271483081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:21.223110176 +0000 UTC m=+5.651408470" watchObservedRunningTime="2026-09-07 12:59:37.271483081 +0000 UTC m=+21.699781355"
Sep 07 12:59:37.293902 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:37.293782    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-114.ec2.internal" podStartSLOduration=20.293759368 podStartE2EDuration="20.293759368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:37.293595622 +0000 UTC m=+21.721893895" watchObservedRunningTime="2026-09-07 12:59:37.293759368 +0000 UTC m=+21.722057642"
Sep 07 12:59:37.356943 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:37.356771    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6xmwx" podStartSLOduration=2.5649784479999997 podStartE2EDuration="21.356755795s" totalImagesPullingTime="18.791777347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:37.356558435 +0000 UTC m=+21.784856710" watchObservedRunningTime="2026-09-07 12:59:37.356755795 +0000 UTC m=+21.785054068"
Sep 07 12:59:38.069846 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.069817    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:38.069998 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.069855    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:38.069998 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:38.069934    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:38.070149 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:38.070048    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:38.266800 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.266764    2567 generic.go:356] "Generic (PLEG): container finished" podID="813ca344-4b86-40c6-b18c-f30a83c23504" containerID="31f9009285a55152d2c7aca4bdefee650f8580f96c70e409737b8fc8e7bd3b5e" exitCode=0
Sep 07 12:59:38.267732 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.266845    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtj6g" event={"ID":"813ca344-4b86-40c6-b18c-f30a83c23504","Type":"ContainerDied","Data":"31f9009285a55152d2c7aca4bdefee650f8580f96c70e409737b8fc8e7bd3b5e"}
Sep 07 12:59:38.269948 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.269917    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fkppx" event={"ID":"fa91b78f-8f6e-41e5-83df-322e9825cec5","Type":"ContainerStarted","Data":"d6dddd38f11cb4066c56e344a7038bc26c9aed49c28cb4b7b84553d0c22e4085"}
Sep 07 12:59:38.273109 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.273084    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx" event={"ID":"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb","Type":"ContainerStarted","Data":"d9fbec0ec3bbef31b4c0919a45f615adbfc558ef53ae81c6bba921235fe6d850"}
Sep 07 12:59:38.273199 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.273163    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx" event={"ID":"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb","Type":"ContainerStarted","Data":"f53b0d0a15e7ba3c31a68b9739006da497136efe175c6cb16d3d08157fadfe12"}
Sep 07 12:59:38.273243 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.273214    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx" event={"ID":"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb","Type":"ContainerStarted","Data":"27ba647bfeadf6670ca96e925153f8927b9828ea3477572799cb06788e227798"}
Sep 07 12:59:38.273279 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.273263    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx" event={"ID":"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb","Type":"ContainerStarted","Data":"18bb1e4c57bbd3d14ac0cc9fce33828e02592cd79284e4dcee1679913127f0b3"}
Sep 07 12:59:38.273316 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.273295    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx" event={"ID":"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb","Type":"ContainerStarted","Data":"4dac6acf43d99ee12365029737c2b9ac4cedfc29f216a8b493ae654d7cea24b6"}
Sep 07 12:59:38.273357 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.273334    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx" event={"ID":"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb","Type":"ContainerStarted","Data":"be1a7574eceaadfc36a7353018e3d11f75699dd1c1daac48fe18dc3375a95f66"}
Sep 07 12:59:38.274580 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.274559    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sqnx4" event={"ID":"8f1b14f3-6875-4d07-859d-ada2d3bb1f0f","Type":"ContainerStarted","Data":"ff5136f67a6608beb3f107758f36a17ac8c2ecb80309ba88f34ceb9e796f470b"}
Sep 07 12:59:38.276108 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.276087    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kdkj7" event={"ID":"ea2d8b03-e721-4102-846e-ee7d41d13e96","Type":"ContainerStarted","Data":"6403ca362935117eef907d7cd2aacc03a53663b9ae5401a610f3fd4064ef4f7d"}
Sep 07 12:59:38.277429 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.277406    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6" event={"ID":"6b681468-002e-4500-b222-49da180d381c","Type":"ContainerStarted","Data":"def75f79f466d8360945cb02e8a389bdbfbf7d2d0f9df245c402fdede29e29a1"}
Sep 07 12:59:38.310468 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.310427    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sqnx4" podStartSLOduration=3.565353171 podStartE2EDuration="22.310412484s" totalImagesPullingTime="18.745059313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:38.309459316 +0000 UTC m=+22.737757588" watchObservedRunningTime="2026-09-07 12:59:38.310412484 +0000 UTC m=+22.738710756"
Sep 07 12:59:38.328517 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:38.328430    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fkppx" podStartSLOduration=3.588726738 podStartE2EDuration="22.328416609s" totalImagesPullingTime="18.739689871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:38.327858185 +0000 UTC m=+22.756156458" watchObservedRunningTime="2026-09-07 12:59:38.328416609 +0000 UTC m=+22.756714881"
Sep 07 12:59:39.061360 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:39.061332    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 07 12:59:39.069841 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:39.069823    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:39.069926 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:39.069910    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:39.281393 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:39.281357    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2hsq7" event={"ID":"25a5233b-4688-4609-abb4-94ad4eeecd29","Type":"ContainerStarted","Data":"afe2db3e611dee2946d9e28e53391dac9df07acbfbbb540444c4a59adc0b6e0a"}
Sep 07 12:59:39.283229 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:39.283200    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6" event={"ID":"6b681468-002e-4500-b222-49da180d381c","Type":"ContainerStarted","Data":"ce22a3d764890f742ade737197969b5659ca21bc14328f98ab39f8968ae46156"}
Sep 07 12:59:39.296695 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:39.296657    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2hsq7" podStartSLOduration=4.599961834 podStartE2EDuration="23.296642598s" totalImagesPullingTime="18.696680764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:39.296545759 +0000 UTC m=+23.724844044" watchObservedRunningTime="2026-09-07 12:59:39.296642598 +0000 UTC m=+23.724940870"
Sep 07 12:59:39.296978 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:39.296959    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kdkj7" podStartSLOduration=4.594430727 podStartE2EDuration="23.296953043s" totalImagesPullingTime="18.702522316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:38.346738434 +0000 UTC m=+22.775036707" watchObservedRunningTime="2026-09-07 12:59:39.296953043 +0000 UTC m=+23.725251315"
Sep 07 12:59:40.032946 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:40.032826    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T12:59:39.061350074Z","UUID":"a268a1f3-32ac-4ac3-ad4e-023a1e7d01aa","Handler":null,"Name":"","Endpoint":""}
Sep 07 12:59:40.034630 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:40.034606    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 07 12:59:40.034630 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:40.034634    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 07 12:59:40.069864 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:40.069768    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:40.070045 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:40.069905    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:40.070045 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:40.069905    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:40.070194 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:40.070037    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:40.289814 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:40.289731    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx" event={"ID":"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb","Type":"ContainerStarted","Data":"9f691c3e83a9d1a52f5f1bca6f97aa06dd2904984ac408442025f9ce64021ce5"}
Sep 07 12:59:41.069226 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:41.069193    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:41.069381 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:41.069319    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:42.069793 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:42.069764    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:42.070239 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:42.069767    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:42.070239 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:42.069871    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:42.070239 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:42.069940    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:42.296185 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:42.296146    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6" event={"ID":"6b681468-002e-4500-b222-49da180d381c","Type":"ContainerStarted","Data":"413f601067c7b0701cc2446e570c493f801c025fa8fd220e7a5d9e0691879e12"}
Sep 07 12:59:42.299658 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:42.299626    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx" event={"ID":"3e4e1f64-dbc9-4677-a251-b03a65ae3ddb","Type":"ContainerStarted","Data":"ea9d4f3077db6d4a1d2af1564b12c106bed20c5a2bcdcc0a72b75ffba5b6b3b6"}
Sep 07 12:59:42.299895 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:42.299877    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:42.299975 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:42.299942    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:42.309288 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:42.309091    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fkppx"
Sep 07 12:59:42.309739 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:42.309704    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fkppx"
Sep 07 12:59:42.317198 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:42.317171    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:42.362314 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:42.362257    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g7qx6" podStartSLOduration=3.475267736 podStartE2EDuration="26.36223569s" totalImagesPullingTime="22.886967954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:42.320999188 +0000 UTC m=+26.749297481" watchObservedRunningTime="2026-09-07 12:59:42.36223569 +0000 UTC m=+26.790533959"
Sep 07 12:59:42.383946 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:42.383898    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx" podStartSLOduration=7.634345173 podStartE2EDuration="26.38388138s" totalImagesPullingTime="18.749536207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:42.363271583 +0000 UTC m=+26.791569882" watchObservedRunningTime="2026-09-07 12:59:42.38388138 +0000 UTC m=+26.812179653"
Sep 07 12:59:43.069348 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:43.069310    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:43.069542 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:43.069445    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:43.302465 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:43.302408    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fkppx"
Sep 07 12:59:43.302907 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:43.302869    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fkppx"
Sep 07 12:59:43.302907 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:43.302903    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:43.319328 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:43.319299    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 12:59:44.069802 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:44.069764    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:44.070022 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:44.069778    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:44.070022 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:44.069898    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:44.070022 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:44.069974    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:44.242529 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:44.242498    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-njk59"]
Sep 07 12:59:44.248141 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:44.248111    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p7mj7"]
Sep 07 12:59:44.248280 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:44.248216    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:44.248324 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:44.248297    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:44.250346 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:44.250320    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tsqvk"]
Sep 07 12:59:44.304361 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:44.304327    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:44.304361 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:44.304342    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:44.304922 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:44.304531    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:44.304922 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:44.304667    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:45.310823 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:45.310775    2567 generic.go:356] "Generic (PLEG): container finished" podID="813ca344-4b86-40c6-b18c-f30a83c23504" containerID="40ef553d057e2d05aa1e48f4eb3b8648da0390654fa260f1d7738fe0ec4587cf" exitCode=0
Sep 07 12:59:45.311588 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:45.310846    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtj6g" event={"ID":"813ca344-4b86-40c6-b18c-f30a83c23504","Type":"ContainerDied","Data":"40ef553d057e2d05aa1e48f4eb3b8648da0390654fa260f1d7738fe0ec4587cf"}
Sep 07 12:59:46.070848 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:46.070403    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:46.070848 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:46.070430    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:46.070848 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:46.070538    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:46.070848 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:46.070617    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:46.070848 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:46.070633    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:46.070848 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:46.070722    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:47.316983 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:47.316715    2567 generic.go:356] "Generic (PLEG): container finished" podID="813ca344-4b86-40c6-b18c-f30a83c23504" containerID="b20952eae1152365a0ebd3d65a240e50c703ac07518d8f0afe8103000cec31bd" exitCode=0
Sep 07 12:59:47.317487 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:47.316797    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtj6g" event={"ID":"813ca344-4b86-40c6-b18c-f30a83c23504","Type":"ContainerDied","Data":"b20952eae1152365a0ebd3d65a240e50c703ac07518d8f0afe8103000cec31bd"}
Sep 07 12:59:48.069083 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:48.069036    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:48.069219 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:48.069095    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:48.069282 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:48.069217    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7mj7" podUID="a2da8de4-8df5-4b96-9449-1c8d346c5fec"
Sep 07 12:59:48.069282 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:48.069273    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:48.069366 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:48.069348    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njk59" podUID="d1dbf089-2e69-485b-9e8b-eb30c69e15aa"
Sep 07 12:59:48.069477 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:48.069453    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tsqvk" podUID="f79dcf5c-ddc9-47c8-b61e-96fd4401c89f"
Sep 07 12:59:49.323731 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.323697    2567 generic.go:356] "Generic (PLEG): container finished" podID="813ca344-4b86-40c6-b18c-f30a83c23504" containerID="98fd3fe8895e32a293b775bf80aef27a3f8f5ceeb64adeae096845c9a74d13e2" exitCode=0
Sep 07 12:59:49.324389 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.323741    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtj6g" event={"ID":"813ca344-4b86-40c6-b18c-f30a83c23504","Type":"ContainerDied","Data":"98fd3fe8895e32a293b775bf80aef27a3f8f5ceeb64adeae096845c9a74d13e2"}
Sep 07 12:59:49.736086 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.736025    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:49.736266 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:49.736183    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:49.736266 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:49.736248    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs podName:f79dcf5c-ddc9-47c8-b61e-96fd4401c89f nodeName:}" failed. No retries permitted until 2026-09-07 13:00:21.736234127 +0000 UTC m=+66.164532377 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs") pod "network-metrics-daemon-tsqvk" (UID: "f79dcf5c-ddc9-47c8-b61e-96fd4401c89f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:59:49.836697 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.836648    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:49.836887 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.836736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqzq\" (UniqueName: \"kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq\") pod \"network-check-target-p7mj7\" (UID: \"a2da8de4-8df5-4b96-9449-1c8d346c5fec\") " pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:49.836887 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:49.836788    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:49.836887 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:49.836859    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret podName:d1dbf089-2e69-485b-9e8b-eb30c69e15aa nodeName:}" failed. No retries permitted until 2026-09-07 13:00:21.836842664 +0000 UTC m=+66.265140927 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret") pod "global-pull-secret-syncer-njk59" (UID: "d1dbf089-2e69-485b-9e8b-eb30c69e15aa") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:59:49.836887 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:49.836867    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:59:49.836887 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:49.836885    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:59:49.837141 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:49.836898    2567 projected.go:196] Error preparing data for projected volume kube-api-access-dkqzq for pod openshift-network-diagnostics/network-check-target-p7mj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:49.837141 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:49.836952    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq podName:a2da8de4-8df5-4b96-9449-1c8d346c5fec nodeName:}" failed. No retries permitted until 2026-09-07 13:00:21.836936688 +0000 UTC m=+66.265234956 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkqzq" (UniqueName: "kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq") pod "network-check-target-p7mj7" (UID: "a2da8de4-8df5-4b96-9449-1c8d346c5fec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:59:49.895161 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.895075    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-114.ec2.internal" event="NodeReady"
Sep 07 12:59:49.895317 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.895238    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 07 12:59:49.941783 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.941750    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d759cdf6-xw4nh"]
Sep 07 12:59:49.945248 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.945231    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:49.947704 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.947682    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 07 12:59:49.948128 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.948109    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 07 12:59:49.948280 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.948233    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 07 12:59:49.948280 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.948258    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dn6gs\""
Sep 07 12:59:49.955409 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.955389    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 07 12:59:49.959720 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.959630    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xvkgl"]
Sep 07 12:59:49.965754 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.965728    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g54bn"]
Sep 07 12:59:49.965890 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.965865    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xvkgl"
Sep 07 12:59:49.968757 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.968736    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 07 12:59:49.969299 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.969285    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 07 12:59:49.969446 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.969428    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d759cdf6-xw4nh"]
Sep 07 12:59:49.969524 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.969511    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:49.969565 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.969553    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 07 12:59:49.969661 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.969643    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n5q6z\""
Sep 07 12:59:49.971460 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.971441    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 07 12:59:49.971751 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.971737    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 07 12:59:49.971802 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.971768    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rqgx2\""
Sep 07 12:59:49.974483 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.974456    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xvkgl"]
Sep 07 12:59:49.977506 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:49.977452    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g54bn"]
Sep 07 12:59:50.038689 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.038607    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-image-registry-private-configuration\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.038689 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.038654    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69vp9\" (UniqueName: \"kubernetes.io/projected/6c700ee4-1744-4956-9728-eb2911022015-kube-api-access-69vp9\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:50.038689 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.038683    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wt6n\" (UniqueName: \"kubernetes.io/projected/4d36321e-8fe0-4c81-a143-d99bf792bd99-kube-api-access-2wt6n\") pod \"ingress-canary-xvkgl\" (UID: \"4d36321e-8fe0-4c81-a143-d99bf792bd99\") " pod="openshift-ingress-canary/ingress-canary-xvkgl"
Sep 07 12:59:50.039027 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.038730    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c700ee4-1744-4956-9728-eb2911022015-tmp-dir\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:50.039027 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.038817    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-bound-sa-token\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.039027 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.038846    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d36321e-8fe0-4c81-a143-d99bf792bd99-cert\") pod \"ingress-canary-xvkgl\" (UID: \"4d36321e-8fe0-4c81-a143-d99bf792bd99\") " pod="openshift-ingress-canary/ingress-canary-xvkgl"
Sep 07 12:59:50.039027 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.038862    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c700ee4-1744-4956-9728-eb2911022015-config-volume\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:50.039027 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.038908    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-trusted-ca\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.039027 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.038938    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-ca-trust-extracted\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.039027 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.038967    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-installation-pull-secrets\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.039306 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.039080    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkx7r\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-kube-api-access-tkx7r\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.039306 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.039135    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:50.039306 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.039166    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.039306 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.039186    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-certificates\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.069527 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.069488    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 12:59:50.069733 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.069501    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 12:59:50.069733 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.069662    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 12:59:50.073762 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.073731    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 12:59:50.073979 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.073767    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 12:59:50.073979 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.073834    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 12:59:50.073979 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.073772    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5v9xw\""
Sep 07 12:59:50.074594 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.074570    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8d5qt\""
Sep 07 12:59:50.075075 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.075040    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 12:59:50.139630 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.139598    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-trusted-ca\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.139630 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.139635    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-ca-trust-extracted\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.139884 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.139661    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-installation-pull-secrets\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.139884 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.139705    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkx7r\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-kube-api-access-tkx7r\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.139884 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.139751    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:50.140042 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.139919    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.140042 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.139962    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-certificates\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.140164 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.140097    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-image-registry-private-configuration\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.140164 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.140139    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69vp9\" (UniqueName: \"kubernetes.io/projected/6c700ee4-1744-4956-9728-eb2911022015-kube-api-access-69vp9\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:50.140252 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.140167    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wt6n\" (UniqueName: \"kubernetes.io/projected/4d36321e-8fe0-4c81-a143-d99bf792bd99-kube-api-access-2wt6n\") pod \"ingress-canary-xvkgl\" (UID: \"4d36321e-8fe0-4c81-a143-d99bf792bd99\") " pod="openshift-ingress-canary/ingress-canary-xvkgl"
Sep 07 12:59:50.140252 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.140193    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c700ee4-1744-4956-9728-eb2911022015-tmp-dir\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:50.140252 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.140222    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-bound-sa-token\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.140380 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.140248    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d36321e-8fe0-4c81-a143-d99bf792bd99-cert\") pod \"ingress-canary-xvkgl\" (UID: \"4d36321e-8fe0-4c81-a143-d99bf792bd99\") " pod="openshift-ingress-canary/ingress-canary-xvkgl"
Sep 07 12:59:50.140380 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.140273    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c700ee4-1744-4956-9728-eb2911022015-config-volume\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:50.140462 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:50.140418    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 12:59:50.140509 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:50.140470    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls podName:6c700ee4-1744-4956-9728-eb2911022015 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:50.640457641 +0000 UTC m=+35.068755895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls") pod "dns-default-g54bn" (UID: "6c700ee4-1744-4956-9728-eb2911022015") : secret "dns-default-metrics-tls" not found
Sep 07 12:59:50.140738 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.140138    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-ca-trust-extracted\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.140738 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:50.140707    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 12:59:50.140738 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:50.140721    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d759cdf6-xw4nh: secret "image-registry-tls" not found
Sep 07 12:59:50.140913 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:50.140761    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls podName:d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:50.640746691 +0000 UTC m=+35.069044951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls") pod "image-registry-d759cdf6-xw4nh" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8") : secret "image-registry-tls" not found
Sep 07 12:59:50.140913 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.140788    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-trusted-ca\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.140913 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.140880    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c700ee4-1744-4956-9728-eb2911022015-config-volume\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:50.141085 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.141028    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c700ee4-1744-4956-9728-eb2911022015-tmp-dir\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:50.141144 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.141087    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-certificates\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.143982 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.143935    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkppx_fa91b78f-8f6e-41e5-83df-322e9825cec5/konnectivity-agent/0.log" limit=1024
Sep 07 12:59:50.145293 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.145128    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-image-registry-private-configuration\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.145293 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.145181    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-installation-pull-secrets\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.145293 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.145213    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d36321e-8fe0-4c81-a143-d99bf792bd99-cert\") pod \"ingress-canary-xvkgl\" (UID: \"4d36321e-8fe0-4c81-a143-d99bf792bd99\") " pod="openshift-ingress-canary/ingress-canary-xvkgl"
Sep 07 12:59:50.153136 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.153105    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkx7r\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-kube-api-access-tkx7r\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.153462 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.153435    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wt6n\" (UniqueName: \"kubernetes.io/projected/4d36321e-8fe0-4c81-a143-d99bf792bd99-kube-api-access-2wt6n\") pod \"ingress-canary-xvkgl\" (UID: \"4d36321e-8fe0-4c81-a143-d99bf792bd99\") " pod="openshift-ingress-canary/ingress-canary-xvkgl"
Sep 07 12:59:50.153733 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.153704    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69vp9\" (UniqueName: \"kubernetes.io/projected/6c700ee4-1744-4956-9728-eb2911022015-kube-api-access-69vp9\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:50.153855 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.153824    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-bound-sa-token\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.247488 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.247450    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkppx_fa91b78f-8f6e-41e5-83df-322e9825cec5/konnectivity-agent/0.log" limit=1024
Sep 07 12:59:50.275359 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.275321    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xvkgl"
Sep 07 12:59:50.571541 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.571501    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xvkgl"]
Sep 07 12:59:50.577152 ip-10-0-128-114 kubenswrapper[2567]: W0907 12:59:50.577107    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d36321e_8fe0_4c81_a143_d99bf792bd99.slice/crio-22346cf4b8f7d2f7ebc5b781b475f58cae40529523299efa0a993ecbc17bca1b WatchSource:0}: Error finding container 22346cf4b8f7d2f7ebc5b781b475f58cae40529523299efa0a993ecbc17bca1b: Status 404 returned error can't find the container with id 22346cf4b8f7d2f7ebc5b781b475f58cae40529523299efa0a993ecbc17bca1b
Sep 07 12:59:50.645520 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.645476    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:50.645520 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:50.645520    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:50.645748 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:50.645636    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 12:59:50.645748 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:50.645650    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d759cdf6-xw4nh: secret "image-registry-tls" not found
Sep 07 12:59:50.645748 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:50.645717    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls podName:d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:51.645702173 +0000 UTC m=+36.074000447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls") pod "image-registry-d759cdf6-xw4nh" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8") : secret "image-registry-tls" not found
Sep 07 12:59:50.645856 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:50.645743    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 12:59:50.645856 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:50.645805    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls podName:6c700ee4-1744-4956-9728-eb2911022015 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:51.645789662 +0000 UTC m=+36.074087914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls") pod "dns-default-g54bn" (UID: "6c700ee4-1744-4956-9728-eb2911022015") : secret "dns-default-metrics-tls" not found
Sep 07 12:59:51.331332 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:51.331293    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xvkgl" event={"ID":"4d36321e-8fe0-4c81-a143-d99bf792bd99","Type":"ContainerStarted","Data":"22346cf4b8f7d2f7ebc5b781b475f58cae40529523299efa0a993ecbc17bca1b"}
Sep 07 12:59:51.653465 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:51.653271    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:51.653465 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:51.653326    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:51.653465 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:51.653397    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 12:59:51.653465 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:51.653460    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls podName:6c700ee4-1744-4956-9728-eb2911022015 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:53.653443321 +0000 UTC m=+38.081741585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls") pod "dns-default-g54bn" (UID: "6c700ee4-1744-4956-9728-eb2911022015") : secret "dns-default-metrics-tls" not found
Sep 07 12:59:51.654093 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:51.653487    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 12:59:51.654093 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:51.653511    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d759cdf6-xw4nh: secret "image-registry-tls" not found
Sep 07 12:59:51.654093 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:51.653560    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls podName:d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:53.653547535 +0000 UTC m=+38.081845791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls") pod "image-registry-d759cdf6-xw4nh" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8") : secret "image-registry-tls" not found
Sep 07 12:59:53.670934 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:53.670894    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:53.670934 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:53.670938    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:53.671597 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:53.671070    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 12:59:53.671597 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:53.671073    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 12:59:53.671597 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:53.671144    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d759cdf6-xw4nh: secret "image-registry-tls" not found
Sep 07 12:59:53.671597 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:53.671153    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls podName:6c700ee4-1744-4956-9728-eb2911022015 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:57.671128264 +0000 UTC m=+42.099426539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls") pod "dns-default-g54bn" (UID: "6c700ee4-1744-4956-9728-eb2911022015") : secret "dns-default-metrics-tls" not found
Sep 07 12:59:53.671597 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:53.671182    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls podName:d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:57.671174098 +0000 UTC m=+42.099472352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls") pod "image-registry-d759cdf6-xw4nh" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8") : secret "image-registry-tls" not found
Sep 07 12:59:54.601231 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:54.601198    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkppx_fa91b78f-8f6e-41e5-83df-322e9825cec5/konnectivity-agent/0.log" limit=1024
Sep 07 12:59:54.703529 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:54.703497    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkppx_fa91b78f-8f6e-41e5-83df-322e9825cec5/konnectivity-agent/0.log" limit=1024
Sep 07 12:59:55.344427 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:55.344388    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xvkgl" event={"ID":"4d36321e-8fe0-4c81-a143-d99bf792bd99","Type":"ContainerStarted","Data":"204ad87fa3d4aa2a0ff3082abb0dc1491cd17323ca9c8771897274bce3a8163b"}
Sep 07 12:59:55.347786 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:55.347752    2567 generic.go:356] "Generic (PLEG): container finished" podID="813ca344-4b86-40c6-b18c-f30a83c23504" containerID="e6a0977f27628c95e07dd5011ea8e30e85bfd307e98f0e6d202cb05a3711a66e" exitCode=0
Sep 07 12:59:55.347928 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:55.347801    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtj6g" event={"ID":"813ca344-4b86-40c6-b18c-f30a83c23504","Type":"ContainerDied","Data":"e6a0977f27628c95e07dd5011ea8e30e85bfd307e98f0e6d202cb05a3711a66e"}
Sep 07 12:59:55.380757 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:55.380305    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xvkgl" podStartSLOduration=1.96644059 podStartE2EDuration="6.380283057s" totalImagesPullingTime="4.413842467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:55.380177564 +0000 UTC m=+39.808475858" watchObservedRunningTime="2026-09-07 12:59:55.380283057 +0000 UTC m=+39.808581333"
Sep 07 12:59:55.428186 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:55.428151    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fkppx_fa91b78f-8f6e-41e5-83df-322e9825cec5/konnectivity-agent/0.log" limit=1024
Sep 07 12:59:56.352780 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:56.352738    2567 generic.go:356] "Generic (PLEG): container finished" podID="813ca344-4b86-40c6-b18c-f30a83c23504" containerID="a0fff65ef98a9512c2c560844f0214ecb89e5c7d21acd60dfbcbb0b081496dba" exitCode=0
Sep 07 12:59:56.353448 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:56.352823    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtj6g" event={"ID":"813ca344-4b86-40c6-b18c-f30a83c23504","Type":"ContainerDied","Data":"a0fff65ef98a9512c2c560844f0214ecb89e5c7d21acd60dfbcbb0b081496dba"}
Sep 07 12:59:57.358952 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:57.358914    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtj6g" event={"ID":"813ca344-4b86-40c6-b18c-f30a83c23504","Type":"ContainerStarted","Data":"78bae34ae13d173a48fd68f53007628f619912d29425aa4446b1c2f94007f9c7"}
Sep 07 12:59:57.380863 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:57.380801    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dtj6g" podStartSLOduration=8.737936911 podStartE2EDuration="41.380779177s" totalImagesPullingTime="32.642842266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:57.380014358 +0000 UTC m=+41.808312658" watchObservedRunningTime="2026-09-07 12:59:57.380779177 +0000 UTC m=+41.809077449"
Sep 07 12:59:57.701460 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:57.701366    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 12:59:57.701460 ip-10-0-128-114 kubenswrapper[2567]: I0907 12:59:57.701419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 12:59:57.701645 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:57.701521    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 12:59:57.701645 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:57.701534    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 12:59:57.701645 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:57.701545    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d759cdf6-xw4nh: secret "image-registry-tls" not found
Sep 07 12:59:57.701645 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:57.701595    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls podName:6c700ee4-1744-4956-9728-eb2911022015 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:05.701580719 +0000 UTC m=+50.129878990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls") pod "dns-default-g54bn" (UID: "6c700ee4-1744-4956-9728-eb2911022015") : secret "dns-default-metrics-tls" not found
Sep 07 12:59:57.701645 ip-10-0-128-114 kubenswrapper[2567]: E0907 12:59:57.701609    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls podName:d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:05.701604038 +0000 UTC m=+50.129902289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls") pod "image-registry-d759cdf6-xw4nh" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8") : secret "image-registry-tls" not found
Sep 07 13:00:05.758626 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:05.758581    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 13:00:05.758626 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:05.758630    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 13:00:05.759192 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:05.758754    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 13:00:05.759192 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:05.758778    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 13:00:05.759192 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:05.758789    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d759cdf6-xw4nh: secret "image-registry-tls" not found
Sep 07 13:00:05.759192 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:05.758826    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls podName:6c700ee4-1744-4956-9728-eb2911022015 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:21.758807117 +0000 UTC m=+66.187105382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls") pod "dns-default-g54bn" (UID: "6c700ee4-1744-4956-9728-eb2911022015") : secret "dns-default-metrics-tls" not found
Sep 07 13:00:05.759192 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:05.758840    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls podName:d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:21.758834138 +0000 UTC m=+66.187132388 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls") pod "image-registry-d759cdf6-xw4nh" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8") : secret "image-registry-tls" not found
Sep 07 13:00:13.893769 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.893733    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"]
Sep 07 13:00:13.900904 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.900880    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"
Sep 07 13:00:13.908432 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.908413    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 07 13:00:13.908528 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.908446    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-b4ptt\""
Sep 07 13:00:13.915570 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.915549    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gtfhw\" (UID: \"c576bf35-1368-4bcb-9d09-69010dc2f9fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"
Sep 07 13:00:13.915632 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.915577    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phpxv\" (UniqueName: \"kubernetes.io/projected/c576bf35-1368-4bcb-9d09-69010dc2f9fe-kube-api-access-phpxv\") pod \"cluster-samples-operator-7cdcfcf76c-gtfhw\" (UID: \"c576bf35-1368-4bcb-9d09-69010dc2f9fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"
Sep 07 13:00:13.918217 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.918200    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 07 13:00:13.920577 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.920559    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 07 13:00:13.936654 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.936630    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"]
Sep 07 13:00:13.946590 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.946558    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"]
Sep 07 13:00:13.951411 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.951394    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"
Sep 07 13:00:13.953937 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.953919    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 07 13:00:13.954278 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.954259    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 07 13:00:13.954369 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.954261    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 07 13:00:13.955408 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.955387    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 07 13:00:13.955520 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.955496    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8fjlv\""
Sep 07 13:00:13.971962 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:13.971937    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"]
Sep 07 13:00:14.016070 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.016029    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6bk2\" (UniqueName: \"kubernetes.io/projected/d29420b3-de04-4ece-82d5-3cc4fbc8d56f-kube-api-access-l6bk2\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2zrp5\" (UID: \"d29420b3-de04-4ece-82d5-3cc4fbc8d56f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"
Sep 07 13:00:14.016237 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.016112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gtfhw\" (UID: \"c576bf35-1368-4bcb-9d09-69010dc2f9fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"
Sep 07 13:00:14.016237 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.016131    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phpxv\" (UniqueName: \"kubernetes.io/projected/c576bf35-1368-4bcb-9d09-69010dc2f9fe-kube-api-access-phpxv\") pod \"cluster-samples-operator-7cdcfcf76c-gtfhw\" (UID: \"c576bf35-1368-4bcb-9d09-69010dc2f9fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"
Sep 07 13:00:14.016237 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.016149    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d29420b3-de04-4ece-82d5-3cc4fbc8d56f-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2zrp5\" (UID: \"d29420b3-de04-4ece-82d5-3cc4fbc8d56f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"
Sep 07 13:00:14.016237 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.016168    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d29420b3-de04-4ece-82d5-3cc4fbc8d56f-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2zrp5\" (UID: \"d29420b3-de04-4ece-82d5-3cc4fbc8d56f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"
Sep 07 13:00:14.016381 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:14.016270    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 13:00:14.016381 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:14.016337    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls podName:c576bf35-1368-4bcb-9d09-69010dc2f9fe nodeName:}" failed. No retries permitted until 2026-09-07 13:00:14.516323795 +0000 UTC m=+58.944622046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gtfhw" (UID: "c576bf35-1368-4bcb-9d09-69010dc2f9fe") : secret "samples-operator-tls" not found
Sep 07 13:00:14.041103 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.041053    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phpxv\" (UniqueName: \"kubernetes.io/projected/c576bf35-1368-4bcb-9d09-69010dc2f9fe-kube-api-access-phpxv\") pod \"cluster-samples-operator-7cdcfcf76c-gtfhw\" (UID: \"c576bf35-1368-4bcb-9d09-69010dc2f9fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"
Sep 07 13:00:14.089554 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.089523    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"]
Sep 07 13:00:14.109330 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.109298    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"]
Sep 07 13:00:14.109472 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.109392    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"
Sep 07 13:00:14.111410 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.111377    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 07 13:00:14.111410 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.111393    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-slpbt\""
Sep 07 13:00:14.111899 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.111879    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 07 13:00:14.111972 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.111958    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 07 13:00:14.112029 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.112012    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 07 13:00:14.116407 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.116385    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d29420b3-de04-4ece-82d5-3cc4fbc8d56f-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2zrp5\" (UID: \"d29420b3-de04-4ece-82d5-3cc4fbc8d56f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"
Sep 07 13:00:14.116522 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.116421    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d29420b3-de04-4ece-82d5-3cc4fbc8d56f-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2zrp5\" (UID: \"d29420b3-de04-4ece-82d5-3cc4fbc8d56f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"
Sep 07 13:00:14.116522 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.116460    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f4572efa-98bc-4777-a77e-662b2481229a-config\") pod \"service-ca-operator-859f94c4b-l8qsk\" (UID: \"f4572efa-98bc-4777-a77e-662b2481229a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"
Sep 07 13:00:14.116522 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.116485    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6fjh\" (UniqueName: \"kubernetes.io/projected/f4572efa-98bc-4777-a77e-662b2481229a-kube-api-access-b6fjh\") pod \"service-ca-operator-859f94c4b-l8qsk\" (UID: \"f4572efa-98bc-4777-a77e-662b2481229a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"
Sep 07 13:00:14.116683 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.116594    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6bk2\" (UniqueName: \"kubernetes.io/projected/d29420b3-de04-4ece-82d5-3cc4fbc8d56f-kube-api-access-l6bk2\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2zrp5\" (UID: \"d29420b3-de04-4ece-82d5-3cc4fbc8d56f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"
Sep 07 13:00:14.116683 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.116619    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4572efa-98bc-4777-a77e-662b2481229a-serving-cert\") pod \"service-ca-operator-859f94c4b-l8qsk\" (UID: \"f4572efa-98bc-4777-a77e-662b2481229a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"
Sep 07 13:00:14.116999 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.116981    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d29420b3-de04-4ece-82d5-3cc4fbc8d56f-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2zrp5\" (UID: \"d29420b3-de04-4ece-82d5-3cc4fbc8d56f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"
Sep 07 13:00:14.118667 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.118649    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d29420b3-de04-4ece-82d5-3cc4fbc8d56f-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2zrp5\" (UID: \"d29420b3-de04-4ece-82d5-3cc4fbc8d56f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"
Sep 07 13:00:14.130753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.130727    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6bk2\" (UniqueName: \"kubernetes.io/projected/d29420b3-de04-4ece-82d5-3cc4fbc8d56f-kube-api-access-l6bk2\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2zrp5\" (UID: \"d29420b3-de04-4ece-82d5-3cc4fbc8d56f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"
Sep 07 13:00:14.217195 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.217101    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4572efa-98bc-4777-a77e-662b2481229a-serving-cert\") pod \"service-ca-operator-859f94c4b-l8qsk\" (UID: \"f4572efa-98bc-4777-a77e-662b2481229a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"
Sep 07 13:00:14.217195 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.217175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f4572efa-98bc-4777-a77e-662b2481229a-config\") pod \"service-ca-operator-859f94c4b-l8qsk\" (UID: \"f4572efa-98bc-4777-a77e-662b2481229a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"
Sep 07 13:00:14.217195 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.217198    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6fjh\" (UniqueName: \"kubernetes.io/projected/f4572efa-98bc-4777-a77e-662b2481229a-kube-api-access-b6fjh\") pod \"service-ca-operator-859f94c4b-l8qsk\" (UID: \"f4572efa-98bc-4777-a77e-662b2481229a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"
Sep 07 13:00:14.217823 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.217803    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f4572efa-98bc-4777-a77e-662b2481229a-config\") pod \"service-ca-operator-859f94c4b-l8qsk\" (UID: \"f4572efa-98bc-4777-a77e-662b2481229a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"
Sep 07 13:00:14.219449 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.219423    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4572efa-98bc-4777-a77e-662b2481229a-serving-cert\") pod \"service-ca-operator-859f94c4b-l8qsk\" (UID: \"f4572efa-98bc-4777-a77e-662b2481229a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"
Sep 07 13:00:14.224877 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.224852    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6fjh\" (UniqueName: \"kubernetes.io/projected/f4572efa-98bc-4777-a77e-662b2481229a-kube-api-access-b6fjh\") pod \"service-ca-operator-859f94c4b-l8qsk\" (UID: \"f4572efa-98bc-4777-a77e-662b2481229a\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"
Sep 07 13:00:14.259877 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.259837    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"
Sep 07 13:00:14.418773 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.418740    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"
Sep 07 13:00:14.519698 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.519665    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gtfhw\" (UID: \"c576bf35-1368-4bcb-9d09-69010dc2f9fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"
Sep 07 13:00:14.519869 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:14.519798    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 13:00:14.519869 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:14.519863    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls podName:c576bf35-1368-4bcb-9d09-69010dc2f9fe nodeName:}" failed. No retries permitted until 2026-09-07 13:00:15.519848635 +0000 UTC m=+59.948146891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gtfhw" (UID: "c576bf35-1368-4bcb-9d09-69010dc2f9fe") : secret "samples-operator-tls" not found
Sep 07 13:00:14.556869 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.556824    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5"]
Sep 07 13:00:14.559856 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:00:14.559828    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29420b3_de04_4ece_82d5_3cc4fbc8d56f.slice/crio-fe037259f75e98a9e1506664a1349b8c329560d613d524582141dd8d486fbcf4 WatchSource:0}: Error finding container fe037259f75e98a9e1506664a1349b8c329560d613d524582141dd8d486fbcf4: Status 404 returned error can't find the container with id fe037259f75e98a9e1506664a1349b8c329560d613d524582141dd8d486fbcf4
Sep 07 13:00:14.706940 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:14.706906    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk"]
Sep 07 13:00:14.710131 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:00:14.710102    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4572efa_98bc_4777_a77e_662b2481229a.slice/crio-10760a69777c7a50cf5c33f8727daf18451dc1b3ef53c1fcf68a6708d32ea6be WatchSource:0}: Error finding container 10760a69777c7a50cf5c33f8727daf18451dc1b3ef53c1fcf68a6708d32ea6be: Status 404 returned error can't find the container with id 10760a69777c7a50cf5c33f8727daf18451dc1b3ef53c1fcf68a6708d32ea6be
Sep 07 13:00:15.324971 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:15.324936    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vwcx"
Sep 07 13:00:15.407135 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:15.407097    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk" event={"ID":"f4572efa-98bc-4777-a77e-662b2481229a","Type":"ContainerStarted","Data":"10760a69777c7a50cf5c33f8727daf18451dc1b3ef53c1fcf68a6708d32ea6be"}
Sep 07 13:00:15.408268 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:15.408237    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5" event={"ID":"d29420b3-de04-4ece-82d5-3cc4fbc8d56f","Type":"ContainerStarted","Data":"fe037259f75e98a9e1506664a1349b8c329560d613d524582141dd8d486fbcf4"}
Sep 07 13:00:15.526415 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:15.526377    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gtfhw\" (UID: \"c576bf35-1368-4bcb-9d09-69010dc2f9fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"
Sep 07 13:00:15.526589 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:15.526530    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 13:00:15.526643 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:15.526634    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls podName:c576bf35-1368-4bcb-9d09-69010dc2f9fe nodeName:}" failed. No retries permitted until 2026-09-07 13:00:17.526616989 +0000 UTC m=+61.954915241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gtfhw" (UID: "c576bf35-1368-4bcb-9d09-69010dc2f9fe") : secret "samples-operator-tls" not found
Sep 07 13:00:17.415304 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:17.415208    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5" event={"ID":"d29420b3-de04-4ece-82d5-3cc4fbc8d56f","Type":"ContainerStarted","Data":"7a4d7aedf63e33e6bbc82e6c038b0ecf95a67aaf850b5a6c84e3c38c04bbb405"}
Sep 07 13:00:17.433519 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:17.433440    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5" podStartSLOduration=1.968977127 podStartE2EDuration="4.433418862s" totalImagesPullingTime="2.464441735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:17.432287467 +0000 UTC m=+61.860585829" watchObservedRunningTime="2026-09-07 13:00:17.433418862 +0000 UTC m=+61.861717137"
Sep 07 13:00:17.544706 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:17.544661    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gtfhw\" (UID: \"c576bf35-1368-4bcb-9d09-69010dc2f9fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"
Sep 07 13:00:17.544898 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:17.544795    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 13:00:17.544898 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:17.544863    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls podName:c576bf35-1368-4bcb-9d09-69010dc2f9fe nodeName:}" failed. No retries permitted until 2026-09-07 13:00:21.544843709 +0000 UTC m=+65.973141983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gtfhw" (UID: "c576bf35-1368-4bcb-9d09-69010dc2f9fe") : secret "samples-operator-tls" not found
Sep 07 13:00:18.419410 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:18.419362    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk" event={"ID":"f4572efa-98bc-4777-a77e-662b2481229a","Type":"ContainerStarted","Data":"51317606daf62c33866752f44d374ca086aeb6878880370330fe1c50d9ccbc19"}
Sep 07 13:00:18.439717 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:18.439672    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk" podStartSLOduration=1.18136718 podStartE2EDuration="4.439654993s" totalImagesPullingTime="3.258287813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:18.439582784 +0000 UTC m=+62.867881057" watchObservedRunningTime="2026-09-07 13:00:18.439654993 +0000 UTC m=+62.867953266"
Sep 07 13:00:21.575790 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.575735    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gtfhw\" (UID: \"c576bf35-1368-4bcb-9d09-69010dc2f9fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"
Sep 07 13:00:21.576200 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:21.575894    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 13:00:21.576200 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:21.575964    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls podName:c576bf35-1368-4bcb-9d09-69010dc2f9fe nodeName:}" failed. No retries permitted until 2026-09-07 13:00:29.575947763 +0000 UTC m=+74.004246041 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-gtfhw" (UID: "c576bf35-1368-4bcb-9d09-69010dc2f9fe") : secret "samples-operator-tls" not found
Sep 07 13:00:21.762655 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.762616    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hzp8t"]
Sep 07 13:00:21.765539 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.765522    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:21.767548 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.767523    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 07 13:00:21.767684 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.767574    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 07 13:00:21.767748 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.767678    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 07 13:00:21.768203 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.768185    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 07 13:00:21.768203 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.768199    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9l2d6\""
Sep 07 13:00:21.777230 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.777210    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 13:00:21.777326 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.777246    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 13:00:21.777326 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.777277    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 13:00:21.777429 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:21.777362    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 13:00:21.777473 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:21.777431    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls podName:6c700ee4-1744-4956-9728-eb2911022015 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:53.777410845 +0000 UTC m=+98.205709101 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls") pod "dns-default-g54bn" (UID: "6c700ee4-1744-4956-9728-eb2911022015") : secret "dns-default-metrics-tls" not found
Sep 07 13:00:21.777473 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:21.777366    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 13:00:21.777473 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:21.777465    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d759cdf6-xw4nh: secret "image-registry-tls" not found
Sep 07 13:00:21.777583 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:21.777521    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls podName:d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:53.777500589 +0000 UTC m=+98.205798868 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls") pod "image-registry-d759cdf6-xw4nh" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8") : secret "image-registry-tls" not found
Sep 07 13:00:21.779307 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.779288    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 13:00:21.782348 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.782322    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzp8t"]
Sep 07 13:00:21.788362 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:21.788342    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 07 13:00:21.788460 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:21.788427    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs podName:f79dcf5c-ddc9-47c8-b61e-96fd4401c89f nodeName:}" failed. No retries permitted until 2026-09-07 13:01:25.788412518 +0000 UTC m=+130.216710769 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs") pod "network-metrics-daemon-tsqvk" (UID: "f79dcf5c-ddc9-47c8-b61e-96fd4401c89f") : secret "metrics-daemon-secret" not found
Sep 07 13:00:21.878142 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.878028    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqzq\" (UniqueName: \"kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq\") pod \"network-check-target-p7mj7\" (UID: \"a2da8de4-8df5-4b96-9449-1c8d346c5fec\") " pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 13:00:21.878142 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.878121    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2469a9d8-5736-4c2b-a707-8128164910e8-data-volume\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:21.878346 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.878171    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2469a9d8-5736-4c2b-a707-8128164910e8-crio-socket\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:21.878346 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.878196    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbdzj\" (UniqueName: \"kubernetes.io/projected/2469a9d8-5736-4c2b-a707-8128164910e8-kube-api-access-bbdzj\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:21.878346 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.878245    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 13:00:21.878346 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.878338    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:21.878491 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.878363    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:21.880422 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.880404    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 13:00:21.880849 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.880835    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 13:00:21.890366 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.890343    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 13:00:21.890731 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.890714    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1dbf089-2e69-485b-9e8b-eb30c69e15aa-original-pull-secret\") pod \"global-pull-secret-syncer-njk59\" (UID: \"d1dbf089-2e69-485b-9e8b-eb30c69e15aa\") " pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 13:00:21.901394 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.901372    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkqzq\" (UniqueName: \"kubernetes.io/projected/a2da8de4-8df5-4b96-9449-1c8d346c5fec-kube-api-access-dkqzq\") pod \"network-check-target-p7mj7\" (UID: \"a2da8de4-8df5-4b96-9449-1c8d346c5fec\") " pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 13:00:21.978923 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.978876    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:21.978923 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.978923    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:21.979169 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:21.979009    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 13:00:21.979169 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.979078    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2469a9d8-5736-4c2b-a707-8128164910e8-data-volume\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:21.979169 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:21.979108    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm podName:2469a9d8-5736-4c2b-a707-8128164910e8 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:22.479093801 +0000 UTC m=+66.907392071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzp8t" (UID: "2469a9d8-5736-4c2b-a707-8128164910e8") : configmap "kube-rbac-proxy" not found
Sep 07 13:00:21.979169 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:21.979022    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 13:00:21.979169 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.979151    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2469a9d8-5736-4c2b-a707-8128164910e8-crio-socket\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:21.979376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.979174    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbdzj\" (UniqueName: \"kubernetes.io/projected/2469a9d8-5736-4c2b-a707-8128164910e8-kube-api-access-bbdzj\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:21.979376 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:21.979193    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls podName:2469a9d8-5736-4c2b-a707-8128164910e8 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:22.4791729 +0000 UTC m=+66.907471165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzp8t" (UID: "2469a9d8-5736-4c2b-a707-8128164910e8") : secret "insights-runtime-extractor-tls" not found
Sep 07 13:00:21.979376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.979263    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2469a9d8-5736-4c2b-a707-8128164910e8-crio-socket\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:21.979376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.979345    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2469a9d8-5736-4c2b-a707-8128164910e8-data-volume\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:21.996604 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:21.996573    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbdzj\" (UniqueName: \"kubernetes.io/projected/2469a9d8-5736-4c2b-a707-8128164910e8-kube-api-access-bbdzj\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:22.181674 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.181575    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njk59"
Sep 07 13:00:22.194283 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.194253    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8d5qt\""
Sep 07 13:00:22.202152 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.202127    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 13:00:22.341768 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.341735    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-cz64z"]
Sep 07 13:00:22.343753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.343734    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-cz64z"
Sep 07 13:00:22.346966 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.346942    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 07 13:00:22.346966 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.346954    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 07 13:00:22.347272 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.347256    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 07 13:00:22.347422 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.347403    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j59xm\""
Sep 07 13:00:22.347572 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.347554    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 07 13:00:22.357052 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.357029    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-cz64z"]
Sep 07 13:00:22.382839 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.382811    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0af9d659-61d2-4066-ac26-78eec940e914-signing-cabundle\") pod \"service-ca-75d99c484-cz64z\" (UID: \"0af9d659-61d2-4066-ac26-78eec940e914\") " pod="openshift-service-ca/service-ca-75d99c484-cz64z"
Sep 07 13:00:22.382969 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.382849    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0af9d659-61d2-4066-ac26-78eec940e914-signing-key\") pod \"service-ca-75d99c484-cz64z\" (UID: \"0af9d659-61d2-4066-ac26-78eec940e914\") " pod="openshift-service-ca/service-ca-75d99c484-cz64z"
Sep 07 13:00:22.382969 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.382957    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkv6g\" (UniqueName: \"kubernetes.io/projected/0af9d659-61d2-4066-ac26-78eec940e914-kube-api-access-fkv6g\") pod \"service-ca-75d99c484-cz64z\" (UID: \"0af9d659-61d2-4066-ac26-78eec940e914\") " pod="openshift-service-ca/service-ca-75d99c484-cz64z"
Sep 07 13:00:22.454667 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.454593    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-njk59"]
Sep 07 13:00:22.457910 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:00:22.457878    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1dbf089_2e69_485b_9e8b_eb30c69e15aa.slice/crio-c6abdcba839d3f48a01bbf67a7a492aca051c8a053d8554f12964539eb8a2721 WatchSource:0}: Error finding container c6abdcba839d3f48a01bbf67a7a492aca051c8a053d8554f12964539eb8a2721: Status 404 returned error can't find the container with id c6abdcba839d3f48a01bbf67a7a492aca051c8a053d8554f12964539eb8a2721
Sep 07 13:00:22.477857 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.477828    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p7mj7"]
Sep 07 13:00:22.479897 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:00:22.479873    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2da8de4_8df5_4b96_9449_1c8d346c5fec.slice/crio-05adfa592ef1ede5937fa3299ff076b6fff16c0bd82d5568d70876f22890be56 WatchSource:0}: Error finding container 05adfa592ef1ede5937fa3299ff076b6fff16c0bd82d5568d70876f22890be56: Status 404 returned error can't find the container with id 05adfa592ef1ede5937fa3299ff076b6fff16c0bd82d5568d70876f22890be56
Sep 07 13:00:22.483623 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.483600    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0af9d659-61d2-4066-ac26-78eec940e914-signing-cabundle\") pod \"service-ca-75d99c484-cz64z\" (UID: \"0af9d659-61d2-4066-ac26-78eec940e914\") " pod="openshift-service-ca/service-ca-75d99c484-cz64z"
Sep 07 13:00:22.483711 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.483636    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0af9d659-61d2-4066-ac26-78eec940e914-signing-key\") pod \"service-ca-75d99c484-cz64z\" (UID: \"0af9d659-61d2-4066-ac26-78eec940e914\") " pod="openshift-service-ca/service-ca-75d99c484-cz64z"
Sep 07 13:00:22.483711 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.483677    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:22.483781 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.483756    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkv6g\" (UniqueName: \"kubernetes.io/projected/0af9d659-61d2-4066-ac26-78eec940e914-kube-api-access-fkv6g\") pod \"service-ca-75d99c484-cz64z\" (UID: \"0af9d659-61d2-4066-ac26-78eec940e914\") " pod="openshift-service-ca/service-ca-75d99c484-cz64z"
Sep 07 13:00:22.483781 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:22.483774    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 13:00:22.483928 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:22.483820    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls podName:2469a9d8-5736-4c2b-a707-8128164910e8 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:23.483803223 +0000 UTC m=+67.912101474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzp8t" (UID: "2469a9d8-5736-4c2b-a707-8128164910e8") : secret "insights-runtime-extractor-tls" not found
Sep 07 13:00:22.483928 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.483877    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:22.484046 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:22.483982    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 13:00:22.484046 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:22.484029    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm podName:2469a9d8-5736-4c2b-a707-8128164910e8 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:23.484018603 +0000 UTC m=+67.912316855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzp8t" (UID: "2469a9d8-5736-4c2b-a707-8128164910e8") : configmap "kube-rbac-proxy" not found
Sep 07 13:00:22.484475 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.484455    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0af9d659-61d2-4066-ac26-78eec940e914-signing-cabundle\") pod \"service-ca-75d99c484-cz64z\" (UID: \"0af9d659-61d2-4066-ac26-78eec940e914\") " pod="openshift-service-ca/service-ca-75d99c484-cz64z"
Sep 07 13:00:22.486010 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.485990    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0af9d659-61d2-4066-ac26-78eec940e914-signing-key\") pod \"service-ca-75d99c484-cz64z\" (UID: \"0af9d659-61d2-4066-ac26-78eec940e914\") " pod="openshift-service-ca/service-ca-75d99c484-cz64z"
Sep 07 13:00:22.495955 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.495936    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkv6g\" (UniqueName: \"kubernetes.io/projected/0af9d659-61d2-4066-ac26-78eec940e914-kube-api-access-fkv6g\") pod \"service-ca-75d99c484-cz64z\" (UID: \"0af9d659-61d2-4066-ac26-78eec940e914\") " pod="openshift-service-ca/service-ca-75d99c484-cz64z"
Sep 07 13:00:22.652397 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.652363    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-cz64z"
Sep 07 13:00:22.932875 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:22.932813    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-cz64z"]
Sep 07 13:00:22.936475 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:00:22.936441    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af9d659_61d2_4066_ac26_78eec940e914.slice/crio-5a1eeac1f44ce6e1f282f999f68384c397392fa46a4189cb2af095927a27a3e4 WatchSource:0}: Error finding container 5a1eeac1f44ce6e1f282f999f68384c397392fa46a4189cb2af095927a27a3e4: Status 404 returned error can't find the container with id 5a1eeac1f44ce6e1f282f999f68384c397392fa46a4189cb2af095927a27a3e4
Sep 07 13:00:23.433644 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:23.433605    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p7mj7" event={"ID":"a2da8de4-8df5-4b96-9449-1c8d346c5fec","Type":"ContainerStarted","Data":"05adfa592ef1ede5937fa3299ff076b6fff16c0bd82d5568d70876f22890be56"}
Sep 07 13:00:23.435176 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:23.435136    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-njk59" event={"ID":"d1dbf089-2e69-485b-9e8b-eb30c69e15aa","Type":"ContainerStarted","Data":"c6abdcba839d3f48a01bbf67a7a492aca051c8a053d8554f12964539eb8a2721"}
Sep 07 13:00:23.437010 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:23.436980    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-cz64z" event={"ID":"0af9d659-61d2-4066-ac26-78eec940e914","Type":"ContainerStarted","Data":"a1c0ebf96355037ad7983106ebb3c4d9d832fe1b2725eb23e57b56336a8c4195"}
Sep 07 13:00:23.437128 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:23.437028    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-cz64z" event={"ID":"0af9d659-61d2-4066-ac26-78eec940e914","Type":"ContainerStarted","Data":"5a1eeac1f44ce6e1f282f999f68384c397392fa46a4189cb2af095927a27a3e4"}
Sep 07 13:00:23.493671 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:23.493607    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-cz64z" podStartSLOduration=1.493581437 podStartE2EDuration="1.493581437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:23.492644186 +0000 UTC m=+67.920942458" watchObservedRunningTime="2026-09-07 13:00:23.493581437 +0000 UTC m=+67.921879711"
Sep 07 13:00:23.493915 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:23.493887    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:23.494092 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:23.494044    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:23.494808 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:23.494787    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 13:00:23.494877 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:23.494854    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm podName:2469a9d8-5736-4c2b-a707-8128164910e8 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:25.494837321 +0000 UTC m=+69.923135589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzp8t" (UID: "2469a9d8-5736-4c2b-a707-8128164910e8") : configmap "kube-rbac-proxy" not found
Sep 07 13:00:23.494941 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:23.494929    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 13:00:23.494991 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:23.494978    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls podName:2469a9d8-5736-4c2b-a707-8128164910e8 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:25.494963128 +0000 UTC m=+69.923261386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzp8t" (UID: "2469a9d8-5736-4c2b-a707-8128164910e8") : secret "insights-runtime-extractor-tls" not found
Sep 07 13:00:24.545329 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:24.540654    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sqnx4_8f1b14f3-6875-4d07-859d-ada2d3bb1f0f/dns-node-resolver/0.log"
Sep 07 13:00:25.514461 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:25.514417    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:25.514642 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:25.514550    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:25.514755 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:25.514690    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 13:00:25.514755 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:25.514751    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls podName:2469a9d8-5736-4c2b-a707-8128164910e8 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:29.514731622 +0000 UTC m=+73.943029875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzp8t" (UID: "2469a9d8-5736-4c2b-a707-8128164910e8") : secret "insights-runtime-extractor-tls" not found
Sep 07 13:00:25.515186 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:25.515164    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 13:00:25.515370 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:25.515221    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm podName:2469a9d8-5736-4c2b-a707-8128164910e8 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:29.51520659 +0000 UTC m=+73.943504840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzp8t" (UID: "2469a9d8-5736-4c2b-a707-8128164910e8") : configmap "kube-rbac-proxy" not found
Sep 07 13:00:25.540932 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:25.540899    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kdkj7_ea2d8b03-e721-4102-846e-ee7d41d13e96/node-ca/0.log"
Sep 07 13:00:26.540526 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:26.540465    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xvkgl_4d36321e-8fe0-4c81-a143-d99bf792bd99/serve-healthcheck-canary/0.log"
Sep 07 13:00:27.146318 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:27.146281    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-2zrp5_d29420b3-de04-4ece-82d5-3cc4fbc8d56f/kube-storage-version-migrator-operator/0.log"
Sep 07 13:00:28.455144 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:28.455106    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p7mj7" event={"ID":"a2da8de4-8df5-4b96-9449-1c8d346c5fec","Type":"ContainerStarted","Data":"3589e78f6b041e3bf61e0e97b1ae6098a7e360e5681262c8812fc73576404689"}
Sep 07 13:00:28.455643 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:28.455209    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 13:00:28.456661 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:28.456634    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-njk59" event={"ID":"d1dbf089-2e69-485b-9e8b-eb30c69e15aa","Type":"ContainerStarted","Data":"1d2f796072b2d3dad67bf7b52c4055b8ae86d6ca822b388b0328c6809dffdb22"}
Sep 07 13:00:28.499991 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:28.499931    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-njk59" podStartSLOduration=67.248061049 podStartE2EDuration="1m12.499914194s" totalImagesPullingTime="5.251853145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:28.49988414 +0000 UTC m=+72.928182413" watchObservedRunningTime="2026-09-07 13:00:28.499914194 +0000 UTC m=+72.928212467"
Sep 07 13:00:28.500180 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:28.500012    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p7mj7" podStartSLOduration=67.281457901 podStartE2EDuration="1m12.50000741s" totalImagesPullingTime="5.218549509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:28.476981287 +0000 UTC m=+72.905279581" watchObservedRunningTime="2026-09-07 13:00:28.50000741 +0000 UTC m=+72.928305705"
Sep 07 13:00:29.553224 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:29.553179    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:29.553716 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:29.553269    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:29.553716 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:29.553337    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 13:00:29.553716 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:29.553348    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 13:00:29.553716 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:29.553410    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm podName:2469a9d8-5736-4c2b-a707-8128164910e8 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:37.553390279 +0000 UTC m=+81.981688550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzp8t" (UID: "2469a9d8-5736-4c2b-a707-8128164910e8") : configmap "kube-rbac-proxy" not found
Sep 07 13:00:29.553716 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:00:29.553428    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls podName:2469a9d8-5736-4c2b-a707-8128164910e8 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:37.55341977 +0000 UTC m=+81.981718022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzp8t" (UID: "2469a9d8-5736-4c2b-a707-8128164910e8") : secret "insights-runtime-extractor-tls" not found
Sep 07 13:00:29.654478 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:29.654440    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gtfhw\" (UID: \"c576bf35-1368-4bcb-9d09-69010dc2f9fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"
Sep 07 13:00:29.657149 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:29.657121    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c576bf35-1368-4bcb-9d09-69010dc2f9fe-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-gtfhw\" (UID: \"c576bf35-1368-4bcb-9d09-69010dc2f9fe\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"
Sep 07 13:00:29.812795 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:29.812704    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-b4ptt\""
Sep 07 13:00:29.820489 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:29.820466    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"
Sep 07 13:00:30.099439 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:30.099367    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw"]
Sep 07 13:00:30.464717 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:30.464634    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw" event={"ID":"c576bf35-1368-4bcb-9d09-69010dc2f9fe","Type":"ContainerStarted","Data":"dc6e654fb90349b0ab9301d5a70c99af70dd880329ca5b5f9131dd1a2bb6e2ef"}
Sep 07 13:00:32.472290 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:32.472193    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw" event={"ID":"c576bf35-1368-4bcb-9d09-69010dc2f9fe","Type":"ContainerStarted","Data":"abe034ca0d97c52c74e99c533f083515e4b0166c5e5a35c3fcda8fe958959341"}
Sep 07 13:00:32.472290 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:32.472250    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw" event={"ID":"c576bf35-1368-4bcb-9d09-69010dc2f9fe","Type":"ContainerStarted","Data":"7b9326e4d24186a356fc30c45a62b0919a5835af1629bf12372395d5989fdf62"}
Sep 07 13:00:32.498850 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:32.498794    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-gtfhw" podStartSLOduration=17.552745094 podStartE2EDuration="19.498777806s" totalImagesPullingTime="1.946032712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:32.498461633 +0000 UTC m=+76.926759906" watchObservedRunningTime="2026-09-07 13:00:32.498777806 +0000 UTC m=+76.927076078"
Sep 07 13:00:37.619038 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:37.618995    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:37.619547 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:37.619114    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:37.619707 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:37.619681    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2469a9d8-5736-4c2b-a707-8128164910e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:37.621437 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:37.621415    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2469a9d8-5736-4c2b-a707-8128164910e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzp8t\" (UID: \"2469a9d8-5736-4c2b-a707-8128164910e8\") " pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:37.674644 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:37.674603    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzp8t"
Sep 07 13:00:37.943700 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:37.943625    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzp8t"]
Sep 07 13:00:37.946721 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:00:37.946683    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2469a9d8_5736_4c2b_a707_8128164910e8.slice/crio-cc56b9a64eeead7822cfd1256eb74f75c9a88008fde505e3cdd32d2555831117 WatchSource:0}: Error finding container cc56b9a64eeead7822cfd1256eb74f75c9a88008fde505e3cdd32d2555831117: Status 404 returned error can't find the container with id cc56b9a64eeead7822cfd1256eb74f75c9a88008fde505e3cdd32d2555831117
Sep 07 13:00:38.492760 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:38.492723    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzp8t" event={"ID":"2469a9d8-5736-4c2b-a707-8128164910e8","Type":"ContainerStarted","Data":"e417e6178aeff2a94d8449af0147bbe3fb470326b3bb9761ffd3bbced4d27d0d"}
Sep 07 13:00:38.492922 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:38.492787    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzp8t" event={"ID":"2469a9d8-5736-4c2b-a707-8128164910e8","Type":"ContainerStarted","Data":"cc56b9a64eeead7822cfd1256eb74f75c9a88008fde505e3cdd32d2555831117"}
Sep 07 13:00:39.497304 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:39.497267    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzp8t" event={"ID":"2469a9d8-5736-4c2b-a707-8128164910e8","Type":"ContainerStarted","Data":"9210e9a315e2c55f49123f73dbc8ddfd375ec400a62f9e353a3bdb7a700be299"}
Sep 07 13:00:41.506917 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:41.506879    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzp8t" event={"ID":"2469a9d8-5736-4c2b-a707-8128164910e8","Type":"ContainerStarted","Data":"93f71b588440a38e3b6c9dcae81490c7342a97615fff5e1e346129d6d28f11ca"}
Sep 07 13:00:41.528678 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:41.528621    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hzp8t" podStartSLOduration=17.984468868 podStartE2EDuration="20.528605423s" totalImagesPullingTime="2.544136555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 13:00:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:41.528465256 +0000 UTC m=+85.956763547" watchObservedRunningTime="2026-09-07 13:00:41.528605423 +0000 UTC m=+85.956903696"
Sep 07 13:00:46.039912 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.039872    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"]
Sep 07 13:00:46.044586 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.044563    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.046416 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.046389    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 07 13:00:46.046550 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.046443    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 07 13:00:46.046760 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.046745    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 07 13:00:46.046807 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.046757    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 07 13:00:46.046997 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.046982    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 07 13:00:46.047070 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.047018    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 07 13:00:46.047127 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.047088    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 07 13:00:46.058915 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.058894    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"]
Sep 07 13:00:46.188632 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.188596    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.188785 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.188649    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sp7s\" (UniqueName: \"kubernetes.io/projected/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-kube-api-access-9sp7s\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.188785 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.188704    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-ca\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.188785 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.188764    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.188884 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.188783    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.188884 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.188807    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-hub\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.289616 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.289578    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sp7s\" (UniqueName: \"kubernetes.io/projected/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-kube-api-access-9sp7s\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.289616 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.289619    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-ca\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.289859 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.289666    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.289859 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.289682    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.289859 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.289704    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-hub\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.289859 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.289742    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.290597 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.290528    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.292391 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.292359    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-ca\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.292499 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.292469    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.292499 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.292483    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-hub\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.292609 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.292593    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.310888 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.310865    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sp7s\" (UniqueName: \"kubernetes.io/projected/f2cdc56c-ee6e-4780-b2c5-5327f94714c1-kube-api-access-9sp7s\") pod \"cluster-proxy-proxy-agent-cc98b9d97-k2cn4\" (UID: \"f2cdc56c-ee6e-4780-b2c5-5327f94714c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.363873 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.363841    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"
Sep 07 13:00:46.636102 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:46.636013    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4"]
Sep 07 13:00:46.638777 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:00:46.638752    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2cdc56c_ee6e_4780_b2c5_5327f94714c1.slice/crio-ba0a5ce74346d1fcd0e19f17997abc52903674452c91700f19f3745d3b7f81de WatchSource:0}: Error finding container ba0a5ce74346d1fcd0e19f17997abc52903674452c91700f19f3745d3b7f81de: Status 404 returned error can't find the container with id ba0a5ce74346d1fcd0e19f17997abc52903674452c91700f19f3745d3b7f81de
Sep 07 13:00:47.525668 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:47.525629    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4" event={"ID":"f2cdc56c-ee6e-4780-b2c5-5327f94714c1","Type":"ContainerStarted","Data":"ba0a5ce74346d1fcd0e19f17997abc52903674452c91700f19f3745d3b7f81de"}
Sep 07 13:00:51.541732 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:51.541694    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4" event={"ID":"f2cdc56c-ee6e-4780-b2c5-5327f94714c1","Type":"ContainerStarted","Data":"0e519cc15512ec313ac2d4fb922958d5ec1c120660c9294aee2c52e2f60aa4c6"}
Sep 07 13:00:51.542151 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:51.541747    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4" event={"ID":"f2cdc56c-ee6e-4780-b2c5-5327f94714c1","Type":"ContainerStarted","Data":"b293dfbe87c7cd9b1c6a2c526a0e140d8a9cf66528b922172ec232bee9a1125c"}
Sep 07 13:00:51.542151 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:51.541768    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4" event={"ID":"f2cdc56c-ee6e-4780-b2c5-5327f94714c1","Type":"ContainerStarted","Data":"4a6f24bc688acbb475652701e45e76455405fac4920655cd981a6ff7ba928331"}
Sep 07 13:00:51.561909 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:51.561864    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cc98b9d97-k2cn4" podStartSLOduration=1.457549845 podStartE2EDuration="5.561849534s" totalImagesPullingTime="4.104299689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:51.561739992 +0000 UTC m=+95.990038267" watchObservedRunningTime="2026-09-07 13:00:51.561849534 +0000 UTC m=+95.990147806"
Sep 07 13:00:53.854251 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:53.854213    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 13:00:53.854586 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:53.854274    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 13:00:53.856484 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:53.856452    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c700ee4-1744-4956-9728-eb2911022015-metrics-tls\") pod \"dns-default-g54bn\" (UID: \"6c700ee4-1744-4956-9728-eb2911022015\") " pod="openshift-dns/dns-default-g54bn"
Sep 07 13:00:53.856606 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:53.856587    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls\") pod \"image-registry-d759cdf6-xw4nh\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") " pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 13:00:53.883815 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:53.883791    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rqgx2\""
Sep 07 13:00:53.892541 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:53.892524    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g54bn"
Sep 07 13:00:54.157479 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:54.157449    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dn6gs\""
Sep 07 13:00:54.166399 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:54.166366    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 13:00:54.167679 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:54.167653    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g54bn"]
Sep 07 13:00:54.170890 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:00:54.170869    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c700ee4_1744_4956_9728_eb2911022015.slice/crio-aa6f008bdf783c229af860183f335a3274d66ee3d259e1f5b5008f69c69b0d36 WatchSource:0}: Error finding container aa6f008bdf783c229af860183f335a3274d66ee3d259e1f5b5008f69c69b0d36: Status 404 returned error can't find the container with id aa6f008bdf783c229af860183f335a3274d66ee3d259e1f5b5008f69c69b0d36
Sep 07 13:00:54.439097 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:54.439051    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d759cdf6-xw4nh"]
Sep 07 13:00:54.442259 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:00:54.442235    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50a32a2_1a5f_437b_9813_5b9c0a4d4ca8.slice/crio-6733b7061f0cbc1f3160f61ce4c90f08a7839ff290446fe8a63102df52fe33e1 WatchSource:0}: Error finding container 6733b7061f0cbc1f3160f61ce4c90f08a7839ff290446fe8a63102df52fe33e1: Status 404 returned error can't find the container with id 6733b7061f0cbc1f3160f61ce4c90f08a7839ff290446fe8a63102df52fe33e1
Sep 07 13:00:54.552559 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:54.552515    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d759cdf6-xw4nh" event={"ID":"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8","Type":"ContainerStarted","Data":"3c050cca63675b1b3de009b76135e24d8e216a8d38ac902bd99c2bc3396b398b"}
Sep 07 13:00:54.552717 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:54.552570    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d759cdf6-xw4nh" event={"ID":"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8","Type":"ContainerStarted","Data":"6733b7061f0cbc1f3160f61ce4c90f08a7839ff290446fe8a63102df52fe33e1"}
Sep 07 13:00:54.552717 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:54.552604    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 13:00:54.553845 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:54.553815    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g54bn" event={"ID":"6c700ee4-1744-4956-9728-eb2911022015","Type":"ContainerStarted","Data":"aa6f008bdf783c229af860183f335a3274d66ee3d259e1f5b5008f69c69b0d36"}
Sep 07 13:00:54.575427 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:54.575367    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d759cdf6-xw4nh" podStartSLOduration=98.575343314 podStartE2EDuration="1m38.575343314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:54.573500514 +0000 UTC m=+99.001798833" watchObservedRunningTime="2026-09-07 13:00:54.575343314 +0000 UTC m=+99.003641588"
Sep 07 13:00:56.565192 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:56.565151    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g54bn" event={"ID":"6c700ee4-1744-4956-9728-eb2911022015","Type":"ContainerStarted","Data":"e64eb481f3391ae75eedc1ae752809302aac88926518346b96aca6c398c00c8a"}
Sep 07 13:00:56.565665 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:56.565240    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g54bn" event={"ID":"6c700ee4-1744-4956-9728-eb2911022015","Type":"ContainerStarted","Data":"9b60b1165b288463cf8e0c46b054275b628dd45ca9b987b25070d2a8f2796129"}
Sep 07 13:00:56.565665 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:56.565282    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g54bn"
Sep 07 13:00:56.586626 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:56.586581    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-g54bn" podStartSLOduration=65.667908993 podStartE2EDuration="1m7.586567096s" totalImagesPullingTime="1.918658103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:56.586051273 +0000 UTC m=+101.014349547" watchObservedRunningTime="2026-09-07 13:00:56.586567096 +0000 UTC m=+101.014865395"
Sep 07 13:00:59.463222 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:00:59.463188    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p7mj7"
Sep 07 13:01:06.571303 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:06.571272    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g54bn"
Sep 07 13:01:08.218605 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:08.218573    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d759cdf6-xw4nh"]
Sep 07 13:01:13.239805 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.239772    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9lc2x"]
Sep 07 13:01:13.242368 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.242352    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.248619 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.248325    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 07 13:01:13.248872 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.248850    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 07 13:01:13.249448 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.249426    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 07 13:01:13.249448 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.249438    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2jq42\""
Sep 07 13:01:13.249606 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.249428    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 07 13:01:13.249606 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.249516    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 07 13:01:13.249606 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.249574    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 07 13:01:13.294290 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.294258    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlx2c\" (UniqueName: \"kubernetes.io/projected/e473f2b3-d603-4b42-b40d-3616dbcbb50b-kube-api-access-dlx2c\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.294445 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.294316    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.294445 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.294370    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-tls\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.294445 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.294417    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-wtmp\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.294445 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.294446    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e473f2b3-d603-4b42-b40d-3616dbcbb50b-metrics-client-ca\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.294620 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.294469    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.294620 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.294506    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e473f2b3-d603-4b42-b40d-3616dbcbb50b-root\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.294620 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.294569    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e473f2b3-d603-4b42-b40d-3616dbcbb50b-sys\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.294620 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.294583    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-textfile\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.395426 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.395393    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.395426 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.395431    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-tls\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.395678 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.395451    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-wtmp\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.395678 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.395474    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e473f2b3-d603-4b42-b40d-3616dbcbb50b-metrics-client-ca\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.395678 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.395501    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.395678 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.395540    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e473f2b3-d603-4b42-b40d-3616dbcbb50b-root\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.395678 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.395602    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e473f2b3-d603-4b42-b40d-3616dbcbb50b-sys\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.395678 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.395617    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-textfile\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.395678 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.395635    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-wtmp\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.395678 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.395658    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlx2c\" (UniqueName: \"kubernetes.io/projected/e473f2b3-d603-4b42-b40d-3616dbcbb50b-kube-api-access-dlx2c\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.396090 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.395704    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e473f2b3-d603-4b42-b40d-3616dbcbb50b-root\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.396090 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.395759    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e473f2b3-d603-4b42-b40d-3616dbcbb50b-sys\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.396206 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.396104    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-textfile\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.396206 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.396120    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.396273 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.396244    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e473f2b3-d603-4b42-b40d-3616dbcbb50b-metrics-client-ca\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.398392 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.398369    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-tls\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.398392 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.398385    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e473f2b3-d603-4b42-b40d-3616dbcbb50b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.415003 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.414978    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlx2c\" (UniqueName: \"kubernetes.io/projected/e473f2b3-d603-4b42-b40d-3616dbcbb50b-kube-api-access-dlx2c\") pod \"node-exporter-9lc2x\" (UID: \"e473f2b3-d603-4b42-b40d-3616dbcbb50b\") " pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.552081 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.552015    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9lc2x"
Sep 07 13:01:13.561398 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:01:13.561375    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode473f2b3_d603_4b42_b40d_3616dbcbb50b.slice/crio-53de02fdea575c20ca2977391ce485b0b10dbd12db9c8a360b5e694cb00f2195 WatchSource:0}: Error finding container 53de02fdea575c20ca2977391ce485b0b10dbd12db9c8a360b5e694cb00f2195: Status 404 returned error can't find the container with id 53de02fdea575c20ca2977391ce485b0b10dbd12db9c8a360b5e694cb00f2195
Sep 07 13:01:13.620022 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:13.619994    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lc2x" event={"ID":"e473f2b3-d603-4b42-b40d-3616dbcbb50b","Type":"ContainerStarted","Data":"53de02fdea575c20ca2977391ce485b0b10dbd12db9c8a360b5e694cb00f2195"}
Sep 07 13:01:15.628891 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:15.628855    2567 generic.go:356] "Generic (PLEG): container finished" podID="e473f2b3-d603-4b42-b40d-3616dbcbb50b" containerID="63b2ac93a02d1c7b513f063acd949d65555db273b2fae211d29cfcae9ccfe9c6" exitCode=0
Sep 07 13:01:15.629311 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:15.628946    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lc2x" event={"ID":"e473f2b3-d603-4b42-b40d-3616dbcbb50b","Type":"ContainerDied","Data":"63b2ac93a02d1c7b513f063acd949d65555db273b2fae211d29cfcae9ccfe9c6"}
Sep 07 13:01:16.635031 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:16.634997    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lc2x" event={"ID":"e473f2b3-d603-4b42-b40d-3616dbcbb50b","Type":"ContainerStarted","Data":"7240832c40a6f0d58a9e46e5976d0577b6677ab77cf6782d8685d8ba0b32dce3"}
Sep 07 13:01:16.635447 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:16.635048    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lc2x" event={"ID":"e473f2b3-d603-4b42-b40d-3616dbcbb50b","Type":"ContainerStarted","Data":"15bbd218d06079f081585411910a793bdbc512b472ce97109f2735952560f361"}
Sep 07 13:01:16.656902 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:16.656849    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9lc2x" podStartSLOduration=2.493203465 podStartE2EDuration="3.65683222s" totalImagesPullingTime="1.101550174s" totalInitContainerRuntime="62.078581ms" isStatefulPod=true podCreationTimestamp="2026-09-07 13:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:01:16.655419393 +0000 UTC m=+121.083717667" watchObservedRunningTime="2026-09-07 13:01:16.65683222 +0000 UTC m=+121.085130493"
Sep 07 13:01:18.224537 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:18.224506    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 13:01:25.793535 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:25.793473    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 13:01:25.795828 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:25.795804    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79dcf5c-ddc9-47c8-b61e-96fd4401c89f-metrics-certs\") pod \"network-metrics-daemon-tsqvk\" (UID: \"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f\") " pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 13:01:26.089683 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:26.089604    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5v9xw\""
Sep 07 13:01:26.098182 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:26.098156    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tsqvk"
Sep 07 13:01:26.370782 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:26.370706    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tsqvk"]
Sep 07 13:01:26.374029 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:01:26.374005    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79dcf5c_ddc9_47c8_b61e_96fd4401c89f.slice/crio-6189905f1acb2503c6d7b61b086c3cd11d1a88036c689c08b1e250bd1cf20144 WatchSource:0}: Error finding container 6189905f1acb2503c6d7b61b086c3cd11d1a88036c689c08b1e250bd1cf20144: Status 404 returned error can't find the container with id 6189905f1acb2503c6d7b61b086c3cd11d1a88036c689c08b1e250bd1cf20144
Sep 07 13:01:26.667850 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:26.667749    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tsqvk" event={"ID":"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f","Type":"ContainerStarted","Data":"6189905f1acb2503c6d7b61b086c3cd11d1a88036c689c08b1e250bd1cf20144"}
Sep 07 13:01:27.674022 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:27.673924    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tsqvk" event={"ID":"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f","Type":"ContainerStarted","Data":"986523b6a7b592780edb5327d0315d46df48685611a905acfbd0751f8ad6d1df"}
Sep 07 13:01:27.674022 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:27.673987    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tsqvk" event={"ID":"f79dcf5c-ddc9-47c8-b61e-96fd4401c89f","Type":"ContainerStarted","Data":"5fc7a0e4189120027d31b47c83abf1c8853d740e90b7b824c1f50b5f6c017d73"}
Sep 07 13:01:27.696420 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:27.696361    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tsqvk" podStartSLOduration=130.704996272 podStartE2EDuration="2m11.696338443s" totalImagesPullingTime="991.342171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:01:27.693407309 +0000 UTC m=+132.121705581" watchObservedRunningTime="2026-09-07 13:01:27.696338443 +0000 UTC m=+132.124636718"
Sep 07 13:01:33.237601 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.237554    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d759cdf6-xw4nh" podUID="d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" containerName="registry" containerID="cri-o://3c050cca63675b1b3de009b76135e24d8e216a8d38ac902bd99c2bc3396b398b" gracePeriod=30
Sep 07 13:01:33.476900 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.476874    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 13:01:33.553139 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.553100    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-trusted-ca\") pod \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") "
Sep 07 13:01:33.553318 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.553148    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-certificates\") pod \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") "
Sep 07 13:01:33.553318 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.553178    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls\") pod \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") "
Sep 07 13:01:33.553318 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.553199    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-installation-pull-secrets\") pod \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") "
Sep 07 13:01:33.553444 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.553322    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-ca-trust-extracted\") pod \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") "
Sep 07 13:01:33.553444 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.553361    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-bound-sa-token\") pod \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") "
Sep 07 13:01:33.553444 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.553395    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-image-registry-private-configuration\") pod \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") "
Sep 07 13:01:33.553444 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.553432    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkx7r\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-kube-api-access-tkx7r\") pod \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\" (UID: \"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8\") "
Sep 07 13:01:33.553673 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.553626    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:01:33.553673 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.553651    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:01:33.555783 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.555750    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:01:33.556004 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.555896    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:01:33.556004 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.555933    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:01:33.556159 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.556078    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:01:33.556248 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.556229    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-kube-api-access-tkx7r" (OuterVolumeSpecName: "kube-api-access-tkx7r") pod "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8"). InnerVolumeSpecName "kube-api-access-tkx7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:01:33.562171 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.562140    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" (UID: "d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:01:33.654281 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.654224    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-trusted-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:01:33.654281 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.654273    2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-certificates\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:01:33.654281 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.654284    2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-registry-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:01:33.654281 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.654293    2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-installation-pull-secrets\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:01:33.654281 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.654303    2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-ca-trust-extracted\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:01:33.654585 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.654311    2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-bound-sa-token\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:01:33.654585 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.654322    2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-image-registry-private-configuration\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:01:33.654585 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.654331    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkx7r\" (UniqueName: \"kubernetes.io/projected/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8-kube-api-access-tkx7r\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:01:33.695146 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.695103    2567 generic.go:356] "Generic (PLEG): container finished" podID="d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" containerID="3c050cca63675b1b3de009b76135e24d8e216a8d38ac902bd99c2bc3396b398b" exitCode=0
Sep 07 13:01:33.695303 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.695162    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d759cdf6-xw4nh"
Sep 07 13:01:33.695303 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.695194    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d759cdf6-xw4nh" event={"ID":"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8","Type":"ContainerDied","Data":"3c050cca63675b1b3de009b76135e24d8e216a8d38ac902bd99c2bc3396b398b"}
Sep 07 13:01:33.695303 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.695246    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d759cdf6-xw4nh" event={"ID":"d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8","Type":"ContainerDied","Data":"6733b7061f0cbc1f3160f61ce4c90f08a7839ff290446fe8a63102df52fe33e1"}
Sep 07 13:01:33.695303 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.695266    2567 scope.go:117] "RemoveContainer" containerID="3c050cca63675b1b3de009b76135e24d8e216a8d38ac902bd99c2bc3396b398b"
Sep 07 13:01:33.703573 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.703549    2567 scope.go:117] "RemoveContainer" containerID="3c050cca63675b1b3de009b76135e24d8e216a8d38ac902bd99c2bc3396b398b"
Sep 07 13:01:33.703869 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:01:33.703847    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c050cca63675b1b3de009b76135e24d8e216a8d38ac902bd99c2bc3396b398b\": container with ID starting with 3c050cca63675b1b3de009b76135e24d8e216a8d38ac902bd99c2bc3396b398b not found: ID does not exist" containerID="3c050cca63675b1b3de009b76135e24d8e216a8d38ac902bd99c2bc3396b398b"
Sep 07 13:01:33.703916 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.703878    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c050cca63675b1b3de009b76135e24d8e216a8d38ac902bd99c2bc3396b398b"} err="failed to get container status \"3c050cca63675b1b3de009b76135e24d8e216a8d38ac902bd99c2bc3396b398b\": rpc error: code = NotFound desc = could not find container \"3c050cca63675b1b3de009b76135e24d8e216a8d38ac902bd99c2bc3396b398b\": container with ID starting with 3c050cca63675b1b3de009b76135e24d8e216a8d38ac902bd99c2bc3396b398b not found: ID does not exist"
Sep 07 13:01:33.721750 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.721723    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d759cdf6-xw4nh"]
Sep 07 13:01:33.731508 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:33.731479    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d759cdf6-xw4nh"]
Sep 07 13:01:34.074139 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:34.074106    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" path="/var/lib/kubelet/pods/d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8/volumes"
Sep 07 13:01:34.700173 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:34.700133    2567 generic.go:356] "Generic (PLEG): container finished" podID="f4572efa-98bc-4777-a77e-662b2481229a" containerID="51317606daf62c33866752f44d374ca086aeb6878880370330fe1c50d9ccbc19" exitCode=0
Sep 07 13:01:34.700600 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:34.700206    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk" event={"ID":"f4572efa-98bc-4777-a77e-662b2481229a","Type":"ContainerDied","Data":"51317606daf62c33866752f44d374ca086aeb6878880370330fe1c50d9ccbc19"}
Sep 07 13:01:34.700600 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:34.700538    2567 scope.go:117] "RemoveContainer" containerID="51317606daf62c33866752f44d374ca086aeb6878880370330fe1c50d9ccbc19"
Sep 07 13:01:35.706309 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:35.706275    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l8qsk" event={"ID":"f4572efa-98bc-4777-a77e-662b2481229a","Type":"ContainerStarted","Data":"33695c85b41eb59506c898bfdf84283d154c4215c8eeb04e8e95bac33fd8b771"}
Sep 07 13:01:38.716693 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:38.716655    2567 generic.go:356] "Generic (PLEG): container finished" podID="d29420b3-de04-4ece-82d5-3cc4fbc8d56f" containerID="7a4d7aedf63e33e6bbc82e6c038b0ecf95a67aaf850b5a6c84e3c38c04bbb405" exitCode=0
Sep 07 13:01:38.717153 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:38.716729    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5" event={"ID":"d29420b3-de04-4ece-82d5-3cc4fbc8d56f","Type":"ContainerDied","Data":"7a4d7aedf63e33e6bbc82e6c038b0ecf95a67aaf850b5a6c84e3c38c04bbb405"}
Sep 07 13:01:38.717153 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:38.717054    2567 scope.go:117] "RemoveContainer" containerID="7a4d7aedf63e33e6bbc82e6c038b0ecf95a67aaf850b5a6c84e3c38c04bbb405"
Sep 07 13:01:39.722072 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:01:39.722026    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2zrp5" event={"ID":"d29420b3-de04-4ece-82d5-3cc4fbc8d56f","Type":"ContainerStarted","Data":"91694b09d6b1a7c7a6b9f09965196c952a6c8abeb2488979a09edb43ed6c4c03"}
Sep 07 13:04:15.993503 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:04:15.993477    2567 kubelet.go:1648] "Image garbage collection succeeded"
Sep 07 13:05:08.882191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:08.882110    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"]
Sep 07 13:05:08.882597 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:08.882365    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" containerName="registry"
Sep 07 13:05:08.882597 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:08.882379    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" containerName="registry"
Sep 07 13:05:08.882597 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:08.882427    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50a32a2-1a5f-437b-9813-5b9c0a4d4ca8" containerName="registry"
Sep 07 13:05:08.885210 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:08.885193    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"
Sep 07 13:05:08.887382 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:08.887354    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4mdkr\""
Sep 07 13:05:08.887486 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:08.887354    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 13:05:08.887991 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:08.887976    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 13:05:08.897281 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:08.897263    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"]
Sep 07 13:05:08.937358 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:08.937329    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf\" (UID: \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"
Sep 07 13:05:08.937501 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:08.937369    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf\" (UID: \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"
Sep 07 13:05:08.937501 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:08.937408    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc887\" (UniqueName: \"kubernetes.io/projected/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-kube-api-access-sc887\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf\" (UID: \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"
Sep 07 13:05:09.037974 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:09.037942    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf\" (UID: \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"
Sep 07 13:05:09.038158 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:09.037982    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf\" (UID: \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"
Sep 07 13:05:09.038158 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:09.038023    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc887\" (UniqueName: \"kubernetes.io/projected/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-kube-api-access-sc887\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf\" (UID: \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"
Sep 07 13:05:09.038340 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:09.038317    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf\" (UID: \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"
Sep 07 13:05:09.038410 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:09.038357    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf\" (UID: \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"
Sep 07 13:05:09.047163 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:09.047142    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc887\" (UniqueName: \"kubernetes.io/projected/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-kube-api-access-sc887\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf\" (UID: \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"
Sep 07 13:05:09.193650 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:09.193574    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"
Sep 07 13:05:09.464429 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:09.464362    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"]
Sep 07 13:05:09.467032 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:05:09.467005    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a456add_2f0e_4ce3_9b5c_caffbe73b10c.slice/crio-3f4888a42d288ed203b0ad8d05eb0b96f7485b28ebb8cb1954b767007e354409 WatchSource:0}: Error finding container 3f4888a42d288ed203b0ad8d05eb0b96f7485b28ebb8cb1954b767007e354409: Status 404 returned error can't find the container with id 3f4888a42d288ed203b0ad8d05eb0b96f7485b28ebb8cb1954b767007e354409
Sep 07 13:05:09.468900 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:09.468885    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 13:05:10.427528 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:10.427484    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf" event={"ID":"3a456add-2f0e-4ce3-9b5c-caffbe73b10c","Type":"ContainerStarted","Data":"3f4888a42d288ed203b0ad8d05eb0b96f7485b28ebb8cb1954b767007e354409"}
Sep 07 13:05:17.454654 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:17.454618    2567 generic.go:356] "Generic (PLEG): container finished" podID="3a456add-2f0e-4ce3-9b5c-caffbe73b10c" containerID="7fe56984700fa6f029bd0348c08c65ef0a2992607b090550d4f724598771402a" exitCode=0
Sep 07 13:05:17.455045 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:17.454711    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf" event={"ID":"3a456add-2f0e-4ce3-9b5c-caffbe73b10c","Type":"ContainerDied","Data":"7fe56984700fa6f029bd0348c08c65ef0a2992607b090550d4f724598771402a"}
Sep 07 13:05:20.469337 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:20.469302    2567 generic.go:356] "Generic (PLEG): container finished" podID="3a456add-2f0e-4ce3-9b5c-caffbe73b10c" containerID="3315ed896b1626664763bce6d99875ade75c4b7aa7c7ef7763b08dd232af5847" exitCode=0
Sep 07 13:05:20.469801 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:20.469355    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf" event={"ID":"3a456add-2f0e-4ce3-9b5c-caffbe73b10c","Type":"ContainerDied","Data":"3315ed896b1626664763bce6d99875ade75c4b7aa7c7ef7763b08dd232af5847"}
Sep 07 13:05:27.498510 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:27.498479    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf" event={"ID":"3a456add-2f0e-4ce3-9b5c-caffbe73b10c","Type":"ContainerStarted","Data":"a66674fe3e81276102c3abb9df39f64a3ea1a22c1fb960a3e9959dedfa24d78c"}
Sep 07 13:05:27.518353 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:27.518277    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf" podStartSLOduration=2.9536935890000002 podStartE2EDuration="19.518258889s" totalImagesPullingTime="16.5645653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:05:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:05:27.517303582 +0000 UTC m=+371.945601855" watchObservedRunningTime="2026-09-07 13:05:27.518258889 +0000 UTC m=+371.946557179"
Sep 07 13:05:28.503848 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:28.503808    2567 generic.go:356] "Generic (PLEG): container finished" podID="3a456add-2f0e-4ce3-9b5c-caffbe73b10c" containerID="a66674fe3e81276102c3abb9df39f64a3ea1a22c1fb960a3e9959dedfa24d78c" exitCode=0
Sep 07 13:05:28.504342 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:28.503898    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf" event={"ID":"3a456add-2f0e-4ce3-9b5c-caffbe73b10c","Type":"ContainerDied","Data":"a66674fe3e81276102c3abb9df39f64a3ea1a22c1fb960a3e9959dedfa24d78c"}
Sep 07 13:05:29.635949 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:29.635925    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"
Sep 07 13:05:29.688550 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:29.688518    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-bundle\") pod \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\" (UID: \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\") "
Sep 07 13:05:29.688690 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:29.688606    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc887\" (UniqueName: \"kubernetes.io/projected/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-kube-api-access-sc887\") pod \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\" (UID: \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\") "
Sep 07 13:05:29.688690 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:29.688652    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-util\") pod \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\" (UID: \"3a456add-2f0e-4ce3-9b5c-caffbe73b10c\") "
Sep 07 13:05:29.689304 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:29.689274    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-bundle" (OuterVolumeSpecName: "bundle") pod "3a456add-2f0e-4ce3-9b5c-caffbe73b10c" (UID: "3a456add-2f0e-4ce3-9b5c-caffbe73b10c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:05:29.690740 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:29.690720    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-kube-api-access-sc887" (OuterVolumeSpecName: "kube-api-access-sc887") pod "3a456add-2f0e-4ce3-9b5c-caffbe73b10c" (UID: "3a456add-2f0e-4ce3-9b5c-caffbe73b10c"). InnerVolumeSpecName "kube-api-access-sc887". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:05:29.693384 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:29.693363    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-util" (OuterVolumeSpecName: "util") pod "3a456add-2f0e-4ce3-9b5c-caffbe73b10c" (UID: "3a456add-2f0e-4ce3-9b5c-caffbe73b10c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:05:29.789672 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:29.789645    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-bundle\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:05:29.789672 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:29.789672    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc887\" (UniqueName: \"kubernetes.io/projected/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-kube-api-access-sc887\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:05:29.789839 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:29.789682    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a456add-2f0e-4ce3-9b5c-caffbe73b10c-util\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:05:30.513004 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:30.512930    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf" event={"ID":"3a456add-2f0e-4ce3-9b5c-caffbe73b10c","Type":"ContainerDied","Data":"3f4888a42d288ed203b0ad8d05eb0b96f7485b28ebb8cb1954b767007e354409"}
Sep 07 13:05:30.513004 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:30.512970    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mw8lf"
Sep 07 13:05:30.513204 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:30.512977    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f4888a42d288ed203b0ad8d05eb0b96f7485b28ebb8cb1954b767007e354409"
Sep 07 13:05:37.450171 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.450133    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk"]
Sep 07 13:05:37.450546 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.450366    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a456add-2f0e-4ce3-9b5c-caffbe73b10c" containerName="util"
Sep 07 13:05:37.450546 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.450377    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a456add-2f0e-4ce3-9b5c-caffbe73b10c" containerName="util"
Sep 07 13:05:37.450546 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.450392    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a456add-2f0e-4ce3-9b5c-caffbe73b10c" containerName="extract"
Sep 07 13:05:37.450546 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.450398    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a456add-2f0e-4ce3-9b5c-caffbe73b10c" containerName="extract"
Sep 07 13:05:37.450546 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.450406    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a456add-2f0e-4ce3-9b5c-caffbe73b10c" containerName="pull"
Sep 07 13:05:37.450546 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.450412    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a456add-2f0e-4ce3-9b5c-caffbe73b10c" containerName="pull"
Sep 07 13:05:37.450546 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.450449    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a456add-2f0e-4ce3-9b5c-caffbe73b10c" containerName="extract"
Sep 07 13:05:37.454356 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.454335    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk"
Sep 07 13:05:37.456912 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.456891    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 07 13:05:37.457096 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.457083    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 07 13:05:37.457368 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.457349    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-nnbwh\""
Sep 07 13:05:37.474490 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.474460    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk"]
Sep 07 13:05:37.549601 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.549570    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch57j\" (UniqueName: \"kubernetes.io/projected/a4d9cdad-af30-470e-ba5f-e1b6a916147c-kube-api-access-ch57j\") pod \"cert-manager-operator-controller-manager-55c444bf86-jllpk\" (UID: \"a4d9cdad-af30-470e-ba5f-e1b6a916147c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk"
Sep 07 13:05:37.549763 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.549623    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4d9cdad-af30-470e-ba5f-e1b6a916147c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jllpk\" (UID: \"a4d9cdad-af30-470e-ba5f-e1b6a916147c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk"
Sep 07 13:05:37.649998 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.649969    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4d9cdad-af30-470e-ba5f-e1b6a916147c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jllpk\" (UID: \"a4d9cdad-af30-470e-ba5f-e1b6a916147c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk"
Sep 07 13:05:37.650165 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.650022    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch57j\" (UniqueName: \"kubernetes.io/projected/a4d9cdad-af30-470e-ba5f-e1b6a916147c-kube-api-access-ch57j\") pod \"cert-manager-operator-controller-manager-55c444bf86-jllpk\" (UID: \"a4d9cdad-af30-470e-ba5f-e1b6a916147c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk"
Sep 07 13:05:37.650384 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.650361    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4d9cdad-af30-470e-ba5f-e1b6a916147c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jllpk\" (UID: \"a4d9cdad-af30-470e-ba5f-e1b6a916147c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk"
Sep 07 13:05:37.658565 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.658542    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch57j\" (UniqueName: \"kubernetes.io/projected/a4d9cdad-af30-470e-ba5f-e1b6a916147c-kube-api-access-ch57j\") pod \"cert-manager-operator-controller-manager-55c444bf86-jllpk\" (UID: \"a4d9cdad-af30-470e-ba5f-e1b6a916147c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk"
Sep 07 13:05:37.763841 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:37.763755    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk"
Sep 07 13:05:38.039869 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:38.039837    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk"]
Sep 07 13:05:38.043761 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:05:38.043732    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d9cdad_af30_470e_ba5f_e1b6a916147c.slice/crio-670d0aa25ac6909e3779c8a374edeacc6c4e5b279901a7eb58d5008307f30bef WatchSource:0}: Error finding container 670d0aa25ac6909e3779c8a374edeacc6c4e5b279901a7eb58d5008307f30bef: Status 404 returned error can't find the container with id 670d0aa25ac6909e3779c8a374edeacc6c4e5b279901a7eb58d5008307f30bef
Sep 07 13:05:38.541299 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:38.541267    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk" event={"ID":"a4d9cdad-af30-470e-ba5f-e1b6a916147c","Type":"ContainerStarted","Data":"670d0aa25ac6909e3779c8a374edeacc6c4e5b279901a7eb58d5008307f30bef"}
Sep 07 13:05:40.551249 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:40.551209    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk" event={"ID":"a4d9cdad-af30-470e-ba5f-e1b6a916147c","Type":"ContainerStarted","Data":"03e4c9f8c65321a52e69b2a1525b13c08a08b70f0bcbe46248cec4336cf393c2"}
Sep 07 13:05:40.577434 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:40.574991    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jllpk" podStartSLOduration=1.4937894329999999 podStartE2EDuration="3.574970053s" totalImagesPullingTime="2.08118062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:05:40.572785541 +0000 UTC m=+385.001083839" watchObservedRunningTime="2026-09-07 13:05:40.574970053 +0000 UTC m=+385.003268304"
Sep 07 13:05:41.286991 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.286947    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"]
Sep 07 13:05:41.289301 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.289284    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"
Sep 07 13:05:41.293389 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.293366    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4mdkr\""
Sep 07 13:05:41.293789 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.293773    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 13:05:41.294907 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.294886    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 13:05:41.316894 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.316869    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"]
Sep 07 13:05:41.378627 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.378597    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj\" (UID: \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"
Sep 07 13:05:41.378777 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.378633    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7klt\" (UniqueName: \"kubernetes.io/projected/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-kube-api-access-k7klt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj\" (UID: \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"
Sep 07 13:05:41.378777 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.378668    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj\" (UID: \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"
Sep 07 13:05:41.479364 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.479329    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj\" (UID: \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"
Sep 07 13:05:41.479364 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.479365    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7klt\" (UniqueName: \"kubernetes.io/projected/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-kube-api-access-k7klt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj\" (UID: \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"
Sep 07 13:05:41.479545 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.479398    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj\" (UID: \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"
Sep 07 13:05:41.479734 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.479715    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj\" (UID: \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"
Sep 07 13:05:41.479792 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.479777    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj\" (UID: \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"
Sep 07 13:05:41.495812 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.495780    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7klt\" (UniqueName: \"kubernetes.io/projected/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-kube-api-access-k7klt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj\" (UID: \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"
Sep 07 13:05:41.598230 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.598146    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"
Sep 07 13:05:41.872121 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:41.872024    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"]
Sep 07 13:05:41.875673 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:05:41.875638    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1497681d_4a1f_4c9e_bfba_b7cb35ca3d9b.slice/crio-04f1cdcd90773ce41953ffe2bb3caa9c25ddde50d70b762dc7762bc556696fb8 WatchSource:0}: Error finding container 04f1cdcd90773ce41953ffe2bb3caa9c25ddde50d70b762dc7762bc556696fb8: Status 404 returned error can't find the container with id 04f1cdcd90773ce41953ffe2bb3caa9c25ddde50d70b762dc7762bc556696fb8
Sep 07 13:05:42.560420 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:42.560381    2567 generic.go:356] "Generic (PLEG): container finished" podID="1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b" containerID="be36822d3e3114c312fcb384dc440de885707571bd206ac917b875aeb4b6bc94" exitCode=0
Sep 07 13:05:42.560566 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:42.560469    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj" event={"ID":"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b","Type":"ContainerDied","Data":"be36822d3e3114c312fcb384dc440de885707571bd206ac917b875aeb4b6bc94"}
Sep 07 13:05:42.560566 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:42.560526    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj" event={"ID":"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b","Type":"ContainerStarted","Data":"04f1cdcd90773ce41953ffe2bb3caa9c25ddde50d70b762dc7762bc556696fb8"}
Sep 07 13:05:44.569298 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:44.569262    2567 generic.go:356] "Generic (PLEG): container finished" podID="1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b" containerID="74cbbe92624998c518e9d4f32bf9e161ce0da14678cdd83de8386ebd5dc3b4e3" exitCode=0
Sep 07 13:05:44.569666 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:44.569318    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj" event={"ID":"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b","Type":"ContainerDied","Data":"74cbbe92624998c518e9d4f32bf9e161ce0da14678cdd83de8386ebd5dc3b4e3"}
Sep 07 13:05:45.574568 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:45.574534    2567 generic.go:356] "Generic (PLEG): container finished" podID="1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b" containerID="4c2cf38c87ab906b4a7950ca71392e7a65555bdaedb3567c527a38c2c5b532d7" exitCode=0
Sep 07 13:05:45.574946 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:45.574614    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj" event={"ID":"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b","Type":"ContainerDied","Data":"4c2cf38c87ab906b4a7950ca71392e7a65555bdaedb3567c527a38c2c5b532d7"}
Sep 07 13:05:46.700428 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:46.700401    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"
Sep 07 13:05:46.821358 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:46.821322    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-bundle\") pod \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\" (UID: \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\") "
Sep 07 13:05:46.821548 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:46.821393    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7klt\" (UniqueName: \"kubernetes.io/projected/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-kube-api-access-k7klt\") pod \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\" (UID: \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\") "
Sep 07 13:05:46.821548 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:46.821422    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-util\") pod \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\" (UID: \"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b\") "
Sep 07 13:05:46.821751 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:46.821723    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-bundle" (OuterVolumeSpecName: "bundle") pod "1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b" (UID: "1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:05:46.823706 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:46.823670    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-kube-api-access-k7klt" (OuterVolumeSpecName: "kube-api-access-k7klt") pod "1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b" (UID: "1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b"). InnerVolumeSpecName "kube-api-access-k7klt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:05:46.826672 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:46.826648    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-util" (OuterVolumeSpecName: "util") pod "1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b" (UID: "1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:05:46.922659 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:46.922570    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7klt\" (UniqueName: \"kubernetes.io/projected/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-kube-api-access-k7klt\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:05:46.922659 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:46.922602    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-util\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:05:46.922659 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:46.922614    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b-bundle\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:05:47.584155 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:47.584120    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj" event={"ID":"1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b","Type":"ContainerDied","Data":"04f1cdcd90773ce41953ffe2bb3caa9c25ddde50d70b762dc7762bc556696fb8"}
Sep 07 13:05:47.584155 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:47.584154    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4vj"
Sep 07 13:05:47.584360 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:47.584171    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04f1cdcd90773ce41953ffe2bb3caa9c25ddde50d70b762dc7762bc556696fb8"
Sep 07 13:05:52.322297 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.322261    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb"]
Sep 07 13:05:52.322697 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.322513    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b" containerName="pull"
Sep 07 13:05:52.322697 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.322526    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b" containerName="pull"
Sep 07 13:05:52.322697 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.322539    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b" containerName="extract"
Sep 07 13:05:52.322697 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.322544    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b" containerName="extract"
Sep 07 13:05:52.322697 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.322556    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b" containerName="util"
Sep 07 13:05:52.322697 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.322561    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b" containerName="util"
Sep 07 13:05:52.322697 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.322599    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1497681d-4a1f-4c9e-bfba-b7cb35ca3d9b" containerName="extract"
Sep 07 13:05:52.324465 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.324447    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb"
Sep 07 13:05:52.329330 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.329304    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 07 13:05:52.329465 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.329350    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-698kk\""
Sep 07 13:05:52.329934 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.329915    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 07 13:05:52.337685 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.337658    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb"]
Sep 07 13:05:52.461987 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.461943    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f006b9b6-5821-48d1-82f4-aa06177b80eb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-f4wpb\" (UID: \"f006b9b6-5821-48d1-82f4-aa06177b80eb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb"
Sep 07 13:05:52.462193 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.462015    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss6xm\" (UniqueName: \"kubernetes.io/projected/f006b9b6-5821-48d1-82f4-aa06177b80eb-kube-api-access-ss6xm\") pod \"openshift-lws-operator-fd8ccff4c-f4wpb\" (UID: \"f006b9b6-5821-48d1-82f4-aa06177b80eb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb"
Sep 07 13:05:52.479859 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.479820    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gwkt4"]
Sep 07 13:05:52.481760 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.481738    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gwkt4"
Sep 07 13:05:52.483695 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.483674    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 07 13:05:52.483813 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.483728    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 07 13:05:52.483881 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.483825    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nk4l2\""
Sep 07 13:05:52.494394 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.494371    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gwkt4"]
Sep 07 13:05:52.563043 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.563004    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjv4\" (UniqueName: \"kubernetes.io/projected/2340ea8e-7fce-431d-bca4-4afbf6f67718-kube-api-access-ndjv4\") pod \"cert-manager-55b4d954d5-gwkt4\" (UID: \"2340ea8e-7fce-431d-bca4-4afbf6f67718\") " pod="cert-manager/cert-manager-55b4d954d5-gwkt4"
Sep 07 13:05:52.563226 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.563055    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss6xm\" (UniqueName: \"kubernetes.io/projected/f006b9b6-5821-48d1-82f4-aa06177b80eb-kube-api-access-ss6xm\") pod \"openshift-lws-operator-fd8ccff4c-f4wpb\" (UID: \"f006b9b6-5821-48d1-82f4-aa06177b80eb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb"
Sep 07 13:05:52.563226 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.563111    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2340ea8e-7fce-431d-bca4-4afbf6f67718-bound-sa-token\") pod \"cert-manager-55b4d954d5-gwkt4\" (UID: \"2340ea8e-7fce-431d-bca4-4afbf6f67718\") " pod="cert-manager/cert-manager-55b4d954d5-gwkt4"
Sep 07 13:05:52.563226 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.563163    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f006b9b6-5821-48d1-82f4-aa06177b80eb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-f4wpb\" (UID: \"f006b9b6-5821-48d1-82f4-aa06177b80eb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb"
Sep 07 13:05:52.563533 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.563510    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f006b9b6-5821-48d1-82f4-aa06177b80eb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-f4wpb\" (UID: \"f006b9b6-5821-48d1-82f4-aa06177b80eb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb"
Sep 07 13:05:52.573955 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.573890    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss6xm\" (UniqueName: \"kubernetes.io/projected/f006b9b6-5821-48d1-82f4-aa06177b80eb-kube-api-access-ss6xm\") pod \"openshift-lws-operator-fd8ccff4c-f4wpb\" (UID: \"f006b9b6-5821-48d1-82f4-aa06177b80eb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb"
Sep 07 13:05:52.633650 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.633610    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb"
Sep 07 13:05:52.664184 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.664138    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2340ea8e-7fce-431d-bca4-4afbf6f67718-bound-sa-token\") pod \"cert-manager-55b4d954d5-gwkt4\" (UID: \"2340ea8e-7fce-431d-bca4-4afbf6f67718\") " pod="cert-manager/cert-manager-55b4d954d5-gwkt4"
Sep 07 13:05:52.664315 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.664272    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjv4\" (UniqueName: \"kubernetes.io/projected/2340ea8e-7fce-431d-bca4-4afbf6f67718-kube-api-access-ndjv4\") pod \"cert-manager-55b4d954d5-gwkt4\" (UID: \"2340ea8e-7fce-431d-bca4-4afbf6f67718\") " pod="cert-manager/cert-manager-55b4d954d5-gwkt4"
Sep 07 13:05:52.673478 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.673449    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjv4\" (UniqueName: \"kubernetes.io/projected/2340ea8e-7fce-431d-bca4-4afbf6f67718-kube-api-access-ndjv4\") pod \"cert-manager-55b4d954d5-gwkt4\" (UID: \"2340ea8e-7fce-431d-bca4-4afbf6f67718\") " pod="cert-manager/cert-manager-55b4d954d5-gwkt4"
Sep 07 13:05:52.673652 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.673632    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2340ea8e-7fce-431d-bca4-4afbf6f67718-bound-sa-token\") pod \"cert-manager-55b4d954d5-gwkt4\" (UID: \"2340ea8e-7fce-431d-bca4-4afbf6f67718\") " pod="cert-manager/cert-manager-55b4d954d5-gwkt4"
Sep 07 13:05:52.790338 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.790301    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-gwkt4"
Sep 07 13:05:52.916728 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:52.916685    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb"]
Sep 07 13:05:52.921476 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:05:52.921446    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf006b9b6_5821_48d1_82f4_aa06177b80eb.slice/crio-5c454173a344e3bef00de89280d9ac74edbdc4d2f57a38c75dba6219f95cf7f6 WatchSource:0}: Error finding container 5c454173a344e3bef00de89280d9ac74edbdc4d2f57a38c75dba6219f95cf7f6: Status 404 returned error can't find the container with id 5c454173a344e3bef00de89280d9ac74edbdc4d2f57a38c75dba6219f95cf7f6
Sep 07 13:05:53.066094 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:53.066052    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-gwkt4"]
Sep 07 13:05:53.068240 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:05:53.068210    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2340ea8e_7fce_431d_bca4_4afbf6f67718.slice/crio-8eeeafc913910f4f682d1a9d77aeadeede3462560e73a20c8390783ac8f4a284 WatchSource:0}: Error finding container 8eeeafc913910f4f682d1a9d77aeadeede3462560e73a20c8390783ac8f4a284: Status 404 returned error can't find the container with id 8eeeafc913910f4f682d1a9d77aeadeede3462560e73a20c8390783ac8f4a284
Sep 07 13:05:53.609557 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:53.609515    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb" event={"ID":"f006b9b6-5821-48d1-82f4-aa06177b80eb","Type":"ContainerStarted","Data":"5c454173a344e3bef00de89280d9ac74edbdc4d2f57a38c75dba6219f95cf7f6"}
Sep 07 13:05:53.610647 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:53.610620    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gwkt4" event={"ID":"2340ea8e-7fce-431d-bca4-4afbf6f67718","Type":"ContainerStarted","Data":"8eeeafc913910f4f682d1a9d77aeadeede3462560e73a20c8390783ac8f4a284"}
Sep 07 13:05:56.625293 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:56.625247    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb" event={"ID":"f006b9b6-5821-48d1-82f4-aa06177b80eb","Type":"ContainerStarted","Data":"16158a81204eae0a48d91f5db4591d65241956e43e991b9a3bcd19a89acff724"}
Sep 07 13:05:56.626765 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:56.626740    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-gwkt4" event={"ID":"2340ea8e-7fce-431d-bca4-4afbf6f67718","Type":"ContainerStarted","Data":"6a77ad9b7d02027df3e2dd12a8b256081db1d8eb3813ad9691f8a14339cf172a"}
Sep 07 13:05:56.644811 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:56.644760    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4wpb" podStartSLOduration=1.726876874 podStartE2EDuration="4.644744961s" totalImagesPullingTime="2.917868087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:05:56.64358446 +0000 UTC m=+401.071882735" watchObservedRunningTime="2026-09-07 13:05:56.644744961 +0000 UTC m=+401.073043234"
Sep 07 13:05:56.661158 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:56.661102    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-gwkt4" podStartSLOduration=1.8609807649999999 podStartE2EDuration="4.661081087s" totalImagesPullingTime="2.800100322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:05:56.660180204 +0000 UTC m=+401.088478480" watchObservedRunningTime="2026-09-07 13:05:56.661081087 +0000 UTC m=+401.089379357"
Sep 07 13:05:58.865027 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:58.864991    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"]
Sep 07 13:05:58.867474 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:58.867457    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"
Sep 07 13:05:58.869368 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:58.869339    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 13:05:58.869494 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:58.869462    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 13:05:58.869758 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:58.869742    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4mdkr\""
Sep 07 13:05:58.880490 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:58.880467    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"]
Sep 07 13:05:59.016999 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.016957    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m\" (UID: \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"
Sep 07 13:05:59.017193 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.017025    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m\" (UID: \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"
Sep 07 13:05:59.017193 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.017082    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9ksh\" (UniqueName: \"kubernetes.io/projected/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-kube-api-access-z9ksh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m\" (UID: \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"
Sep 07 13:05:59.118115 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.118020    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m\" (UID: \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"
Sep 07 13:05:59.118115 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.118112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m\" (UID: \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"
Sep 07 13:05:59.118282 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.118145    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9ksh\" (UniqueName: \"kubernetes.io/projected/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-kube-api-access-z9ksh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m\" (UID: \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"
Sep 07 13:05:59.118490 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.118465    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m\" (UID: \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"
Sep 07 13:05:59.118528 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.118475    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m\" (UID: \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"
Sep 07 13:05:59.126648 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.126627    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9ksh\" (UniqueName: \"kubernetes.io/projected/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-kube-api-access-z9ksh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m\" (UID: \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"
Sep 07 13:05:59.176634 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.176600    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"
Sep 07 13:05:59.460851 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.460822    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"]
Sep 07 13:05:59.462769 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:05:59.462744    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fe7207_7ecc_4f37_9086_cc52a3e0aee1.slice/crio-c19beb2c5f8eb89beb3554c3619773aa53d81508cae26cd5616ddb564eb41b0e WatchSource:0}: Error finding container c19beb2c5f8eb89beb3554c3619773aa53d81508cae26cd5616ddb564eb41b0e: Status 404 returned error can't find the container with id c19beb2c5f8eb89beb3554c3619773aa53d81508cae26cd5616ddb564eb41b0e
Sep 07 13:05:59.640343 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.640308    2567 generic.go:356] "Generic (PLEG): container finished" podID="44fe7207-7ecc-4f37-9086-cc52a3e0aee1" containerID="ad8af6b06dc60b71c7e491c500f15b3c0f6372dfd93b8c429f2663be45de869f" exitCode=0
Sep 07 13:05:59.640564 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.640397    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m" event={"ID":"44fe7207-7ecc-4f37-9086-cc52a3e0aee1","Type":"ContainerDied","Data":"ad8af6b06dc60b71c7e491c500f15b3c0f6372dfd93b8c429f2663be45de869f"}
Sep 07 13:05:59.640564 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:05:59.640457    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m" event={"ID":"44fe7207-7ecc-4f37-9086-cc52a3e0aee1","Type":"ContainerStarted","Data":"c19beb2c5f8eb89beb3554c3619773aa53d81508cae26cd5616ddb564eb41b0e"}
Sep 07 13:06:01.650726 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:01.650686    2567 generic.go:356] "Generic (PLEG): container finished" podID="44fe7207-7ecc-4f37-9086-cc52a3e0aee1" containerID="aab1886b6aa8cb83da96c59329f756bd25eabbabeb22c7ba39c5945f70a1f736" exitCode=0
Sep 07 13:06:01.651134 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:01.650765    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m" event={"ID":"44fe7207-7ecc-4f37-9086-cc52a3e0aee1","Type":"ContainerDied","Data":"aab1886b6aa8cb83da96c59329f756bd25eabbabeb22c7ba39c5945f70a1f736"}
Sep 07 13:06:02.657013 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:02.656973    2567 generic.go:356] "Generic (PLEG): container finished" podID="44fe7207-7ecc-4f37-9086-cc52a3e0aee1" containerID="9696a51d58d2fa8c27e0bf9dcab6ab116c2510e93faec7a4942d165f376f5414" exitCode=0
Sep 07 13:06:02.657402 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:02.657074    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m" event={"ID":"44fe7207-7ecc-4f37-9086-cc52a3e0aee1","Type":"ContainerDied","Data":"9696a51d58d2fa8c27e0bf9dcab6ab116c2510e93faec7a4942d165f376f5414"}
Sep 07 13:06:03.786464 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:03.786436    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"
Sep 07 13:06:03.958707 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:03.958608    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9ksh\" (UniqueName: \"kubernetes.io/projected/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-kube-api-access-z9ksh\") pod \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\" (UID: \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\") "
Sep 07 13:06:03.958707 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:03.958673    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-bundle\") pod \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\" (UID: \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\") "
Sep 07 13:06:03.958707 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:03.958696    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-util\") pod \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\" (UID: \"44fe7207-7ecc-4f37-9086-cc52a3e0aee1\") "
Sep 07 13:06:03.959421 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:03.959386    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-bundle" (OuterVolumeSpecName: "bundle") pod "44fe7207-7ecc-4f37-9086-cc52a3e0aee1" (UID: "44fe7207-7ecc-4f37-9086-cc52a3e0aee1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:06:03.960938 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:03.960910    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-kube-api-access-z9ksh" (OuterVolumeSpecName: "kube-api-access-z9ksh") pod "44fe7207-7ecc-4f37-9086-cc52a3e0aee1" (UID: "44fe7207-7ecc-4f37-9086-cc52a3e0aee1"). InnerVolumeSpecName "kube-api-access-z9ksh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:06:03.964605 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:03.964577    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-util" (OuterVolumeSpecName: "util") pod "44fe7207-7ecc-4f37-9086-cc52a3e0aee1" (UID: "44fe7207-7ecc-4f37-9086-cc52a3e0aee1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:06:04.059225 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:04.059185    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-bundle\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:06:04.059225 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:04.059216    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-util\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:06:04.059225 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:04.059227    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9ksh\" (UniqueName: \"kubernetes.io/projected/44fe7207-7ecc-4f37-9086-cc52a3e0aee1-kube-api-access-z9ksh\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:06:04.667039 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:04.666998    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m" event={"ID":"44fe7207-7ecc-4f37-9086-cc52a3e0aee1","Type":"ContainerDied","Data":"c19beb2c5f8eb89beb3554c3619773aa53d81508cae26cd5616ddb564eb41b0e"}
Sep 07 13:06:04.667039 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:04.667040    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftk29m"
Sep 07 13:06:04.667267 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:04.667053    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c19beb2c5f8eb89beb3554c3619773aa53d81508cae26cd5616ddb564eb41b0e"
Sep 07 13:06:10.451680 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.451641    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"]
Sep 07 13:06:10.452041 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.451890    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44fe7207-7ecc-4f37-9086-cc52a3e0aee1" containerName="extract"
Sep 07 13:06:10.452041 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.451901    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="44fe7207-7ecc-4f37-9086-cc52a3e0aee1" containerName="extract"
Sep 07 13:06:10.452041 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.451915    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44fe7207-7ecc-4f37-9086-cc52a3e0aee1" containerName="pull"
Sep 07 13:06:10.452041 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.451922    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="44fe7207-7ecc-4f37-9086-cc52a3e0aee1" containerName="pull"
Sep 07 13:06:10.452041 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.451930    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44fe7207-7ecc-4f37-9086-cc52a3e0aee1" containerName="util"
Sep 07 13:06:10.452041 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.451936    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="44fe7207-7ecc-4f37-9086-cc52a3e0aee1" containerName="util"
Sep 07 13:06:10.452041 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.451979    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="44fe7207-7ecc-4f37-9086-cc52a3e0aee1" containerName="extract"
Sep 07 13:06:10.453886 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.453868    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"
Sep 07 13:06:10.456074 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.456025    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4mdkr\""
Sep 07 13:06:10.456218 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.456173    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 13:06:10.456294 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.456266    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 13:06:10.464530 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.464505    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"]
Sep 07 13:06:10.606727 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.606689    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxhlm\" (UniqueName: \"kubernetes.io/projected/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-kube-api-access-sxhlm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv\" (UID: \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"
Sep 07 13:06:10.606901 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.606758    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv\" (UID: \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"
Sep 07 13:06:10.606901 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.606778    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv\" (UID: \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"
Sep 07 13:06:10.707859 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.707770    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv\" (UID: \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"
Sep 07 13:06:10.707859 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.707806    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv\" (UID: \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"
Sep 07 13:06:10.707859 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.707837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxhlm\" (UniqueName: \"kubernetes.io/projected/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-kube-api-access-sxhlm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv\" (UID: \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"
Sep 07 13:06:10.708200 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.708181    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv\" (UID: \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"
Sep 07 13:06:10.708263 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.708237    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv\" (UID: \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"
Sep 07 13:06:10.720165 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.720140    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxhlm\" (UniqueName: \"kubernetes.io/projected/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-kube-api-access-sxhlm\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv\" (UID: \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"
Sep 07 13:06:10.762832 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:10.762798    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"
Sep 07 13:06:11.035480 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:11.035451    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"]
Sep 07 13:06:11.037762 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:06:11.037728    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8078a8_67a1_4ddd_9a75_1fe9dd53b455.slice/crio-77895ef1426ee4f8e98517c00a645116f28876ab33861d1bbbea6cbb5608010c WatchSource:0}: Error finding container 77895ef1426ee4f8e98517c00a645116f28876ab33861d1bbbea6cbb5608010c: Status 404 returned error can't find the container with id 77895ef1426ee4f8e98517c00a645116f28876ab33861d1bbbea6cbb5608010c
Sep 07 13:06:11.696957 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:11.696925    2567 generic.go:356] "Generic (PLEG): container finished" podID="1e8078a8-67a1-4ddd-9a75-1fe9dd53b455" containerID="b02c43ec3f9e2d7ce720b3b381f135a87fd4c56513a78a8fa46200d5fab4c738" exitCode=0
Sep 07 13:06:11.697363 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:11.697020    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv" event={"ID":"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455","Type":"ContainerDied","Data":"b02c43ec3f9e2d7ce720b3b381f135a87fd4c56513a78a8fa46200d5fab4c738"}
Sep 07 13:06:11.697363 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:11.697097    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv" event={"ID":"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455","Type":"ContainerStarted","Data":"77895ef1426ee4f8e98517c00a645116f28876ab33861d1bbbea6cbb5608010c"}
Sep 07 13:06:12.703292 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:12.703265    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv" event={"ID":"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455","Type":"ContainerStarted","Data":"b3f3948a5d5314926bd431809ee5c6c39969604eace17e6bede7e6956daee47c"}
Sep 07 13:06:13.087256 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.087222    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"]
Sep 07 13:06:13.089497 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.089481    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.092428 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.092401    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 07 13:06:13.092554 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.092498    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vvqfs\""
Sep 07 13:06:13.095082 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.095043    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 07 13:06:13.095217 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.095129    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 07 13:06:13.095217 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.095135    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 07 13:06:13.121248 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.121223    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"]
Sep 07 13:06:13.123973 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.123949    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-manifests\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.124105 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.123989    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-webhook-cert\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.124105 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.124082    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.124236 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.124113    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-tmp\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.124236 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.124157    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lc7t\" (UniqueName: \"kubernetes.io/projected/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-kube-api-access-9lc7t\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.224982 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.224945    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-webhook-cert\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.225166 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.225003    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.225166 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.225028    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-tmp\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.225243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.225205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lc7t\" (UniqueName: \"kubernetes.io/projected/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-kube-api-access-9lc7t\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.225303 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.225286    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-manifests\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.225622 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.225592    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-manifests\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.227483 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.227460    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-tmp\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.227627 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.227607    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.227797 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.227780    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-webhook-cert\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.239467 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.239442    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lc7t\" (UniqueName: \"kubernetes.io/projected/fc7d25ca-6842-4db6-a4c4-ebc46f8d3023-kube-api-access-9lc7t\") pod \"opendatahub-operator-controller-manager-5545ff49d6-pkmvq\" (UID: \"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.398821 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.398736    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:13.690202 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.690107    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"]
Sep 07 13:06:13.696663 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:06:13.696630    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7d25ca_6842_4db6_a4c4_ebc46f8d3023.slice/crio-5a8898fc8dcaebbf0f4350acb2caeb4a4da4140e2b633b1afd333908cb66d5a6 WatchSource:0}: Error finding container 5a8898fc8dcaebbf0f4350acb2caeb4a4da4140e2b633b1afd333908cb66d5a6: Status 404 returned error can't find the container with id 5a8898fc8dcaebbf0f4350acb2caeb4a4da4140e2b633b1afd333908cb66d5a6
Sep 07 13:06:13.707988 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.707962    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq" event={"ID":"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023","Type":"ContainerStarted","Data":"5a8898fc8dcaebbf0f4350acb2caeb4a4da4140e2b633b1afd333908cb66d5a6"}
Sep 07 13:06:13.709608 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.709580    2567 generic.go:356] "Generic (PLEG): container finished" podID="1e8078a8-67a1-4ddd-9a75-1fe9dd53b455" containerID="b3f3948a5d5314926bd431809ee5c6c39969604eace17e6bede7e6956daee47c" exitCode=0
Sep 07 13:06:13.709703 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:13.709610    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv" event={"ID":"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455","Type":"ContainerDied","Data":"b3f3948a5d5314926bd431809ee5c6c39969604eace17e6bede7e6956daee47c"}
Sep 07 13:06:14.718019 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:14.717936    2567 generic.go:356] "Generic (PLEG): container finished" podID="1e8078a8-67a1-4ddd-9a75-1fe9dd53b455" containerID="a9cca98b64a82bcab80f601444d8b2b5668f0ee50a36dfb9cd2366369fa1f055" exitCode=0
Sep 07 13:06:14.718508 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:14.718022    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv" event={"ID":"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455","Type":"ContainerDied","Data":"a9cca98b64a82bcab80f601444d8b2b5668f0ee50a36dfb9cd2366369fa1f055"}
Sep 07 13:06:16.744941 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:16.744912    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"
Sep 07 13:06:16.753199 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:16.753178    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-util\") pod \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\" (UID: \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\") "
Sep 07 13:06:16.753292 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:16.753217    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-bundle\") pod \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\" (UID: \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\") "
Sep 07 13:06:16.753292 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:16.753243    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxhlm\" (UniqueName: \"kubernetes.io/projected/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-kube-api-access-sxhlm\") pod \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\" (UID: \"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455\") "
Sep 07 13:06:16.753843 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:16.753816    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-bundle" (OuterVolumeSpecName: "bundle") pod "1e8078a8-67a1-4ddd-9a75-1fe9dd53b455" (UID: "1e8078a8-67a1-4ddd-9a75-1fe9dd53b455"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:06:16.755532 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:16.755509    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-kube-api-access-sxhlm" (OuterVolumeSpecName: "kube-api-access-sxhlm") pod "1e8078a8-67a1-4ddd-9a75-1fe9dd53b455" (UID: "1e8078a8-67a1-4ddd-9a75-1fe9dd53b455"). InnerVolumeSpecName "kube-api-access-sxhlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:06:16.757743 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:16.757705    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-util" (OuterVolumeSpecName: "util") pod "1e8078a8-67a1-4ddd-9a75-1fe9dd53b455" (UID: "1e8078a8-67a1-4ddd-9a75-1fe9dd53b455"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:06:16.853993 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:16.853963    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-bundle\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:06:16.853993 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:16.853992    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxhlm\" (UniqueName: \"kubernetes.io/projected/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-kube-api-access-sxhlm\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:06:16.854151 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:16.854003    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e8078a8-67a1-4ddd-9a75-1fe9dd53b455-util\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:06:17.734586 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:17.734543    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq" event={"ID":"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023","Type":"ContainerStarted","Data":"a93b4f444d4b45452af427381085cca70ba1b7728880bc59a565d610fc9bc27f"}
Sep 07 13:06:17.736420 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:17.736392    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv" event={"ID":"1e8078a8-67a1-4ddd-9a75-1fe9dd53b455","Type":"ContainerDied","Data":"77895ef1426ee4f8e98517c00a645116f28876ab33861d1bbbea6cbb5608010c"}
Sep 07 13:06:17.736580 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:17.736425    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xccnv"
Sep 07 13:06:17.736580 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:17.736432    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77895ef1426ee4f8e98517c00a645116f28876ab33861d1bbbea6cbb5608010c"
Sep 07 13:06:18.742126 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:18.742091    2567 generic.go:356] "Generic (PLEG): container finished" podID="fc7d25ca-6842-4db6-a4c4-ebc46f8d3023" containerID="a93b4f444d4b45452af427381085cca70ba1b7728880bc59a565d610fc9bc27f" exitCode=0
Sep 07 13:06:18.742502 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:18.742144    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq" event={"ID":"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023","Type":"ContainerDied","Data":"a93b4f444d4b45452af427381085cca70ba1b7728880bc59a565d610fc9bc27f"}
Sep 07 13:06:19.748091 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:19.747968    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq" event={"ID":"fc7d25ca-6842-4db6-a4c4-ebc46f8d3023","Type":"ContainerStarted","Data":"93e6b0ca37a86b35c2f35f38820f899584fc4da273905beb977c5648b49ee52e"}
Sep 07 13:06:19.748547 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:19.748188    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:19.789719 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:19.789664    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq" podStartSLOduration=2.124922581 podStartE2EDuration="6.789646974s" totalImagesPullingTime="3.621975441s" totalInitContainerRuntime="1.042748952s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:19.786655378 +0000 UTC m=+424.214953700" watchObservedRunningTime="2026-09-07 13:06:19.789646974 +0000 UTC m=+424.217945247"
Sep 07 13:06:30.755525 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:30.755486    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-pkmvq"
Sep 07 13:06:33.529048 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.529006    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"]
Sep 07 13:06:33.529452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.529278    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8078a8-67a1-4ddd-9a75-1fe9dd53b455" containerName="extract"
Sep 07 13:06:33.529452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.529290    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8078a8-67a1-4ddd-9a75-1fe9dd53b455" containerName="extract"
Sep 07 13:06:33.529452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.529308    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8078a8-67a1-4ddd-9a75-1fe9dd53b455" containerName="util"
Sep 07 13:06:33.529452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.529313    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8078a8-67a1-4ddd-9a75-1fe9dd53b455" containerName="util"
Sep 07 13:06:33.529452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.529319    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8078a8-67a1-4ddd-9a75-1fe9dd53b455" containerName="pull"
Sep 07 13:06:33.529452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.529325    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8078a8-67a1-4ddd-9a75-1fe9dd53b455" containerName="pull"
Sep 07 13:06:33.529452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.529366    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e8078a8-67a1-4ddd-9a75-1fe9dd53b455" containerName="extract"
Sep 07 13:06:33.535759 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.535738    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"
Sep 07 13:06:33.542223 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.542197    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 07 13:06:33.542950 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.542931    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 07 13:06:33.543107 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.543085    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 07 13:06:33.543659 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.543644    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 07 13:06:33.570962 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.570936    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"]
Sep 07 13:06:33.581672 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.581641    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea763de-018b-432b-8322-0345cec5ec7e-tls-certs\") pod \"kube-auth-proxy-546b89bbb7-99rvx\" (UID: \"2ea763de-018b-432b-8322-0345cec5ec7e\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"
Sep 07 13:06:33.581811 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.581675    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ea763de-018b-432b-8322-0345cec5ec7e-tmp\") pod \"kube-auth-proxy-546b89bbb7-99rvx\" (UID: \"2ea763de-018b-432b-8322-0345cec5ec7e\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"
Sep 07 13:06:33.581811 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.581698    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkq4n\" (UniqueName: \"kubernetes.io/projected/2ea763de-018b-432b-8322-0345cec5ec7e-kube-api-access-vkq4n\") pod \"kube-auth-proxy-546b89bbb7-99rvx\" (UID: \"2ea763de-018b-432b-8322-0345cec5ec7e\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"
Sep 07 13:06:33.682667 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.682625    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea763de-018b-432b-8322-0345cec5ec7e-tls-certs\") pod \"kube-auth-proxy-546b89bbb7-99rvx\" (UID: \"2ea763de-018b-432b-8322-0345cec5ec7e\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"
Sep 07 13:06:33.682667 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.682670    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ea763de-018b-432b-8322-0345cec5ec7e-tmp\") pod \"kube-auth-proxy-546b89bbb7-99rvx\" (UID: \"2ea763de-018b-432b-8322-0345cec5ec7e\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"
Sep 07 13:06:33.682882 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.682693    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkq4n\" (UniqueName: \"kubernetes.io/projected/2ea763de-018b-432b-8322-0345cec5ec7e-kube-api-access-vkq4n\") pod \"kube-auth-proxy-546b89bbb7-99rvx\" (UID: \"2ea763de-018b-432b-8322-0345cec5ec7e\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"
Sep 07 13:06:33.684898 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.684861    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ea763de-018b-432b-8322-0345cec5ec7e-tmp\") pod \"kube-auth-proxy-546b89bbb7-99rvx\" (UID: \"2ea763de-018b-432b-8322-0345cec5ec7e\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"
Sep 07 13:06:33.685176 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.685154    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea763de-018b-432b-8322-0345cec5ec7e-tls-certs\") pod \"kube-auth-proxy-546b89bbb7-99rvx\" (UID: \"2ea763de-018b-432b-8322-0345cec5ec7e\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"
Sep 07 13:06:33.724146 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.724118    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkq4n\" (UniqueName: \"kubernetes.io/projected/2ea763de-018b-432b-8322-0345cec5ec7e-kube-api-access-vkq4n\") pod \"kube-auth-proxy-546b89bbb7-99rvx\" (UID: \"2ea763de-018b-432b-8322-0345cec5ec7e\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"
Sep 07 13:06:33.845605 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:33.845514    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"
Sep 07 13:06:34.140664 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:34.140637    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx"]
Sep 07 13:06:34.142246 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:06:34.142217    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea763de_018b_432b_8322_0345cec5ec7e.slice/crio-91835eff467a1cd2a59ea57b7290273ad7c3e9cbe649ab61b1a58d84a9e9ff73 WatchSource:0}: Error finding container 91835eff467a1cd2a59ea57b7290273ad7c3e9cbe649ab61b1a58d84a9e9ff73: Status 404 returned error can't find the container with id 91835eff467a1cd2a59ea57b7290273ad7c3e9cbe649ab61b1a58d84a9e9ff73
Sep 07 13:06:34.814726 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:34.814678    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx" event={"ID":"2ea763de-018b-432b-8322-0345cec5ec7e","Type":"ContainerStarted","Data":"91835eff467a1cd2a59ea57b7290273ad7c3e9cbe649ab61b1a58d84a9e9ff73"}
Sep 07 13:06:35.678363 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.678327    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"]
Sep 07 13:06:35.688668 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.688625    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.692083 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.692014    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 07 13:06:35.693101 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.692508    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2ppnm\""
Sep 07 13:06:35.693101 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.692716    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 07 13:06:35.693101 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.692932    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 13:06:35.693101 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.693020    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 07 13:06:35.703404 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.703374    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"]
Sep 07 13:06:35.803562 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.803524    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct9n4\" (UniqueName: \"kubernetes.io/projected/c8789395-0cff-4904-a9fb-75c07c7079b9-kube-api-access-ct9n4\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.803825 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.803597    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.803825 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.803627    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.803825 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.803690    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.803825 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.803719    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.803825 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.803751    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8789395-0cff-4904-a9fb-75c07c7079b9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.803825 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.803779    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.904685 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.904649    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.904685 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.904686    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.905243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.904715    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.905243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.904739    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.905243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.904911    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8789395-0cff-4904-a9fb-75c07c7079b9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.905243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.904955    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.905243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.905026    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct9n4\" (UniqueName: \"kubernetes.io/projected/c8789395-0cff-4904-a9fb-75c07c7079b9-kube-api-access-ct9n4\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.905499 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.905322    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.907512 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.907481    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.907851 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.907821    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.907851 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.907844    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.908021 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.907841    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8789395-0cff-4904-a9fb-75c07c7079b9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.918599 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.918561    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:35.919285 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:35.919254    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct9n4\" (UniqueName: \"kubernetes.io/projected/c8789395-0cff-4904-a9fb-75c07c7079b9-kube-api-access-ct9n4\") pod \"istiod-openshift-gateway-7b7995d5cd-prmd7\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:36.003666 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:36.003569    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:37.077443 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:37.077416    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"]
Sep 07 13:06:37.079360 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:06:37.079337    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8789395_0cff_4904_a9fb_75c07c7079b9.slice/crio-6c63134ac964516e4b57273a2ddb749963c49aed9cf612d6a4898ae06820363a WatchSource:0}: Error finding container 6c63134ac964516e4b57273a2ddb749963c49aed9cf612d6a4898ae06820363a: Status 404 returned error can't find the container with id 6c63134ac964516e4b57273a2ddb749963c49aed9cf612d6a4898ae06820363a
Sep 07 13:06:37.830644 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:37.830603    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx" event={"ID":"2ea763de-018b-432b-8322-0345cec5ec7e","Type":"ContainerStarted","Data":"9468647f0f646a0b3d5b8390de4f8f59d96b9dc5b86a10a393ab6ba73c1c25d2"}
Sep 07 13:06:37.831893 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:37.831863    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7" event={"ID":"c8789395-0cff-4904-a9fb-75c07c7079b9","Type":"ContainerStarted","Data":"6c63134ac964516e4b57273a2ddb749963c49aed9cf612d6a4898ae06820363a"}
Sep 07 13:06:37.850555 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:37.850506    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-546b89bbb7-99rvx" podStartSLOduration=2.129310283 podStartE2EDuration="4.850488496s" totalImagesPullingTime="2.721178213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:37.848807056 +0000 UTC m=+442.277105364" watchObservedRunningTime="2026-09-07 13:06:37.850488496 +0000 UTC m=+442.278786768"
Sep 07 13:06:39.962990 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:39.962952    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:06:39.963301 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:39.963017    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:06:40.850256 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:40.850218    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7" event={"ID":"c8789395-0cff-4904-a9fb-75c07c7079b9","Type":"ContainerStarted","Data":"f16424ce17c0e9d8e84e65e9f9cc132e24f9153c711acec6d813d962aa414194"}
Sep 07 13:06:40.850450 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:40.850330    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:40.875434 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:40.875383    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7" podStartSLOduration=2.9941239680000002 podStartE2EDuration="5.87536681s" totalImagesPullingTime="2.881242842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:40.874587971 +0000 UTC m=+445.302886277" watchObservedRunningTime="2026-09-07 13:06:40.87536681 +0000 UTC m=+445.303665085"
Sep 07 13:06:41.029334 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.029291    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"]
Sep 07 13:06:41.032851 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.032826    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.035006 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.034981    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-vc7fz\""
Sep 07 13:06:41.046813 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.046781    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"]
Sep 07 13:06:41.147648 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.147564    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.147648 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.147615    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.147882 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.147689    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.147882 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.147748    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17e721b-4419-4dd2-a78e-339f4b969e8d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.147882 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.147787    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17e721b-4419-4dd2-a78e-339f4b969e8d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.147882 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.147816    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.148071 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.147898    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m755n\" (UniqueName: \"kubernetes.io/projected/d17e721b-4419-4dd2-a78e-339f4b969e8d-kube-api-access-m755n\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.148071 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.147935    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17e721b-4419-4dd2-a78e-339f4b969e8d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.148071 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.147991    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.248580 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.248539    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17e721b-4419-4dd2-a78e-339f4b969e8d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.248580 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.248594    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.248849 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.248644    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.248849 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.248679    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.248849 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.248705    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.248849 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.248746    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17e721b-4419-4dd2-a78e-339f4b969e8d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.248849 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.248780    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17e721b-4419-4dd2-a78e-339f4b969e8d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.248849 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.248808    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.249181 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.248869    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m755n\" (UniqueName: \"kubernetes.io/projected/d17e721b-4419-4dd2-a78e-339f4b969e8d-kube-api-access-m755n\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.249181 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.249147    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.249346 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.249328    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.249464 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.249440    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17e721b-4419-4dd2-a78e-339f4b969e8d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.249523 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.249491    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.249666 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.249644    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.251128 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.251107    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17e721b-4419-4dd2-a78e-339f4b969e8d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.251392 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.251371    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17e721b-4419-4dd2-a78e-339f4b969e8d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.257523 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.257505    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17e721b-4419-4dd2-a78e-339f4b969e8d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.257678 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.257661    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m755n\" (UniqueName: \"kubernetes.io/projected/d17e721b-4419-4dd2-a78e-339f4b969e8d-kube-api-access-m755n\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5\" (UID: \"d17e721b-4419-4dd2-a78e-339f4b969e8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.345701 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.345658    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:41.624715 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.624678    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"]
Sep 07 13:06:41.627749 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:06:41.627715    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17e721b_4419_4dd2_a78e_339f4b969e8d.slice/crio-a68dbd49a9ef76f8cee8a2cf38e33aacd2d6db00434e9fcb67f8883df583a3fe WatchSource:0}: Error finding container a68dbd49a9ef76f8cee8a2cf38e33aacd2d6db00434e9fcb67f8883df583a3fe: Status 404 returned error can't find the container with id a68dbd49a9ef76f8cee8a2cf38e33aacd2d6db00434e9fcb67f8883df583a3fe
Sep 07 13:06:41.855714 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.855676    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5" event={"ID":"d17e721b-4419-4dd2-a78e-339f4b969e8d","Type":"ContainerStarted","Data":"a68dbd49a9ef76f8cee8a2cf38e33aacd2d6db00434e9fcb67f8883df583a3fe"}
Sep 07 13:06:41.857594 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:41.857564    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:44.229425 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:44.229387    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:06:44.229682 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:44.229466    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:06:44.229682 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:44.229495    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:06:44.871574 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:44.871534    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5" event={"ID":"d17e721b-4419-4dd2-a78e-339f4b969e8d","Type":"ContainerStarted","Data":"8e87660e90eefaa9289f16f9fb02c3f5cbf39d8d07f73363a05c943e36f10727"}
Sep 07 13:06:44.895195 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:44.895137    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5" podStartSLOduration=1.2959022550000001 podStartE2EDuration="3.895120327s" totalImagesPullingTime="2.599218072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:44.893541561 +0000 UTC m=+449.321839835" watchObservedRunningTime="2026-09-07 13:06:44.895120327 +0000 UTC m=+449.323418599"
Sep 07 13:06:45.346128 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:45.346087    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:45.351443 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:45.351413    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:45.876592 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:45.876563    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:45.877471 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:45.877448    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5"
Sep 07 13:06:50.965822 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:50.965783    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"]
Sep 07 13:06:50.973601 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:50.973578    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:50.984341 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:50.984312    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"]
Sep 07 13:06:51.037813 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.037769    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19341b0-ac2f-46bb-a710-7e86f8bad345-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.037813 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.037812    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdt4g\" (UniqueName: \"kubernetes.io/projected/e19341b0-ac2f-46bb-a710-7e86f8bad345-kube-api-access-kdt4g\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.038163 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.037875    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e19341b0-ac2f-46bb-a710-7e86f8bad345-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.038163 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.037913    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e19341b0-ac2f-46bb-a710-7e86f8bad345-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.038163 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.037970    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e19341b0-ac2f-46bb-a710-7e86f8bad345-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.038163 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.037996    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e19341b0-ac2f-46bb-a710-7e86f8bad345-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.038163 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.038111    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19341b0-ac2f-46bb-a710-7e86f8bad345-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.138835 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.138798    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e19341b0-ac2f-46bb-a710-7e86f8bad345-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.138835 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.138839    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e19341b0-ac2f-46bb-a710-7e86f8bad345-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.139141 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.138873    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19341b0-ac2f-46bb-a710-7e86f8bad345-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.139141 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.138894    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19341b0-ac2f-46bb-a710-7e86f8bad345-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.139141 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.138909    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdt4g\" (UniqueName: \"kubernetes.io/projected/e19341b0-ac2f-46bb-a710-7e86f8bad345-kube-api-access-kdt4g\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.139141 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.138934    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e19341b0-ac2f-46bb-a710-7e86f8bad345-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.139141 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.138958    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e19341b0-ac2f-46bb-a710-7e86f8bad345-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.139757 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.139728    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e19341b0-ac2f-46bb-a710-7e86f8bad345-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.141393 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.141360    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19341b0-ac2f-46bb-a710-7e86f8bad345-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.141506 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.141411    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e19341b0-ac2f-46bb-a710-7e86f8bad345-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.141506 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.141460    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e19341b0-ac2f-46bb-a710-7e86f8bad345-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.141611 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.141557    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e19341b0-ac2f-46bb-a710-7e86f8bad345-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.153596 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.153574    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19341b0-ac2f-46bb-a710-7e86f8bad345-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.159403 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.159374    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdt4g\" (UniqueName: \"kubernetes.io/projected/e19341b0-ac2f-46bb-a710-7e86f8bad345-kube-api-access-kdt4g\") pod \"istiod-openshift-gateway-5677ccb49b-dhxxb\" (UID: \"e19341b0-ac2f-46bb-a710-7e86f8bad345\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.283018 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.282984    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.568932 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.568893    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"]
Sep 07 13:06:51.574189 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.574153    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:06:51.574301 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.574219    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:06:51.906395 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.906292    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb" event={"ID":"e19341b0-ac2f-46bb-a710-7e86f8bad345","Type":"ContainerStarted","Data":"0ac000bacd81cb08a0b1d3f67af7065c24afb6618412c0688c78a53c884652a5"}
Sep 07 13:06:51.906395 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.906356    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb" event={"ID":"e19341b0-ac2f-46bb-a710-7e86f8bad345","Type":"ContainerStarted","Data":"7d25868b6872553e89386a27f26c1f446b8a34f2fd0b933da601de570e6c714f"}
Sep 07 13:06:51.906585 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.906408    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:51.953295 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:51.953223    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb" podStartSLOduration=1.9531975400000001 podStartE2EDuration="1.95319754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:51.949123957 +0000 UTC m=+456.377422257" watchObservedRunningTime="2026-09-07 13:06:51.95319754 +0000 UTC m=+456.381495813"
Sep 07 13:06:52.913484 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:52.913456    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dhxxb"
Sep 07 13:06:53.009655 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.009622    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"]
Sep 07 13:06:53.009827 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.009809    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7" podUID="c8789395-0cff-4904-a9fb-75c07c7079b9" containerName="discovery" containerID="cri-o://f16424ce17c0e9d8e84e65e9f9cc132e24f9153c711acec6d813d962aa414194" gracePeriod=30
Sep 07 13:06:53.254524 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.254493    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:53.359532 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.359492    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-csr-dns-cert\") pod \"c8789395-0cff-4904-a9fb-75c07c7079b9\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") "
Sep 07 13:06:53.359532 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.359540    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-token\") pod \"c8789395-0cff-4904-a9fb-75c07c7079b9\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") "
Sep 07 13:06:53.359762 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.359575    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8789395-0cff-4904-a9fb-75c07c7079b9-local-certs\") pod \"c8789395-0cff-4904-a9fb-75c07c7079b9\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") "
Sep 07 13:06:53.359820 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.359749    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-csr-ca-configmap\") pod \"c8789395-0cff-4904-a9fb-75c07c7079b9\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") "
Sep 07 13:06:53.359820 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.359792    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-cacerts\") pod \"c8789395-0cff-4904-a9fb-75c07c7079b9\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") "
Sep 07 13:06:53.359921 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.359847    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct9n4\" (UniqueName: \"kubernetes.io/projected/c8789395-0cff-4904-a9fb-75c07c7079b9-kube-api-access-ct9n4\") pod \"c8789395-0cff-4904-a9fb-75c07c7079b9\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") "
Sep 07 13:06:53.359967 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.359938    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-kubeconfig\") pod \"c8789395-0cff-4904-a9fb-75c07c7079b9\" (UID: \"c8789395-0cff-4904-a9fb-75c07c7079b9\") "
Sep 07 13:06:53.360387 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.360117    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c8789395-0cff-4904-a9fb-75c07c7079b9" (UID: "c8789395-0cff-4904-a9fb-75c07c7079b9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:06:53.360387 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.360265    2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-csr-ca-configmap\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:06:53.362170 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.362131    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-cacerts" (OuterVolumeSpecName: "cacerts") pod "c8789395-0cff-4904-a9fb-75c07c7079b9" (UID: "c8789395-0cff-4904-a9fb-75c07c7079b9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:06:53.362400 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.362367    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8789395-0cff-4904-a9fb-75c07c7079b9-local-certs" (OuterVolumeSpecName: "local-certs") pod "c8789395-0cff-4904-a9fb-75c07c7079b9" (UID: "c8789395-0cff-4904-a9fb-75c07c7079b9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:06:53.362474 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.362407    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c8789395-0cff-4904-a9fb-75c07c7079b9" (UID: "c8789395-0cff-4904-a9fb-75c07c7079b9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:06:53.362534 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.362518    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-token" (OuterVolumeSpecName: "istio-token") pod "c8789395-0cff-4904-a9fb-75c07c7079b9" (UID: "c8789395-0cff-4904-a9fb-75c07c7079b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:06:53.362582 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.362513    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8789395-0cff-4904-a9fb-75c07c7079b9-kube-api-access-ct9n4" (OuterVolumeSpecName: "kube-api-access-ct9n4") pod "c8789395-0cff-4904-a9fb-75c07c7079b9" (UID: "c8789395-0cff-4904-a9fb-75c07c7079b9"). InnerVolumeSpecName "kube-api-access-ct9n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:06:53.362660 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.362603    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c8789395-0cff-4904-a9fb-75c07c7079b9" (UID: "c8789395-0cff-4904-a9fb-75c07c7079b9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:06:53.461243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.461157    2567 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-kubeconfig\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:06:53.461243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.461193    2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-csr-dns-cert\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:06:53.461243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.461205    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8789395-0cff-4904-a9fb-75c07c7079b9-istio-token\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:06:53.461243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.461213    2567 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8789395-0cff-4904-a9fb-75c07c7079b9-local-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:06:53.461243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.461221    2567 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8789395-0cff-4904-a9fb-75c07c7079b9-cacerts\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:06:53.461243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.461229    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct9n4\" (UniqueName: \"kubernetes.io/projected/c8789395-0cff-4904-a9fb-75c07c7079b9-kube-api-access-ct9n4\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:06:53.917297 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.917257    2567 generic.go:356] "Generic (PLEG): container finished" podID="c8789395-0cff-4904-a9fb-75c07c7079b9" containerID="f16424ce17c0e9d8e84e65e9f9cc132e24f9153c711acec6d813d962aa414194" exitCode=0
Sep 07 13:06:53.917733 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.917307    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"
Sep 07 13:06:53.917733 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.917342    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7" event={"ID":"c8789395-0cff-4904-a9fb-75c07c7079b9","Type":"ContainerDied","Data":"f16424ce17c0e9d8e84e65e9f9cc132e24f9153c711acec6d813d962aa414194"}
Sep 07 13:06:53.917733 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.917396    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7" event={"ID":"c8789395-0cff-4904-a9fb-75c07c7079b9","Type":"ContainerDied","Data":"6c63134ac964516e4b57273a2ddb749963c49aed9cf612d6a4898ae06820363a"}
Sep 07 13:06:53.917733 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.917419    2567 scope.go:117] "RemoveContainer" containerID="f16424ce17c0e9d8e84e65e9f9cc132e24f9153c711acec6d813d962aa414194"
Sep 07 13:06:53.925891 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.925874    2567 scope.go:117] "RemoveContainer" containerID="f16424ce17c0e9d8e84e65e9f9cc132e24f9153c711acec6d813d962aa414194"
Sep 07 13:06:53.926182 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:06:53.926164    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16424ce17c0e9d8e84e65e9f9cc132e24f9153c711acec6d813d962aa414194\": container with ID starting with f16424ce17c0e9d8e84e65e9f9cc132e24f9153c711acec6d813d962aa414194 not found: ID does not exist" containerID="f16424ce17c0e9d8e84e65e9f9cc132e24f9153c711acec6d813d962aa414194"
Sep 07 13:06:53.926243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.926189    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16424ce17c0e9d8e84e65e9f9cc132e24f9153c711acec6d813d962aa414194"} err="failed to get container status \"f16424ce17c0e9d8e84e65e9f9cc132e24f9153c711acec6d813d962aa414194\": rpc error: code = NotFound desc = could not find container \"f16424ce17c0e9d8e84e65e9f9cc132e24f9153c711acec6d813d962aa414194\": container with ID starting with f16424ce17c0e9d8e84e65e9f9cc132e24f9153c711acec6d813d962aa414194 not found: ID does not exist"
Sep 07 13:06:53.995774 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:53.995738    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"]
Sep 07 13:06:54.021898 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:54.021858    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-prmd7"]
Sep 07 13:06:54.075440 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:06:54.075408    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8789395-0cff-4904-a9fb-75c07c7079b9" path="/var/lib/kubelet/pods/c8789395-0cff-4904-a9fb-75c07c7079b9/volumes"
Sep 07 13:07:49.463114 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.463079    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"]
Sep 07 13:07:49.463590 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.463396    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8789395-0cff-4904-a9fb-75c07c7079b9" containerName="discovery"
Sep 07 13:07:49.463590 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.463410    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8789395-0cff-4904-a9fb-75c07c7079b9" containerName="discovery"
Sep 07 13:07:49.463590 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.463479    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8789395-0cff-4904-a9fb-75c07c7079b9" containerName="discovery"
Sep 07 13:07:49.466389 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.466373    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"
Sep 07 13:07:49.468376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.468351    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4mdkr\""
Sep 07 13:07:49.468503 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.468414    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 13:07:49.468768 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.468750    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 13:07:49.477085 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.477044    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"]
Sep 07 13:07:49.568838 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.568806    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"]
Sep 07 13:07:49.572355 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.572333    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"
Sep 07 13:07:49.588104 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.588053    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/190dbfbd-5238-4e4d-b65f-24c0013e2441-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj\" (UID: \"190dbfbd-5238-4e4d-b65f-24c0013e2441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"
Sep 07 13:07:49.588313 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.588286    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/190dbfbd-5238-4e4d-b65f-24c0013e2441-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj\" (UID: \"190dbfbd-5238-4e4d-b65f-24c0013e2441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"
Sep 07 13:07:49.588421 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.588366    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpt9\" (UniqueName: \"kubernetes.io/projected/190dbfbd-5238-4e4d-b65f-24c0013e2441-kube-api-access-rzpt9\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj\" (UID: \"190dbfbd-5238-4e4d-b65f-24c0013e2441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"
Sep 07 13:07:49.589454 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.589433    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"]
Sep 07 13:07:49.689019 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.688989    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/190dbfbd-5238-4e4d-b65f-24c0013e2441-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj\" (UID: \"190dbfbd-5238-4e4d-b65f-24c0013e2441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"
Sep 07 13:07:49.689213 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.689028    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s\" (UID: \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"
Sep 07 13:07:49.689213 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.689088    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s\" (UID: \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"
Sep 07 13:07:49.689213 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.689129    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxc9t\" (UniqueName: \"kubernetes.io/projected/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-kube-api-access-fxc9t\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s\" (UID: \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"
Sep 07 13:07:49.689213 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.689188    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpt9\" (UniqueName: \"kubernetes.io/projected/190dbfbd-5238-4e4d-b65f-24c0013e2441-kube-api-access-rzpt9\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj\" (UID: \"190dbfbd-5238-4e4d-b65f-24c0013e2441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"
Sep 07 13:07:49.689366 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.689298    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/190dbfbd-5238-4e4d-b65f-24c0013e2441-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj\" (UID: \"190dbfbd-5238-4e4d-b65f-24c0013e2441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"
Sep 07 13:07:49.689469 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.689448    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/190dbfbd-5238-4e4d-b65f-24c0013e2441-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj\" (UID: \"190dbfbd-5238-4e4d-b65f-24c0013e2441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"
Sep 07 13:07:49.689559 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.689541    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/190dbfbd-5238-4e4d-b65f-24c0013e2441-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj\" (UID: \"190dbfbd-5238-4e4d-b65f-24c0013e2441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"
Sep 07 13:07:49.698905 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.698879    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpt9\" (UniqueName: \"kubernetes.io/projected/190dbfbd-5238-4e4d-b65f-24c0013e2441-kube-api-access-rzpt9\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj\" (UID: \"190dbfbd-5238-4e4d-b65f-24c0013e2441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"
Sep 07 13:07:49.723375 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.723321    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"]
Sep 07 13:07:49.726659 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.726644    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"
Sep 07 13:07:49.737741 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.737718    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"]
Sep 07 13:07:49.775432 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.775406    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"
Sep 07 13:07:49.780744 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.780721    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"]
Sep 07 13:07:49.784038 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.784020    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"
Sep 07 13:07:49.790513 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.790489    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sn6t\" (UniqueName: \"kubernetes.io/projected/6e4b3c4b-b106-4eef-a014-64fbfda8163e-kube-api-access-8sn6t\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9\" (UID: \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"
Sep 07 13:07:49.790618 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.790523    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e4b3c4b-b106-4eef-a014-64fbfda8163e-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9\" (UID: \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"
Sep 07 13:07:49.790618 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.790563    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s\" (UID: \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"
Sep 07 13:07:49.790618 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.790585    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s\" (UID: \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"
Sep 07 13:07:49.790618 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.790607    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxc9t\" (UniqueName: \"kubernetes.io/projected/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-kube-api-access-fxc9t\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s\" (UID: \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"
Sep 07 13:07:49.790825 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.790719    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e4b3c4b-b106-4eef-a014-64fbfda8163e-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9\" (UID: \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"
Sep 07 13:07:49.790994 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.790943    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s\" (UID: \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"
Sep 07 13:07:49.791091 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.790998    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s\" (UID: \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"
Sep 07 13:07:49.793485 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.793462    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"]
Sep 07 13:07:49.800192 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.800171    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxc9t\" (UniqueName: \"kubernetes.io/projected/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-kube-api-access-fxc9t\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s\" (UID: \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"
Sep 07 13:07:49.880522 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.880485    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"
Sep 07 13:07:49.891904 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.891868    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs\" (UID: \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"
Sep 07 13:07:49.892018 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.891936    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e4b3c4b-b106-4eef-a014-64fbfda8163e-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9\" (UID: \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"
Sep 07 13:07:49.892093 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.892023    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbb9\" (UniqueName: \"kubernetes.io/projected/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-kube-api-access-gzbb9\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs\" (UID: \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"
Sep 07 13:07:49.892149 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.892114    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sn6t\" (UniqueName: \"kubernetes.io/projected/6e4b3c4b-b106-4eef-a014-64fbfda8163e-kube-api-access-8sn6t\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9\" (UID: \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"
Sep 07 13:07:49.892195 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.892156    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e4b3c4b-b106-4eef-a014-64fbfda8163e-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9\" (UID: \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"
Sep 07 13:07:49.892273 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.892246    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e4b3c4b-b106-4eef-a014-64fbfda8163e-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9\" (UID: \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"
Sep 07 13:07:49.892392 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.892271    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs\" (UID: \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"
Sep 07 13:07:49.892538 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.892520    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e4b3c4b-b106-4eef-a014-64fbfda8163e-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9\" (UID: \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"
Sep 07 13:07:49.900750 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.900726    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sn6t\" (UniqueName: \"kubernetes.io/projected/6e4b3c4b-b106-4eef-a014-64fbfda8163e-kube-api-access-8sn6t\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9\" (UID: \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"
Sep 07 13:07:49.992858 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.992766    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs\" (UID: \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"
Sep 07 13:07:49.992858 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.992840    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs\" (UID: \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"
Sep 07 13:07:49.993091 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.992905    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbb9\" (UniqueName: \"kubernetes.io/projected/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-kube-api-access-gzbb9\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs\" (UID: \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"
Sep 07 13:07:49.993272 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.993246    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs\" (UID: \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"
Sep 07 13:07:49.993338 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:49.993270    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs\" (UID: \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"
Sep 07 13:07:50.002697 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:50.002674    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbb9\" (UniqueName: \"kubernetes.io/projected/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-kube-api-access-gzbb9\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs\" (UID: \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"
Sep 07 13:07:50.035392 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:50.035371    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"
Sep 07 13:07:50.053905 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:50.053885    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"]
Sep 07 13:07:50.055184 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:07:50.055164    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190dbfbd_5238_4e4d_b65f_24c0013e2441.slice/crio-c36f6beb1db3eaff13b936e5115eac4ba267e674742882fd1f9c3425004a5b9f WatchSource:0}: Error finding container c36f6beb1db3eaff13b936e5115eac4ba267e674742882fd1f9c3425004a5b9f: Status 404 returned error can't find the container with id c36f6beb1db3eaff13b936e5115eac4ba267e674742882fd1f9c3425004a5b9f
Sep 07 13:07:50.110913 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:50.110890    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"
Sep 07 13:07:50.153248 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:50.153221    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"]
Sep 07 13:07:50.156658 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:07:50.156610    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92f9065_e2ef_4b5d_b2f9_fd86cbcedf1d.slice/crio-d5e3e5585c6cc99d900e8effbf50a0af06a59fc0e74897fdaee5edbf0bcbe458 WatchSource:0}: Error finding container d5e3e5585c6cc99d900e8effbf50a0af06a59fc0e74897fdaee5edbf0bcbe458: Status 404 returned error can't find the container with id d5e3e5585c6cc99d900e8effbf50a0af06a59fc0e74897fdaee5edbf0bcbe458
Sep 07 13:07:50.169661 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:50.169626    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s" event={"ID":"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d","Type":"ContainerStarted","Data":"d5e3e5585c6cc99d900e8effbf50a0af06a59fc0e74897fdaee5edbf0bcbe458"}
Sep 07 13:07:50.178738 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:50.177995    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj" event={"ID":"190dbfbd-5238-4e4d-b65f-24c0013e2441","Type":"ContainerStarted","Data":"b22ce8cb6f19dc9e6629c6c4c640496ec8881185d480628a71e0d700c2f6ce62"}
Sep 07 13:07:50.178738 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:50.178603    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj" event={"ID":"190dbfbd-5238-4e4d-b65f-24c0013e2441","Type":"ContainerStarted","Data":"c36f6beb1db3eaff13b936e5115eac4ba267e674742882fd1f9c3425004a5b9f"}
Sep 07 13:07:50.309017 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:50.308989    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"]
Sep 07 13:07:50.310180 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:07:50.310150    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4b3c4b_b106_4eef_a014_64fbfda8163e.slice/crio-b1d2cc0a06a62693e25f489c885b9c20fe08ee28d0a4d3b32ff4a20261fc2107 WatchSource:0}: Error finding container b1d2cc0a06a62693e25f489c885b9c20fe08ee28d0a4d3b32ff4a20261fc2107: Status 404 returned error can't find the container with id b1d2cc0a06a62693e25f489c885b9c20fe08ee28d0a4d3b32ff4a20261fc2107
Sep 07 13:07:50.395116 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:50.395089    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"]
Sep 07 13:07:50.397189 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:07:50.397164    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c4bf4e_b0e5_4a9f_a718_87be159e19eb.slice/crio-d57c7a4b753a5f2ea1e79a7f080cf970e6e71b327ac60ce7a8a6cba56b13ad64 WatchSource:0}: Error finding container d57c7a4b753a5f2ea1e79a7f080cf970e6e71b327ac60ce7a8a6cba56b13ad64: Status 404 returned error can't find the container with id d57c7a4b753a5f2ea1e79a7f080cf970e6e71b327ac60ce7a8a6cba56b13ad64
Sep 07 13:07:51.183901 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:51.183866    2567 generic.go:356] "Generic (PLEG): container finished" podID="90c4bf4e-b0e5-4a9f-a718-87be159e19eb" containerID="a7bcdc1c1c8f738a1b9d4db5bfc19e509aefa3463f434d670df4b131f95dd161" exitCode=0
Sep 07 13:07:51.184364 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:51.183955    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs" event={"ID":"90c4bf4e-b0e5-4a9f-a718-87be159e19eb","Type":"ContainerDied","Data":"a7bcdc1c1c8f738a1b9d4db5bfc19e509aefa3463f434d670df4b131f95dd161"}
Sep 07 13:07:51.184364 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:51.184009    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs" event={"ID":"90c4bf4e-b0e5-4a9f-a718-87be159e19eb","Type":"ContainerStarted","Data":"d57c7a4b753a5f2ea1e79a7f080cf970e6e71b327ac60ce7a8a6cba56b13ad64"}
Sep 07 13:07:51.185517 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:51.185495    2567 generic.go:356] "Generic (PLEG): container finished" podID="6e4b3c4b-b106-4eef-a014-64fbfda8163e" containerID="700edf04bc0343ff4d9ac5c1c86f13c3ab8859acb5957885b124e202736d36f6" exitCode=0
Sep 07 13:07:51.185580 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:51.185545    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9" event={"ID":"6e4b3c4b-b106-4eef-a014-64fbfda8163e","Type":"ContainerDied","Data":"700edf04bc0343ff4d9ac5c1c86f13c3ab8859acb5957885b124e202736d36f6"}
Sep 07 13:07:51.185637 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:51.185596    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9" event={"ID":"6e4b3c4b-b106-4eef-a014-64fbfda8163e","Type":"ContainerStarted","Data":"b1d2cc0a06a62693e25f489c885b9c20fe08ee28d0a4d3b32ff4a20261fc2107"}
Sep 07 13:07:51.187297 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:51.187270    2567 generic.go:356] "Generic (PLEG): container finished" podID="b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d" containerID="16dc1e8582dd32e2a9ef3b9ebf781b94b6cd408008aa16491546f75068dab94d" exitCode=0
Sep 07 13:07:51.187396 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:51.187351    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s" event={"ID":"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d","Type":"ContainerDied","Data":"16dc1e8582dd32e2a9ef3b9ebf781b94b6cd408008aa16491546f75068dab94d"}
Sep 07 13:07:51.189180 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:51.189161    2567 generic.go:356] "Generic (PLEG): container finished" podID="190dbfbd-5238-4e4d-b65f-24c0013e2441" containerID="b22ce8cb6f19dc9e6629c6c4c640496ec8881185d480628a71e0d700c2f6ce62" exitCode=0
Sep 07 13:07:51.189264 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:51.189189    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj" event={"ID":"190dbfbd-5238-4e4d-b65f-24c0013e2441","Type":"ContainerDied","Data":"b22ce8cb6f19dc9e6629c6c4c640496ec8881185d480628a71e0d700c2f6ce62"}
Sep 07 13:07:53.200249 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:53.200214    2567 generic.go:356] "Generic (PLEG): container finished" podID="b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d" containerID="47e989a1dfb61b23463813d606fe5a99247ec34043bc872b262a00362b9c2e7f" exitCode=0
Sep 07 13:07:53.200703 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:53.200303    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s" event={"ID":"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d","Type":"ContainerDied","Data":"47e989a1dfb61b23463813d606fe5a99247ec34043bc872b262a00362b9c2e7f"}
Sep 07 13:07:53.202425 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:53.202400    2567 generic.go:356] "Generic (PLEG): container finished" podID="190dbfbd-5238-4e4d-b65f-24c0013e2441" containerID="f0f9de301c5439b4294a7e3076981692e49f46ea5a2fa168e937870967efeff1" exitCode=0
Sep 07 13:07:53.202542 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:53.202475    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj" event={"ID":"190dbfbd-5238-4e4d-b65f-24c0013e2441","Type":"ContainerDied","Data":"f0f9de301c5439b4294a7e3076981692e49f46ea5a2fa168e937870967efeff1"}
Sep 07 13:07:53.204525 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:53.204503    2567 generic.go:356] "Generic (PLEG): container finished" podID="6e4b3c4b-b106-4eef-a014-64fbfda8163e" containerID="2e2702ed5d6d20717aa8bf4d53573de772ed018020f482b16261221e0c146298" exitCode=0
Sep 07 13:07:53.204622 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:53.204547    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9" event={"ID":"6e4b3c4b-b106-4eef-a014-64fbfda8163e","Type":"ContainerDied","Data":"2e2702ed5d6d20717aa8bf4d53573de772ed018020f482b16261221e0c146298"}
Sep 07 13:07:54.216113 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:54.216076    2567 generic.go:356] "Generic (PLEG): container finished" podID="190dbfbd-5238-4e4d-b65f-24c0013e2441" containerID="5d100f7ae9d1dc6c07fefe4e027ab19b7b8967c97211e2ba43ce873a45cf253b" exitCode=0
Sep 07 13:07:54.216513 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:54.216184    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj" event={"ID":"190dbfbd-5238-4e4d-b65f-24c0013e2441","Type":"ContainerDied","Data":"5d100f7ae9d1dc6c07fefe4e027ab19b7b8967c97211e2ba43ce873a45cf253b"}
Sep 07 13:07:54.218407 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:54.218381    2567 generic.go:356] "Generic (PLEG): container finished" podID="6e4b3c4b-b106-4eef-a014-64fbfda8163e" containerID="3bf3d83d09389ad72c3796dec2b3a4d59d344c85b2a4198392ecdba10ca1cc4f" exitCode=0
Sep 07 13:07:54.218527 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:54.218463    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9" event={"ID":"6e4b3c4b-b106-4eef-a014-64fbfda8163e","Type":"ContainerDied","Data":"3bf3d83d09389ad72c3796dec2b3a4d59d344c85b2a4198392ecdba10ca1cc4f"}
Sep 07 13:07:54.220598 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:54.220576    2567 generic.go:356] "Generic (PLEG): container finished" podID="b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d" containerID="d9f366e8795fcf14f5f8186584c7d5a8bbabc423d0201585ed39e5ddb33b6b6c" exitCode=0
Sep 07 13:07:54.220713 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:54.220616    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s" event={"ID":"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d","Type":"ContainerDied","Data":"d9f366e8795fcf14f5f8186584c7d5a8bbabc423d0201585ed39e5ddb33b6b6c"}
Sep 07 13:07:55.367565 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.367540    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"
Sep 07 13:07:55.391652 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.391627    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"
Sep 07 13:07:55.394848 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.394823    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"
Sep 07 13:07:55.441966 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.441938    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/190dbfbd-5238-4e4d-b65f-24c0013e2441-bundle\") pod \"190dbfbd-5238-4e4d-b65f-24c0013e2441\" (UID: \"190dbfbd-5238-4e4d-b65f-24c0013e2441\") "
Sep 07 13:07:55.442143 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.442018    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-bundle\") pod \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\" (UID: \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\") "
Sep 07 13:07:55.442143 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.442042    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-util\") pod \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\" (UID: \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\") "
Sep 07 13:07:55.442143 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.442084    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/190dbfbd-5238-4e4d-b65f-24c0013e2441-util\") pod \"190dbfbd-5238-4e4d-b65f-24c0013e2441\" (UID: \"190dbfbd-5238-4e4d-b65f-24c0013e2441\") "
Sep 07 13:07:55.442143 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.442137    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzpt9\" (UniqueName: \"kubernetes.io/projected/190dbfbd-5238-4e4d-b65f-24c0013e2441-kube-api-access-rzpt9\") pod \"190dbfbd-5238-4e4d-b65f-24c0013e2441\" (UID: \"190dbfbd-5238-4e4d-b65f-24c0013e2441\") "
Sep 07 13:07:55.442365 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.442162    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxc9t\" (UniqueName: \"kubernetes.io/projected/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-kube-api-access-fxc9t\") pod \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\" (UID: \"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d\") "
Sep 07 13:07:55.442817 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.442774    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-bundle" (OuterVolumeSpecName: "bundle") pod "b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d" (UID: "b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:07:55.442970 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.442936    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190dbfbd-5238-4e4d-b65f-24c0013e2441-bundle" (OuterVolumeSpecName: "bundle") pod "190dbfbd-5238-4e4d-b65f-24c0013e2441" (UID: "190dbfbd-5238-4e4d-b65f-24c0013e2441"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:07:55.444549 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.444515    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-kube-api-access-fxc9t" (OuterVolumeSpecName: "kube-api-access-fxc9t") pod "b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d" (UID: "b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d"). InnerVolumeSpecName "kube-api-access-fxc9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:07:55.444744 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.444723    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190dbfbd-5238-4e4d-b65f-24c0013e2441-kube-api-access-rzpt9" (OuterVolumeSpecName: "kube-api-access-rzpt9") pod "190dbfbd-5238-4e4d-b65f-24c0013e2441" (UID: "190dbfbd-5238-4e4d-b65f-24c0013e2441"). InnerVolumeSpecName "kube-api-access-rzpt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:07:55.447776 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.447753    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-util" (OuterVolumeSpecName: "util") pod "b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d" (UID: "b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:07:55.448838 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.448807    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190dbfbd-5238-4e4d-b65f-24c0013e2441-util" (OuterVolumeSpecName: "util") pod "190dbfbd-5238-4e4d-b65f-24c0013e2441" (UID: "190dbfbd-5238-4e4d-b65f-24c0013e2441"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:07:55.542863 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.542840    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e4b3c4b-b106-4eef-a014-64fbfda8163e-util\") pod \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\" (UID: \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\") "
Sep 07 13:07:55.543012 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.542920    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sn6t\" (UniqueName: \"kubernetes.io/projected/6e4b3c4b-b106-4eef-a014-64fbfda8163e-kube-api-access-8sn6t\") pod \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\" (UID: \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\") "
Sep 07 13:07:55.543012 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.542939    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e4b3c4b-b106-4eef-a014-64fbfda8163e-bundle\") pod \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\" (UID: \"6e4b3c4b-b106-4eef-a014-64fbfda8163e\") "
Sep 07 13:07:55.543173 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.543093    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-bundle\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:07:55.543173 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.543107    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-util\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:07:55.543173 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.543114    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/190dbfbd-5238-4e4d-b65f-24c0013e2441-util\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:07:55.543173 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.543122    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzpt9\" (UniqueName: \"kubernetes.io/projected/190dbfbd-5238-4e4d-b65f-24c0013e2441-kube-api-access-rzpt9\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:07:55.543173 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.543139    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxc9t\" (UniqueName: \"kubernetes.io/projected/b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d-kube-api-access-fxc9t\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:07:55.543173 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.543148    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/190dbfbd-5238-4e4d-b65f-24c0013e2441-bundle\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:07:55.543541 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.543514    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4b3c4b-b106-4eef-a014-64fbfda8163e-bundle" (OuterVolumeSpecName: "bundle") pod "6e4b3c4b-b106-4eef-a014-64fbfda8163e" (UID: "6e4b3c4b-b106-4eef-a014-64fbfda8163e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:07:55.544915 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.544894    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4b3c4b-b106-4eef-a014-64fbfda8163e-kube-api-access-8sn6t" (OuterVolumeSpecName: "kube-api-access-8sn6t") pod "6e4b3c4b-b106-4eef-a014-64fbfda8163e" (UID: "6e4b3c4b-b106-4eef-a014-64fbfda8163e"). InnerVolumeSpecName "kube-api-access-8sn6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:07:55.548596 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.548576    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4b3c4b-b106-4eef-a014-64fbfda8163e-util" (OuterVolumeSpecName: "util") pod "6e4b3c4b-b106-4eef-a014-64fbfda8163e" (UID: "6e4b3c4b-b106-4eef-a014-64fbfda8163e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:07:55.644418 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.644384    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sn6t\" (UniqueName: \"kubernetes.io/projected/6e4b3c4b-b106-4eef-a014-64fbfda8163e-kube-api-access-8sn6t\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:07:55.644418 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.644411    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e4b3c4b-b106-4eef-a014-64fbfda8163e-bundle\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:07:55.644418 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:55.644423    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e4b3c4b-b106-4eef-a014-64fbfda8163e-util\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:07:56.231628 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:56.231603    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s"
Sep 07 13:07:56.231808 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:56.231609    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7294s" event={"ID":"b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d","Type":"ContainerDied","Data":"d5e3e5585c6cc99d900e8effbf50a0af06a59fc0e74897fdaee5edbf0bcbe458"}
Sep 07 13:07:56.231808 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:56.231717    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5e3e5585c6cc99d900e8effbf50a0af06a59fc0e74897fdaee5edbf0bcbe458"
Sep 07 13:07:56.233679 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:56.233651    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj" event={"ID":"190dbfbd-5238-4e4d-b65f-24c0013e2441","Type":"ContainerDied","Data":"c36f6beb1db3eaff13b936e5115eac4ba267e674742882fd1f9c3425004a5b9f"}
Sep 07 13:07:56.233679 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:56.233671    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a02hmnj"
Sep 07 13:07:56.233938 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:56.233713    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c36f6beb1db3eaff13b936e5115eac4ba267e674742882fd1f9c3425004a5b9f"
Sep 07 13:07:56.235907 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:56.235881    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9" event={"ID":"6e4b3c4b-b106-4eef-a014-64fbfda8163e","Type":"ContainerDied","Data":"b1d2cc0a06a62693e25f489c885b9c20fe08ee28d0a4d3b32ff4a20261fc2107"}
Sep 07 13:07:56.236017 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:56.235914    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1d2cc0a06a62693e25f489c885b9c20fe08ee28d0a4d3b32ff4a20261fc2107"
Sep 07 13:07:56.236017 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:07:56.235927    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbp4ww9"
Sep 07 13:08:15.329633 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:15.329596    2567 generic.go:356] "Generic (PLEG): container finished" podID="90c4bf4e-b0e5-4a9f-a718-87be159e19eb" containerID="2170d8e10d05e388d7133d9480f2801d39811f1e4dd2b6ed714adc1bba187eb6" exitCode=0
Sep 07 13:08:15.330013 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:15.329676    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs" event={"ID":"90c4bf4e-b0e5-4a9f-a718-87be159e19eb","Type":"ContainerDied","Data":"2170d8e10d05e388d7133d9480f2801d39811f1e4dd2b6ed714adc1bba187eb6"}
Sep 07 13:08:16.336803 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:16.336769    2567 generic.go:356] "Generic (PLEG): container finished" podID="90c4bf4e-b0e5-4a9f-a718-87be159e19eb" containerID="c9170fc37e9d4ec9630ec70c0323309e4c56334ef599bba792735e88d9ef9976" exitCode=0
Sep 07 13:08:16.337187 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:16.336824    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs" event={"ID":"90c4bf4e-b0e5-4a9f-a718-87be159e19eb","Type":"ContainerDied","Data":"c9170fc37e9d4ec9630ec70c0323309e4c56334ef599bba792735e88d9ef9976"}
Sep 07 13:08:17.475007 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:17.474985    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"
Sep 07 13:08:17.633033 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:17.632941    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzbb9\" (UniqueName: \"kubernetes.io/projected/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-kube-api-access-gzbb9\") pod \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\" (UID: \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\") "
Sep 07 13:08:17.633033 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:17.632998    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-bundle\") pod \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\" (UID: \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\") "
Sep 07 13:08:17.633285 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:17.633043    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-util\") pod \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\" (UID: \"90c4bf4e-b0e5-4a9f-a718-87be159e19eb\") "
Sep 07 13:08:17.633674 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:17.633640    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-bundle" (OuterVolumeSpecName: "bundle") pod "90c4bf4e-b0e5-4a9f-a718-87be159e19eb" (UID: "90c4bf4e-b0e5-4a9f-a718-87be159e19eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:17.635114 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:17.635089    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-kube-api-access-gzbb9" (OuterVolumeSpecName: "kube-api-access-gzbb9") pod "90c4bf4e-b0e5-4a9f-a718-87be159e19eb" (UID: "90c4bf4e-b0e5-4a9f-a718-87be159e19eb"). InnerVolumeSpecName "kube-api-access-gzbb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:08:17.637610 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:17.637583    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-util" (OuterVolumeSpecName: "util") pod "90c4bf4e-b0e5-4a9f-a718-87be159e19eb" (UID: "90c4bf4e-b0e5-4a9f-a718-87be159e19eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:17.734043 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:17.734009    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzbb9\" (UniqueName: \"kubernetes.io/projected/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-kube-api-access-gzbb9\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:08:17.734043 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:17.734039    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-bundle\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:08:17.734245 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:17.734051    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90c4bf4e-b0e5-4a9f-a718-87be159e19eb-util\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:08:18.348676 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:18.348601    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs"
Sep 07 13:08:18.348808 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:18.348601    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053ftrcs" event={"ID":"90c4bf4e-b0e5-4a9f-a718-87be159e19eb","Type":"ContainerDied","Data":"d57c7a4b753a5f2ea1e79a7f080cf970e6e71b327ac60ce7a8a6cba56b13ad64"}
Sep 07 13:08:18.348808 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:18.348722    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d57c7a4b753a5f2ea1e79a7f080cf970e6e71b327ac60ce7a8a6cba56b13ad64"
Sep 07 13:08:28.644698 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.644662    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-6wh7q"]
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.644957    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4b3c4b-b106-4eef-a014-64fbfda8163e" containerName="pull"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.644969    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4b3c4b-b106-4eef-a014-64fbfda8163e" containerName="pull"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.644983    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c4bf4e-b0e5-4a9f-a718-87be159e19eb" containerName="pull"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.644988    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c4bf4e-b0e5-4a9f-a718-87be159e19eb" containerName="pull"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.644996    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4b3c4b-b106-4eef-a014-64fbfda8163e" containerName="util"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645002    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4b3c4b-b106-4eef-a014-64fbfda8163e" containerName="util"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645012    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190dbfbd-5238-4e4d-b65f-24c0013e2441" containerName="pull"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645018    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="190dbfbd-5238-4e4d-b65f-24c0013e2441" containerName="pull"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645023    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d" containerName="pull"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645028    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d" containerName="pull"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645035    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d" containerName="extract"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645040    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d" containerName="extract"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645047    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4b3c4b-b106-4eef-a014-64fbfda8163e" containerName="extract"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645052    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4b3c4b-b106-4eef-a014-64fbfda8163e" containerName="extract"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645070    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190dbfbd-5238-4e4d-b65f-24c0013e2441" containerName="extract"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645076    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="190dbfbd-5238-4e4d-b65f-24c0013e2441" containerName="extract"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645083    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d" containerName="util"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645088    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d" containerName="util"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645096    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c4bf4e-b0e5-4a9f-a718-87be159e19eb" containerName="util"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645101    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c4bf4e-b0e5-4a9f-a718-87be159e19eb" containerName="util"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645106    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c4bf4e-b0e5-4a9f-a718-87be159e19eb" containerName="extract"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645111    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c4bf4e-b0e5-4a9f-a718-87be159e19eb" containerName="extract"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645118    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190dbfbd-5238-4e4d-b65f-24c0013e2441" containerName="util"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645123    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="190dbfbd-5238-4e4d-b65f-24c0013e2441" containerName="util"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645169    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c4bf4e-b0e5-4a9f-a718-87be159e19eb" containerName="extract"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645176    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e4b3c4b-b106-4eef-a014-64fbfda8163e" containerName="extract"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645181    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b92f9065-e2ef-4b5d-b2f9-fd86cbcedf1d" containerName="extract"
Sep 07 13:08:28.645191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.645186    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="190dbfbd-5238-4e4d-b65f-24c0013e2441" containerName="extract"
Sep 07 13:08:28.648377 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.648360    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-6wh7q"
Sep 07 13:08:28.652067 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.652032    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 07 13:08:28.652207 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.652148    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-j6n5k\""
Sep 07 13:08:28.652275 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.652259    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 13:08:28.652406 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.652390    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 13:08:28.660674 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.660651    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-6wh7q"]
Sep 07 13:08:28.708207 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.708165    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzctn\" (UniqueName: \"kubernetes.io/projected/81c4e661-8058-4bc7-bcfd-8939b6c4f93f-kube-api-access-qzctn\") pod \"dns-operator-controller-manager-5985564776-6wh7q\" (UID: \"81c4e661-8058-4bc7-bcfd-8939b6c4f93f\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-6wh7q"
Sep 07 13:08:28.809635 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.809594    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzctn\" (UniqueName: \"kubernetes.io/projected/81c4e661-8058-4bc7-bcfd-8939b6c4f93f-kube-api-access-qzctn\") pod \"dns-operator-controller-manager-5985564776-6wh7q\" (UID: \"81c4e661-8058-4bc7-bcfd-8939b6c4f93f\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-6wh7q"
Sep 07 13:08:28.829259 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.829221    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzctn\" (UniqueName: \"kubernetes.io/projected/81c4e661-8058-4bc7-bcfd-8939b6c4f93f-kube-api-access-qzctn\") pod \"dns-operator-controller-manager-5985564776-6wh7q\" (UID: \"81c4e661-8058-4bc7-bcfd-8939b6c4f93f\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-6wh7q"
Sep 07 13:08:28.958343 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:28.958249    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-6wh7q"
Sep 07 13:08:29.232895 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:29.232865    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-6wh7q"]
Sep 07 13:08:29.234652 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:08:29.234620    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c4e661_8058_4bc7_bcfd_8939b6c4f93f.slice/crio-a426a535fae68c3ceb7bb03dc134c64259099a6c3fbc0a66de26ce02e9bf702a WatchSource:0}: Error finding container a426a535fae68c3ceb7bb03dc134c64259099a6c3fbc0a66de26ce02e9bf702a: Status 404 returned error can't find the container with id a426a535fae68c3ceb7bb03dc134c64259099a6c3fbc0a66de26ce02e9bf702a
Sep 07 13:08:29.404473 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:29.404432    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-6wh7q" event={"ID":"81c4e661-8058-4bc7-bcfd-8939b6c4f93f","Type":"ContainerStarted","Data":"a426a535fae68c3ceb7bb03dc134c64259099a6c3fbc0a66de26ce02e9bf702a"}
Sep 07 13:08:32.421725 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:32.421687    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-6wh7q" event={"ID":"81c4e661-8058-4bc7-bcfd-8939b6c4f93f","Type":"ContainerStarted","Data":"4624be4c352d3e893bf3e081bc8b2cafa769318959b9f4772c8be805ec28cfc7"}
Sep 07 13:08:32.422133 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:32.421763    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-6wh7q"
Sep 07 13:08:32.457657 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:32.457607    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-6wh7q" podStartSLOduration=2.006497648 podStartE2EDuration="4.457591231s" totalImagesPullingTime="2.451093583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:08:32.455880207 +0000 UTC m=+556.884178480" watchObservedRunningTime="2026-09-07 13:08:32.457591231 +0000 UTC m=+556.885889516"
Sep 07 13:08:33.466154 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:33.466120    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"]
Sep 07 13:08:33.469322 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:33.469305    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:33.473107 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:33.473087    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tgv8t\""
Sep 07 13:08:33.489054 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:33.489030    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"]
Sep 07 13:08:33.544503 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:33.544462    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-5bhd7\" (UID: \"7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:33.544681 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:33.544514    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtxfx\" (UniqueName: \"kubernetes.io/projected/7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8-kube-api-access-rtxfx\") pod \"kuadrant-operator-controller-manager-58fc56757c-5bhd7\" (UID: \"7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:33.644934 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:33.644890    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-5bhd7\" (UID: \"7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:33.645140 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:33.644943    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtxfx\" (UniqueName: \"kubernetes.io/projected/7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8-kube-api-access-rtxfx\") pod \"kuadrant-operator-controller-manager-58fc56757c-5bhd7\" (UID: \"7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:33.645369 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:33.645345    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-5bhd7\" (UID: \"7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:33.655618 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:33.655594    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtxfx\" (UniqueName: \"kubernetes.io/projected/7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8-kube-api-access-rtxfx\") pod \"kuadrant-operator-controller-manager-58fc56757c-5bhd7\" (UID: \"7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:33.779270 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:33.779228    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:34.097969 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:34.097786    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"]
Sep 07 13:08:34.100354 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:08:34.100323    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf17dfd_2f05_4ab3_bffd_cf4485cfc1c8.slice/crio-4f368af16cc27ff6fc53ba74dd370774bf29de329de4616d3ca5a3a98201c5a7 WatchSource:0}: Error finding container 4f368af16cc27ff6fc53ba74dd370774bf29de329de4616d3ca5a3a98201c5a7: Status 404 returned error can't find the container with id 4f368af16cc27ff6fc53ba74dd370774bf29de329de4616d3ca5a3a98201c5a7
Sep 07 13:08:34.434909 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:34.434818    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7" event={"ID":"7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8","Type":"ContainerStarted","Data":"4f368af16cc27ff6fc53ba74dd370774bf29de329de4616d3ca5a3a98201c5a7"}
Sep 07 13:08:38.459262 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:38.459164    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7" event={"ID":"7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8","Type":"ContainerStarted","Data":"cdfe44a9e4db1b08ff77cae9f7fc4dfc9464c52d47eebb647fe64cc6c496730f"}
Sep 07 13:08:38.459696 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:38.459292    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:38.490975 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:38.490918    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7" podStartSLOduration=1.490368187 podStartE2EDuration="5.490901607s" totalImagesPullingTime="4.00053342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:08:38.488285112 +0000 UTC m=+562.916583406" watchObservedRunningTime="2026-09-07 13:08:38.490901607 +0000 UTC m=+562.919199879"
Sep 07 13:08:43.429219 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:43.429179    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-6wh7q"
Sep 07 13:08:49.466322 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:49.466286    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:52.067109 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.067071    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"]
Sep 07 13:08:52.067532 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.067263    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7" podUID="7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" containerName="manager" containerID="cri-o://cdfe44a9e4db1b08ff77cae9f7fc4dfc9464c52d47eebb647fe64cc6c496730f" gracePeriod=2
Sep 07 13:08:52.093766 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.093736    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"]
Sep 07 13:08:52.118924 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:08:52.118885    2567 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-58fc56757c-5bhd7\": pods \"kuadrant-operator-controller-manager-58fc56757c-5bhd7\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:52.132169 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.132128    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"]
Sep 07 13:08:52.132442 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.132428    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" containerName="manager"
Sep 07 13:08:52.132510 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.132442    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" containerName="manager"
Sep 07 13:08:52.132510 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.132493    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" containerName="manager"
Sep 07 13:08:52.135677 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.135651    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"
Sep 07 13:08:52.176395 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.176364    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"]
Sep 07 13:08:52.192934 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.192903    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d39afa7-cdb9-458b-b530-c8f71a220d76-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-f86dp\" (UID: \"1d39afa7-cdb9-458b-b530-c8f71a220d76\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"
Sep 07 13:08:52.193122 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.192968    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xttgd\" (UniqueName: \"kubernetes.io/projected/1d39afa7-cdb9-458b-b530-c8f71a220d76-kube-api-access-xttgd\") pod \"kuadrant-operator-controller-manager-58fc56757c-f86dp\" (UID: \"1d39afa7-cdb9-458b-b530-c8f71a220d76\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"
Sep 07 13:08:52.210375 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:08:52.210337    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-5bhd7\" is forbidden: User \"system:node:ip-10-0-128-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-114.ec2.internal' and this object" podUID="7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:52.293595 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.293532    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d39afa7-cdb9-458b-b530-c8f71a220d76-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-f86dp\" (UID: \"1d39afa7-cdb9-458b-b530-c8f71a220d76\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"
Sep 07 13:08:52.293595 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.293590    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xttgd\" (UniqueName: \"kubernetes.io/projected/1d39afa7-cdb9-458b-b530-c8f71a220d76-kube-api-access-xttgd\") pod \"kuadrant-operator-controller-manager-58fc56757c-f86dp\" (UID: \"1d39afa7-cdb9-458b-b530-c8f71a220d76\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"
Sep 07 13:08:52.294010 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.293971    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d39afa7-cdb9-458b-b530-c8f71a220d76-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-f86dp\" (UID: \"1d39afa7-cdb9-458b-b530-c8f71a220d76\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"
Sep 07 13:08:52.307517 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.307487    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xttgd\" (UniqueName: \"kubernetes.io/projected/1d39afa7-cdb9-458b-b530-c8f71a220d76-kube-api-access-xttgd\") pod \"kuadrant-operator-controller-manager-58fc56757c-f86dp\" (UID: \"1d39afa7-cdb9-458b-b530-c8f71a220d76\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"
Sep 07 13:08:52.307648 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.307627    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:52.309493 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:08:52.309465    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-5bhd7\" is forbidden: User \"system:node:ip-10-0-128-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-114.ec2.internal' and this object" podUID="7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:52.394249 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.394157    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8-extensions-socket-volume\") pod \"7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8\" (UID: \"7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8\") "
Sep 07 13:08:52.394249 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.394233    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtxfx\" (UniqueName: \"kubernetes.io/projected/7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8-kube-api-access-rtxfx\") pod \"7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8\" (UID: \"7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8\") "
Sep 07 13:08:52.394670 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.394636    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" (UID: "7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:52.396401 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.396378    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8-kube-api-access-rtxfx" (OuterVolumeSpecName: "kube-api-access-rtxfx") pod "7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" (UID: "7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8"). InnerVolumeSpecName "kube-api-access-rtxfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:08:52.456851 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.456811    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"
Sep 07 13:08:52.495400 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.495362    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtxfx\" (UniqueName: \"kubernetes.io/projected/7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8-kube-api-access-rtxfx\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:08:52.495400 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.495401    2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8-extensions-socket-volume\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:08:52.529168 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.529132    2567 generic.go:356] "Generic (PLEG): container finished" podID="7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" containerID="cdfe44a9e4db1b08ff77cae9f7fc4dfc9464c52d47eebb647fe64cc6c496730f" exitCode=0
Sep 07 13:08:52.529327 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.529193    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:52.529327 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.529292    2567 scope.go:117] "RemoveContainer" containerID="cdfe44a9e4db1b08ff77cae9f7fc4dfc9464c52d47eebb647fe64cc6c496730f"
Sep 07 13:08:52.531434 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:08:52.531394    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-5bhd7\" is forbidden: User \"system:node:ip-10-0-128-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-114.ec2.internal' and this object" podUID="7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:52.539736 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.539693    2567 scope.go:117] "RemoveContainer" containerID="cdfe44a9e4db1b08ff77cae9f7fc4dfc9464c52d47eebb647fe64cc6c496730f"
Sep 07 13:08:52.540028 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:08:52.540008    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfe44a9e4db1b08ff77cae9f7fc4dfc9464c52d47eebb647fe64cc6c496730f\": container with ID starting with cdfe44a9e4db1b08ff77cae9f7fc4dfc9464c52d47eebb647fe64cc6c496730f not found: ID does not exist" containerID="cdfe44a9e4db1b08ff77cae9f7fc4dfc9464c52d47eebb647fe64cc6c496730f"
Sep 07 13:08:52.540105 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.540039    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfe44a9e4db1b08ff77cae9f7fc4dfc9464c52d47eebb647fe64cc6c496730f"} err="failed to get container status \"cdfe44a9e4db1b08ff77cae9f7fc4dfc9464c52d47eebb647fe64cc6c496730f\": rpc error: code = NotFound desc = could not find container \"cdfe44a9e4db1b08ff77cae9f7fc4dfc9464c52d47eebb647fe64cc6c496730f\": container with ID starting with cdfe44a9e4db1b08ff77cae9f7fc4dfc9464c52d47eebb647fe64cc6c496730f not found: ID does not exist"
Sep 07 13:08:52.541224 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:08:52.541201    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-5bhd7\" is forbidden: User \"system:node:ip-10-0-128-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-114.ec2.internal' and this object" podUID="7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:52.741666 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:52.741592    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"]
Sep 07 13:08:52.745239 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:08:52.745214    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d39afa7_cdb9_458b_b530_c8f71a220d76.slice/crio-51e19c9a8b33b5c6a01cdfa0b70db53f5f0789643b4306c3c4e8c894b5a55205 WatchSource:0}: Error finding container 51e19c9a8b33b5c6a01cdfa0b70db53f5f0789643b4306c3c4e8c894b5a55205: Status 404 returned error can't find the container with id 51e19c9a8b33b5c6a01cdfa0b70db53f5f0789643b4306c3c4e8c894b5a55205
Sep 07 13:08:53.222107 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.222068    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"]
Sep 07 13:08:53.225301 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.225284    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"
Sep 07 13:08:53.228748 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:08:53.228721    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-5bhd7\" is forbidden: User \"system:node:ip-10-0-128-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-114.ec2.internal' and this object" podUID="7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:53.248672 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.248645    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"]
Sep 07 13:08:53.303609 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.303570    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj9jb\" (UniqueName: \"kubernetes.io/projected/6d2c3c54-003d-4d49-8044-f27430dbdcad-kube-api-access-vj9jb\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-pkx4d\" (UID: \"6d2c3c54-003d-4d49-8044-f27430dbdcad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"
Sep 07 13:08:53.303793 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.303636    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d2c3c54-003d-4d49-8044-f27430dbdcad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-pkx4d\" (UID: \"6d2c3c54-003d-4d49-8044-f27430dbdcad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"
Sep 07 13:08:53.404350 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.404317    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj9jb\" (UniqueName: \"kubernetes.io/projected/6d2c3c54-003d-4d49-8044-f27430dbdcad-kube-api-access-vj9jb\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-pkx4d\" (UID: \"6d2c3c54-003d-4d49-8044-f27430dbdcad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"
Sep 07 13:08:53.404523 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.404365    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d2c3c54-003d-4d49-8044-f27430dbdcad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-pkx4d\" (UID: \"6d2c3c54-003d-4d49-8044-f27430dbdcad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"
Sep 07 13:08:53.404755 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.404736    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d2c3c54-003d-4d49-8044-f27430dbdcad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-pkx4d\" (UID: \"6d2c3c54-003d-4d49-8044-f27430dbdcad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"
Sep 07 13:08:53.429529 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.429499    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj9jb\" (UniqueName: \"kubernetes.io/projected/6d2c3c54-003d-4d49-8044-f27430dbdcad-kube-api-access-vj9jb\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-pkx4d\" (UID: \"6d2c3c54-003d-4d49-8044-f27430dbdcad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"
Sep 07 13:08:53.535087 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.535031    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"
Sep 07 13:08:53.537662 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.537613    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp" event={"ID":"1d39afa7-cdb9-458b-b530-c8f71a220d76","Type":"ContainerStarted","Data":"821271f44a440b89131e9d17374df9ec520e53f1f381da9c8069377646c9e4d8"}
Sep 07 13:08:53.537805 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.537679    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp" event={"ID":"1d39afa7-cdb9-458b-b530-c8f71a220d76","Type":"ContainerStarted","Data":"51e19c9a8b33b5c6a01cdfa0b70db53f5f0789643b4306c3c4e8c894b5a55205"}
Sep 07 13:08:53.537805 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.537788    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"
Sep 07 13:08:53.541362 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:08:53.541334    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-5bhd7\" is forbidden: User \"system:node:ip-10-0-128-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-114.ec2.internal' and this object" podUID="7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-5bhd7"
Sep 07 13:08:53.837738 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.837632    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp" podStartSLOduration=1.837613641 podStartE2EDuration="1.837613641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:08:53.583485173 +0000 UTC m=+578.011783494" watchObservedRunningTime="2026-09-07 13:08:53.837613641 +0000 UTC m=+578.265911915"
Sep 07 13:08:53.838762 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:53.838741    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"]
Sep 07 13:08:53.844013 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:08:53.843980    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2c3c54_003d_4d49_8044_f27430dbdcad.slice/crio-e57f913a409dc3e0daf2de82c0e7368d82d2f6fe3ac24faad3f00874a1761619 WatchSource:0}: Error finding container e57f913a409dc3e0daf2de82c0e7368d82d2f6fe3ac24faad3f00874a1761619: Status 404 returned error can't find the container with id e57f913a409dc3e0daf2de82c0e7368d82d2f6fe3ac24faad3f00874a1761619
Sep 07 13:08:54.079080 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:54.079017    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8" path="/var/lib/kubelet/pods/7bf17dfd-2f05-4ab3-bffd-cf4485cfc1c8/volumes"
Sep 07 13:08:54.545169 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:54.545118    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d" event={"ID":"6d2c3c54-003d-4d49-8044-f27430dbdcad","Type":"ContainerStarted","Data":"0962d8897b8927d287040042fb11c1c713caa5668180b5cacbca70768bb38875"}
Sep 07 13:08:54.545601 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:54.545191    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d" event={"ID":"6d2c3c54-003d-4d49-8044-f27430dbdcad","Type":"ContainerStarted","Data":"e57f913a409dc3e0daf2de82c0e7368d82d2f6fe3ac24faad3f00874a1761619"}
Sep 07 13:08:54.567240 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:54.567189    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d" podStartSLOduration=1.567172792 podStartE2EDuration="1.567172792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:08:54.566031257 +0000 UTC m=+578.994329530" watchObservedRunningTime="2026-09-07 13:08:54.567172792 +0000 UTC m=+578.995471057"
Sep 07 13:08:55.550712 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:08:55.550685    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"
Sep 07 13:09:04.547768 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:04.547734    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"
Sep 07 13:09:06.558827 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:06.558794    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"
Sep 07 13:09:06.622098 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:06.622054    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"]
Sep 07 13:09:06.622290 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:06.622269    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp" podUID="1d39afa7-cdb9-458b-b530-c8f71a220d76" containerName="manager" containerID="cri-o://821271f44a440b89131e9d17374df9ec520e53f1f381da9c8069377646c9e4d8" gracePeriod=10
Sep 07 13:09:06.868044 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:06.868024    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"
Sep 07 13:09:07.023847 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.023814    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xttgd\" (UniqueName: \"kubernetes.io/projected/1d39afa7-cdb9-458b-b530-c8f71a220d76-kube-api-access-xttgd\") pod \"1d39afa7-cdb9-458b-b530-c8f71a220d76\" (UID: \"1d39afa7-cdb9-458b-b530-c8f71a220d76\") "
Sep 07 13:09:07.024035 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.023886    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d39afa7-cdb9-458b-b530-c8f71a220d76-extensions-socket-volume\") pod \"1d39afa7-cdb9-458b-b530-c8f71a220d76\" (UID: \"1d39afa7-cdb9-458b-b530-c8f71a220d76\") "
Sep 07 13:09:07.024329 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.024304    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d39afa7-cdb9-458b-b530-c8f71a220d76-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1d39afa7-cdb9-458b-b530-c8f71a220d76" (UID: "1d39afa7-cdb9-458b-b530-c8f71a220d76"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:09:07.025982 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.025962    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d39afa7-cdb9-458b-b530-c8f71a220d76-kube-api-access-xttgd" (OuterVolumeSpecName: "kube-api-access-xttgd") pod "1d39afa7-cdb9-458b-b530-c8f71a220d76" (UID: "1d39afa7-cdb9-458b-b530-c8f71a220d76"). InnerVolumeSpecName "kube-api-access-xttgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:09:07.125019 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.124922    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xttgd\" (UniqueName: \"kubernetes.io/projected/1d39afa7-cdb9-458b-b530-c8f71a220d76-kube-api-access-xttgd\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:09:07.125019 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.124952    2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d39afa7-cdb9-458b-b530-c8f71a220d76-extensions-socket-volume\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:09:07.615906 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.615869    2567 generic.go:356] "Generic (PLEG): container finished" podID="1d39afa7-cdb9-458b-b530-c8f71a220d76" containerID="821271f44a440b89131e9d17374df9ec520e53f1f381da9c8069377646c9e4d8" exitCode=0
Sep 07 13:09:07.616410 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.615927    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"
Sep 07 13:09:07.616410 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.615955    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp" event={"ID":"1d39afa7-cdb9-458b-b530-c8f71a220d76","Type":"ContainerDied","Data":"821271f44a440b89131e9d17374df9ec520e53f1f381da9c8069377646c9e4d8"}
Sep 07 13:09:07.616410 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.616083    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp" event={"ID":"1d39afa7-cdb9-458b-b530-c8f71a220d76","Type":"ContainerDied","Data":"51e19c9a8b33b5c6a01cdfa0b70db53f5f0789643b4306c3c4e8c894b5a55205"}
Sep 07 13:09:07.616410 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.616109    2567 scope.go:117] "RemoveContainer" containerID="821271f44a440b89131e9d17374df9ec520e53f1f381da9c8069377646c9e4d8"
Sep 07 13:09:07.625402 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.625185    2567 scope.go:117] "RemoveContainer" containerID="821271f44a440b89131e9d17374df9ec520e53f1f381da9c8069377646c9e4d8"
Sep 07 13:09:07.625484 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:09:07.625464    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821271f44a440b89131e9d17374df9ec520e53f1f381da9c8069377646c9e4d8\": container with ID starting with 821271f44a440b89131e9d17374df9ec520e53f1f381da9c8069377646c9e4d8 not found: ID does not exist" containerID="821271f44a440b89131e9d17374df9ec520e53f1f381da9c8069377646c9e4d8"
Sep 07 13:09:07.625525 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.625493    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821271f44a440b89131e9d17374df9ec520e53f1f381da9c8069377646c9e4d8"} err="failed to get container status \"821271f44a440b89131e9d17374df9ec520e53f1f381da9c8069377646c9e4d8\": rpc error: code = NotFound desc = could not find container \"821271f44a440b89131e9d17374df9ec520e53f1f381da9c8069377646c9e4d8\": container with ID starting with 821271f44a440b89131e9d17374df9ec520e53f1f381da9c8069377646c9e4d8 not found: ID does not exist"
Sep 07 13:09:07.636815 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.636790    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"]
Sep 07 13:09:07.646400 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:07.646376    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-f86dp"]
Sep 07 13:09:08.074597 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:08.074564    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d39afa7-cdb9-458b-b530-c8f71a220d76" path="/var/lib/kubelet/pods/1d39afa7-cdb9-458b-b530-c8f71a220d76/volumes"
Sep 07 13:09:23.202367 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.202325    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"]
Sep 07 13:09:23.203030 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.203003    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d39afa7-cdb9-458b-b530-c8f71a220d76" containerName="manager"
Sep 07 13:09:23.203152 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.203039    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d39afa7-cdb9-458b-b530-c8f71a220d76" containerName="manager"
Sep 07 13:09:23.203234 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.203218    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d39afa7-cdb9-458b-b530-c8f71a220d76" containerName="manager"
Sep 07 13:09:23.206717 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.206689    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.210522 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.210489    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8sflr\""
Sep 07 13:09:23.225389 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.225360    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"]
Sep 07 13:09:23.361354 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.361313    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a5b4a5-e033-43a7-bf17-a35a40587f00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.361533 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.361365    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.361533 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.361385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.361533 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.361462    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.361533 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.361502    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a5b4a5-e033-43a7-bf17-a35a40587f00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.361533 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.361524    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bhf8\" (UniqueName: \"kubernetes.io/projected/03a5b4a5-e033-43a7-bf17-a35a40587f00-kube-api-access-6bhf8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.361739 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.361654    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a5b4a5-e033-43a7-bf17-a35a40587f00-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.361739 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.361706    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.361815 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.361747    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.463163 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.463040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a5b4a5-e033-43a7-bf17-a35a40587f00-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.463163 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.463113    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.463394 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.463176    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.463394 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.463219    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a5b4a5-e033-43a7-bf17-a35a40587f00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.463394 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.463249    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.463394 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.463274    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.463394 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.463302    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.463394 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.463335    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a5b4a5-e033-43a7-bf17-a35a40587f00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.463394 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.463363    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bhf8\" (UniqueName: \"kubernetes.io/projected/03a5b4a5-e033-43a7-bf17-a35a40587f00-kube-api-access-6bhf8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.463750 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.463588    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.463750 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.463610    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.463923 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.463757    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.464026 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.464004    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.464326 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.464305    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a5b4a5-e033-43a7-bf17-a35a40587f00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.465669 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.465633    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a5b4a5-e033-43a7-bf17-a35a40587f00-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.465887 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.465867    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a5b4a5-e033-43a7-bf17-a35a40587f00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.477267 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.477235    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a5b4a5-e033-43a7-bf17-a35a40587f00-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.479170 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.479147    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bhf8\" (UniqueName: \"kubernetes.io/projected/03a5b4a5-e033-43a7-bf17-a35a40587f00-kube-api-access-6bhf8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6vglg\" (UID: \"03a5b4a5-e033-43a7-bf17-a35a40587f00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.517421 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.517388    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:23.806990 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.806962    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"]
Sep 07 13:09:23.807768 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:09:23.807742    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a5b4a5_e033_43a7_bf17_a35a40587f00.slice/crio-dec97bfc4d09a0aa6130e8cefe6a3c9cbef11b2aafe46af14660a5cfece39216 WatchSource:0}: Error finding container dec97bfc4d09a0aa6130e8cefe6a3c9cbef11b2aafe46af14660a5cfece39216: Status 404 returned error can't find the container with id dec97bfc4d09a0aa6130e8cefe6a3c9cbef11b2aafe46af14660a5cfece39216
Sep 07 13:09:23.809789 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.809761    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:09:23.809858 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.809826    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:09:23.809912 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:23.809855    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:09:24.710255 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:24.710205    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg" event={"ID":"03a5b4a5-e033-43a7-bf17-a35a40587f00","Type":"ContainerStarted","Data":"d8e77ad756f3584653559726d02320401ce5be937dc35b2b317d0c0b7ede5afd"}
Sep 07 13:09:24.710255 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:24.710271    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg" event={"ID":"03a5b4a5-e033-43a7-bf17-a35a40587f00","Type":"ContainerStarted","Data":"dec97bfc4d09a0aa6130e8cefe6a3c9cbef11b2aafe46af14660a5cfece39216"}
Sep 07 13:09:25.518590 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:25.518548    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:25.523964 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:25.523938    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:25.560163 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:25.560104    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg" podStartSLOduration=2.560089588 podStartE2EDuration="2.560089588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:09:24.738232951 +0000 UTC m=+609.166531224" watchObservedRunningTime="2026-09-07 13:09:25.560089588 +0000 UTC m=+609.988387867"
Sep 07 13:09:25.715936 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:25.715897    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:25.716926 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:25.716905    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:09:26.118969 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.118937    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"]
Sep 07 13:09:26.121165 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.121145    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"
Sep 07 13:09:26.123502 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.123476    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 07 13:09:26.123910 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.123892    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hwcc4\""
Sep 07 13:09:26.152984 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.152955    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"]
Sep 07 13:09:26.226974 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.226938    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"]
Sep 07 13:09:26.289185 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.289151    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77252049-c86f-4af6-8242-b6d81fd30cd0-config-file\") pod \"limitador-limitador-859c4d96d6-dnzhc\" (UID: \"77252049-c86f-4af6-8242-b6d81fd30cd0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"
Sep 07 13:09:26.289185 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.289190    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzpk\" (UniqueName: \"kubernetes.io/projected/77252049-c86f-4af6-8242-b6d81fd30cd0-kube-api-access-7gzpk\") pod \"limitador-limitador-859c4d96d6-dnzhc\" (UID: \"77252049-c86f-4af6-8242-b6d81fd30cd0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"
Sep 07 13:09:26.390674 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.390570    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77252049-c86f-4af6-8242-b6d81fd30cd0-config-file\") pod \"limitador-limitador-859c4d96d6-dnzhc\" (UID: \"77252049-c86f-4af6-8242-b6d81fd30cd0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"
Sep 07 13:09:26.390674 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.390622    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzpk\" (UniqueName: \"kubernetes.io/projected/77252049-c86f-4af6-8242-b6d81fd30cd0-kube-api-access-7gzpk\") pod \"limitador-limitador-859c4d96d6-dnzhc\" (UID: \"77252049-c86f-4af6-8242-b6d81fd30cd0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"
Sep 07 13:09:26.391305 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.391279    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77252049-c86f-4af6-8242-b6d81fd30cd0-config-file\") pod \"limitador-limitador-859c4d96d6-dnzhc\" (UID: \"77252049-c86f-4af6-8242-b6d81fd30cd0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"
Sep 07 13:09:26.409747 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.409711    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzpk\" (UniqueName: \"kubernetes.io/projected/77252049-c86f-4af6-8242-b6d81fd30cd0-kube-api-access-7gzpk\") pod \"limitador-limitador-859c4d96d6-dnzhc\" (UID: \"77252049-c86f-4af6-8242-b6d81fd30cd0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"
Sep 07 13:09:26.432283 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.432252    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"
Sep 07 13:09:26.715194 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:26.715170    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"]
Sep 07 13:09:26.717799 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:09:26.717773    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77252049_c86f_4af6_8242_b6d81fd30cd0.slice/crio-67141181c3ce0fcfe7bd8f62e043e75e31e014b469b9e677a60b2f4df6d5ca60 WatchSource:0}: Error finding container 67141181c3ce0fcfe7bd8f62e043e75e31e014b469b9e677a60b2f4df6d5ca60: Status 404 returned error can't find the container with id 67141181c3ce0fcfe7bd8f62e043e75e31e014b469b9e677a60b2f4df6d5ca60
Sep 07 13:09:27.729556 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:27.729521    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc" event={"ID":"77252049-c86f-4af6-8242-b6d81fd30cd0","Type":"ContainerStarted","Data":"67141181c3ce0fcfe7bd8f62e043e75e31e014b469b9e677a60b2f4df6d5ca60"}
Sep 07 13:09:28.736531 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:28.736488    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc" event={"ID":"77252049-c86f-4af6-8242-b6d81fd30cd0","Type":"ContainerStarted","Data":"c0d486322112cbe633d87c5f469156c1361ed2e45ddadfacd18bbeef5122efb3"}
Sep 07 13:09:28.737012 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:28.736559    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"
Sep 07 13:09:28.761708 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:28.761655    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc" podStartSLOduration=1.541514464 podStartE2EDuration="2.761635647s" totalImagesPullingTime="1.220121183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:09:28.760213735 +0000 UTC m=+613.188512020" watchObservedRunningTime="2026-09-07 13:09:28.761635647 +0000 UTC m=+613.189933923"
Sep 07 13:09:39.743976 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:39.743887    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"
Sep 07 13:09:40.016332 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.016244    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"]
Sep 07 13:09:40.016487 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.016447    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc" podUID="77252049-c86f-4af6-8242-b6d81fd30cd0" containerName="limitador" containerID="cri-o://c0d486322112cbe633d87c5f469156c1361ed2e45ddadfacd18bbeef5122efb3" gracePeriod=30
Sep 07 13:09:40.570279 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.570253    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"
Sep 07 13:09:40.604644 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.604567    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gzpk\" (UniqueName: \"kubernetes.io/projected/77252049-c86f-4af6-8242-b6d81fd30cd0-kube-api-access-7gzpk\") pod \"77252049-c86f-4af6-8242-b6d81fd30cd0\" (UID: \"77252049-c86f-4af6-8242-b6d81fd30cd0\") "
Sep 07 13:09:40.604644 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.604611    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77252049-c86f-4af6-8242-b6d81fd30cd0-config-file\") pod \"77252049-c86f-4af6-8242-b6d81fd30cd0\" (UID: \"77252049-c86f-4af6-8242-b6d81fd30cd0\") "
Sep 07 13:09:40.605003 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.604981    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77252049-c86f-4af6-8242-b6d81fd30cd0-config-file" (OuterVolumeSpecName: "config-file") pod "77252049-c86f-4af6-8242-b6d81fd30cd0" (UID: "77252049-c86f-4af6-8242-b6d81fd30cd0"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:09:40.606877 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.606849    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77252049-c86f-4af6-8242-b6d81fd30cd0-kube-api-access-7gzpk" (OuterVolumeSpecName: "kube-api-access-7gzpk") pod "77252049-c86f-4af6-8242-b6d81fd30cd0" (UID: "77252049-c86f-4af6-8242-b6d81fd30cd0"). InnerVolumeSpecName "kube-api-access-7gzpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:09:40.705442 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.705399    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gzpk\" (UniqueName: \"kubernetes.io/projected/77252049-c86f-4af6-8242-b6d81fd30cd0-kube-api-access-7gzpk\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:09:40.705442 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.705437    2567 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77252049-c86f-4af6-8242-b6d81fd30cd0-config-file\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:09:40.796235 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.796197    2567 generic.go:356] "Generic (PLEG): container finished" podID="77252049-c86f-4af6-8242-b6d81fd30cd0" containerID="c0d486322112cbe633d87c5f469156c1361ed2e45ddadfacd18bbeef5122efb3" exitCode=0
Sep 07 13:09:40.796701 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.796245    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"
Sep 07 13:09:40.796701 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.796277    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc" event={"ID":"77252049-c86f-4af6-8242-b6d81fd30cd0","Type":"ContainerDied","Data":"c0d486322112cbe633d87c5f469156c1361ed2e45ddadfacd18bbeef5122efb3"}
Sep 07 13:09:40.796701 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.796331    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-dnzhc" event={"ID":"77252049-c86f-4af6-8242-b6d81fd30cd0","Type":"ContainerDied","Data":"67141181c3ce0fcfe7bd8f62e043e75e31e014b469b9e677a60b2f4df6d5ca60"}
Sep 07 13:09:40.796701 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.796355    2567 scope.go:117] "RemoveContainer" containerID="c0d486322112cbe633d87c5f469156c1361ed2e45ddadfacd18bbeef5122efb3"
Sep 07 13:09:40.805287 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.805270    2567 scope.go:117] "RemoveContainer" containerID="c0d486322112cbe633d87c5f469156c1361ed2e45ddadfacd18bbeef5122efb3"
Sep 07 13:09:40.805585 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:09:40.805565    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d486322112cbe633d87c5f469156c1361ed2e45ddadfacd18bbeef5122efb3\": container with ID starting with c0d486322112cbe633d87c5f469156c1361ed2e45ddadfacd18bbeef5122efb3 not found: ID does not exist" containerID="c0d486322112cbe633d87c5f469156c1361ed2e45ddadfacd18bbeef5122efb3"
Sep 07 13:09:40.805672 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.805592    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d486322112cbe633d87c5f469156c1361ed2e45ddadfacd18bbeef5122efb3"} err="failed to get container status \"c0d486322112cbe633d87c5f469156c1361ed2e45ddadfacd18bbeef5122efb3\": rpc error: code = NotFound desc = could not find container \"c0d486322112cbe633d87c5f469156c1361ed2e45ddadfacd18bbeef5122efb3\": container with ID starting with c0d486322112cbe633d87c5f469156c1361ed2e45ddadfacd18bbeef5122efb3 not found: ID does not exist"
Sep 07 13:09:40.819071 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.819041    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"]
Sep 07 13:09:40.824333 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:40.824310    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-dnzhc"]
Sep 07 13:09:42.079073 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.079020    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77252049-c86f-4af6-8242-b6d81fd30cd0" path="/var/lib/kubelet/pods/77252049-c86f-4af6-8242-b6d81fd30cd0/volumes"
Sep 07 13:09:42.687766 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.687732    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-67b6d"]
Sep 07 13:09:42.688090 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.688077    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77252049-c86f-4af6-8242-b6d81fd30cd0" containerName="limitador"
Sep 07 13:09:42.688140 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.688091    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="77252049-c86f-4af6-8242-b6d81fd30cd0" containerName="limitador"
Sep 07 13:09:42.688175 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.688141    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="77252049-c86f-4af6-8242-b6d81fd30cd0" containerName="limitador"
Sep 07 13:09:42.692305 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.692283    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-67b6d"
Sep 07 13:09:42.698729 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.698700    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 07 13:09:42.703527 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.703508    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 13:09:42.703612 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.703551    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:09:42.712882 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.712857    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vfrcg\""
Sep 07 13:09:42.720520 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.720495    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65gdk\" (UniqueName: \"kubernetes.io/projected/f751fe86-77a6-4fce-ab07-1372f66c6593-kube-api-access-65gdk\") pod \"postgres-69fb4594cf-67b6d\" (UID: \"f751fe86-77a6-4fce-ab07-1372f66c6593\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-67b6d"
Sep 07 13:09:42.720611 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.720576    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f751fe86-77a6-4fce-ab07-1372f66c6593-data\") pod \"postgres-69fb4594cf-67b6d\" (UID: \"f751fe86-77a6-4fce-ab07-1372f66c6593\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-67b6d"
Sep 07 13:09:42.789242 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.789198    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-67b6d"]
Sep 07 13:09:42.821682 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.821641    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f751fe86-77a6-4fce-ab07-1372f66c6593-data\") pod \"postgres-69fb4594cf-67b6d\" (UID: \"f751fe86-77a6-4fce-ab07-1372f66c6593\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-67b6d"
Sep 07 13:09:42.821865 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.821730    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65gdk\" (UniqueName: \"kubernetes.io/projected/f751fe86-77a6-4fce-ab07-1372f66c6593-kube-api-access-65gdk\") pod \"postgres-69fb4594cf-67b6d\" (UID: \"f751fe86-77a6-4fce-ab07-1372f66c6593\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-67b6d"
Sep 07 13:09:42.822540 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.822500    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f751fe86-77a6-4fce-ab07-1372f66c6593-data\") pod \"postgres-69fb4594cf-67b6d\" (UID: \"f751fe86-77a6-4fce-ab07-1372f66c6593\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-67b6d"
Sep 07 13:09:42.846577 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:42.846542    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65gdk\" (UniqueName: \"kubernetes.io/projected/f751fe86-77a6-4fce-ab07-1372f66c6593-kube-api-access-65gdk\") pod \"postgres-69fb4594cf-67b6d\" (UID: \"f751fe86-77a6-4fce-ab07-1372f66c6593\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-67b6d"
Sep 07 13:09:43.002278 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:43.002173    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-67b6d"
Sep 07 13:09:43.278223 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:43.278166    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-67b6d"]
Sep 07 13:09:43.280754 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:09:43.280718    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf751fe86_77a6_4fce_ab07_1372f66c6593.slice/crio-75a01b9fb8c9b94f3495a813b6b034a39a368be7c98b251df7fef039cddd837e WatchSource:0}: Error finding container 75a01b9fb8c9b94f3495a813b6b034a39a368be7c98b251df7fef039cddd837e: Status 404 returned error can't find the container with id 75a01b9fb8c9b94f3495a813b6b034a39a368be7c98b251df7fef039cddd837e
Sep 07 13:09:43.813819 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:43.813785    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-67b6d" event={"ID":"f751fe86-77a6-4fce-ab07-1372f66c6593","Type":"ContainerStarted","Data":"75a01b9fb8c9b94f3495a813b6b034a39a368be7c98b251df7fef039cddd837e"}
Sep 07 13:09:48.844516 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:48.844476    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-67b6d" event={"ID":"f751fe86-77a6-4fce-ab07-1372f66c6593","Type":"ContainerStarted","Data":"4efac487545e963fab50765d8e9ab2954f18dec7348c4ba9f441f0b1f724da7f"}
Sep 07 13:09:48.844925 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:48.844602    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-67b6d"
Sep 07 13:09:48.865732 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:48.865678    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-67b6d" podStartSLOduration=2.077352501 podStartE2EDuration="6.865661151s" totalImagesPullingTime="4.78830865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:09:48.863773916 +0000 UTC m=+633.292072190" watchObservedRunningTime="2026-09-07 13:09:48.865661151 +0000 UTC m=+633.293959425"
Sep 07 13:09:54.878922 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:09:54.878891    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-67b6d"
Sep 07 13:10:03.373861 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.373824    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fbc95c744-rd55g"]
Sep 07 13:10:03.391602 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.391577    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:03.394911 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.394886    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 07 13:10:03.394911 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.394902    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 07 13:10:03.395133 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.394902    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 07 13:10:03.395133 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.394897    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-97pz6\""
Sep 07 13:10:03.408419 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.408394    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fbc95c744-rd55g"]
Sep 07 13:10:03.501425 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.501391    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhdl\" (UniqueName: \"kubernetes.io/projected/27d12791-97ff-4d65-9cf6-716ce2463a4f-kube-api-access-vdhdl\") pod \"maas-controller-fbc95c744-rd55g\" (UID: \"27d12791-97ff-4d65-9cf6-716ce2463a4f\") " pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:03.501642 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.501444    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27d12791-97ff-4d65-9cf6-716ce2463a4f-webhook-cert\") pod \"maas-controller-fbc95c744-rd55g\" (UID: \"27d12791-97ff-4d65-9cf6-716ce2463a4f\") " pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:03.501642 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.501503    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d12791-97ff-4d65-9cf6-716ce2463a4f-metrics-certs\") pod \"maas-controller-fbc95c744-rd55g\" (UID: \"27d12791-97ff-4d65-9cf6-716ce2463a4f\") " pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:03.602774 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.602735    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27d12791-97ff-4d65-9cf6-716ce2463a4f-webhook-cert\") pod \"maas-controller-fbc95c744-rd55g\" (UID: \"27d12791-97ff-4d65-9cf6-716ce2463a4f\") " pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:03.602965 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.602786    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d12791-97ff-4d65-9cf6-716ce2463a4f-metrics-certs\") pod \"maas-controller-fbc95c744-rd55g\" (UID: \"27d12791-97ff-4d65-9cf6-716ce2463a4f\") " pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:03.602965 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.602847    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhdl\" (UniqueName: \"kubernetes.io/projected/27d12791-97ff-4d65-9cf6-716ce2463a4f-kube-api-access-vdhdl\") pod \"maas-controller-fbc95c744-rd55g\" (UID: \"27d12791-97ff-4d65-9cf6-716ce2463a4f\") " pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:03.605351 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.605326    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27d12791-97ff-4d65-9cf6-716ce2463a4f-webhook-cert\") pod \"maas-controller-fbc95c744-rd55g\" (UID: \"27d12791-97ff-4d65-9cf6-716ce2463a4f\") " pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:03.605449 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.605375    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d12791-97ff-4d65-9cf6-716ce2463a4f-metrics-certs\") pod \"maas-controller-fbc95c744-rd55g\" (UID: \"27d12791-97ff-4d65-9cf6-716ce2463a4f\") " pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:03.619300 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.619277    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhdl\" (UniqueName: \"kubernetes.io/projected/27d12791-97ff-4d65-9cf6-716ce2463a4f-kube-api-access-vdhdl\") pod \"maas-controller-fbc95c744-rd55g\" (UID: \"27d12791-97ff-4d65-9cf6-716ce2463a4f\") " pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:03.692661 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.692583    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66fb99b89f-gtwxv"]
Sep 07 13:10:03.695081 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.695048    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:10:03.702158 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.702135    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:03.711945 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.711916    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66fb99b89f-gtwxv"]
Sep 07 13:10:03.804439 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.804408    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4ab6f84-8ff8-4270-8b8b-868329a20cba-metrics-certs\") pod \"maas-controller-66fb99b89f-gtwxv\" (UID: \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\") " pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:10:03.804591 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.804451    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4ab6f84-8ff8-4270-8b8b-868329a20cba-webhook-cert\") pod \"maas-controller-66fb99b89f-gtwxv\" (UID: \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\") " pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:10:03.804591 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.804491    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7dwb\" (UniqueName: \"kubernetes.io/projected/d4ab6f84-8ff8-4270-8b8b-868329a20cba-kube-api-access-b7dwb\") pod \"maas-controller-66fb99b89f-gtwxv\" (UID: \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\") " pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:10:03.905351 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.905313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4ab6f84-8ff8-4270-8b8b-868329a20cba-metrics-certs\") pod \"maas-controller-66fb99b89f-gtwxv\" (UID: \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\") " pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:10:03.905351 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.905358    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4ab6f84-8ff8-4270-8b8b-868329a20cba-webhook-cert\") pod \"maas-controller-66fb99b89f-gtwxv\" (UID: \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\") " pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:10:03.905553 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.905374    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7dwb\" (UniqueName: \"kubernetes.io/projected/d4ab6f84-8ff8-4270-8b8b-868329a20cba-kube-api-access-b7dwb\") pod \"maas-controller-66fb99b89f-gtwxv\" (UID: \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\") " pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:10:03.907810 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.907777    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4ab6f84-8ff8-4270-8b8b-868329a20cba-metrics-certs\") pod \"maas-controller-66fb99b89f-gtwxv\" (UID: \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\") " pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:10:03.907936 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.907823    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4ab6f84-8ff8-4270-8b8b-868329a20cba-webhook-cert\") pod \"maas-controller-66fb99b89f-gtwxv\" (UID: \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\") " pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:10:03.913960 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.913937    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7dwb\" (UniqueName: \"kubernetes.io/projected/d4ab6f84-8ff8-4270-8b8b-868329a20cba-kube-api-access-b7dwb\") pod \"maas-controller-66fb99b89f-gtwxv\" (UID: \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\") " pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:10:03.991309 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:03.991282    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fbc95c744-rd55g"]
Sep 07 13:10:03.993993 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:10:03.993968    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d12791_97ff_4d65_9cf6_716ce2463a4f.slice/crio-ca9404e855d6dc1a61f8fa737d9e64c9ba2116d74210a86fe8945a823a45206f WatchSource:0}: Error finding container ca9404e855d6dc1a61f8fa737d9e64c9ba2116d74210a86fe8945a823a45206f: Status 404 returned error can't find the container with id ca9404e855d6dc1a61f8fa737d9e64c9ba2116d74210a86fe8945a823a45206f
Sep 07 13:10:04.006128 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:04.006105    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:10:04.297878 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:04.297723    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66fb99b89f-gtwxv"]
Sep 07 13:10:04.300652 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:10:04.300628    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ab6f84_8ff8_4270_8b8b_868329a20cba.slice/crio-931f09e6c25ec8fcbc592bec162b6ebe303092326fa8d5104b9376d63dfe9164 WatchSource:0}: Error finding container 931f09e6c25ec8fcbc592bec162b6ebe303092326fa8d5104b9376d63dfe9164: Status 404 returned error can't find the container with id 931f09e6c25ec8fcbc592bec162b6ebe303092326fa8d5104b9376d63dfe9164
Sep 07 13:10:04.932406 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:04.932363    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fbc95c744-rd55g" event={"ID":"27d12791-97ff-4d65-9cf6-716ce2463a4f","Type":"ContainerStarted","Data":"ca9404e855d6dc1a61f8fa737d9e64c9ba2116d74210a86fe8945a823a45206f"}
Sep 07 13:10:04.935292 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:04.935181    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fb99b89f-gtwxv" event={"ID":"d4ab6f84-8ff8-4270-8b8b-868329a20cba","Type":"ContainerStarted","Data":"931f09e6c25ec8fcbc592bec162b6ebe303092326fa8d5104b9376d63dfe9164"}
Sep 07 13:10:06.947610 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:06.947498    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fbc95c744-rd55g" event={"ID":"27d12791-97ff-4d65-9cf6-716ce2463a4f","Type":"ContainerStarted","Data":"f51bdd7244b640755ea48b85e564c52790ea19632ef5105e40a075c3511f3c56"}
Sep 07 13:10:06.947610 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:06.947610    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:06.948948 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:06.948922    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fb99b89f-gtwxv" event={"ID":"d4ab6f84-8ff8-4270-8b8b-868329a20cba","Type":"ContainerStarted","Data":"27ae9f62791ade57413e213f0844dadc68060f19eb9b4c500077761363efd201"}
Sep 07 13:10:06.949109 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:06.949091    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:10:06.984555 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:06.984496    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-fbc95c744-rd55g" podStartSLOduration=1.343147424 podStartE2EDuration="3.984480004s" totalImagesPullingTime="2.64133258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:10:06.982034002 +0000 UTC m=+651.410332297" watchObservedRunningTime="2026-09-07 13:10:06.984480004 +0000 UTC m=+651.412778276"
Sep 07 13:10:07.011637 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:07.011579    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66fb99b89f-gtwxv" podStartSLOduration=1.6701619559999998 podStartE2EDuration="4.01156193s" totalImagesPullingTime="2.341399974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:10:07.010197521 +0000 UTC m=+651.438495828" watchObservedRunningTime="2026-09-07 13:10:07.01156193 +0000 UTC m=+651.439860285"
Sep 07 13:10:17.955652 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:17.955619    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:10:17.956110 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:17.955688    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:18.055947 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:18.055916    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fbc95c744-rd55g"]
Sep 07 13:10:18.056170 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:18.056145    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-fbc95c744-rd55g" podUID="27d12791-97ff-4d65-9cf6-716ce2463a4f" containerName="manager" containerID="cri-o://f51bdd7244b640755ea48b85e564c52790ea19632ef5105e40a075c3511f3c56" gracePeriod=30
Sep 07 13:10:18.302945 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:18.302917    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:18.432363 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:18.432321    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d12791-97ff-4d65-9cf6-716ce2463a4f-metrics-certs\") pod \"27d12791-97ff-4d65-9cf6-716ce2463a4f\" (UID: \"27d12791-97ff-4d65-9cf6-716ce2463a4f\") "
Sep 07 13:10:18.432558 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:18.432388    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27d12791-97ff-4d65-9cf6-716ce2463a4f-webhook-cert\") pod \"27d12791-97ff-4d65-9cf6-716ce2463a4f\" (UID: \"27d12791-97ff-4d65-9cf6-716ce2463a4f\") "
Sep 07 13:10:18.432558 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:18.432489    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdhdl\" (UniqueName: \"kubernetes.io/projected/27d12791-97ff-4d65-9cf6-716ce2463a4f-kube-api-access-vdhdl\") pod \"27d12791-97ff-4d65-9cf6-716ce2463a4f\" (UID: \"27d12791-97ff-4d65-9cf6-716ce2463a4f\") "
Sep 07 13:10:18.434628 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:18.434601    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d12791-97ff-4d65-9cf6-716ce2463a4f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "27d12791-97ff-4d65-9cf6-716ce2463a4f" (UID: "27d12791-97ff-4d65-9cf6-716ce2463a4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:10:18.434906 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:18.434879    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d12791-97ff-4d65-9cf6-716ce2463a4f-kube-api-access-vdhdl" (OuterVolumeSpecName: "kube-api-access-vdhdl") pod "27d12791-97ff-4d65-9cf6-716ce2463a4f" (UID: "27d12791-97ff-4d65-9cf6-716ce2463a4f"). InnerVolumeSpecName "kube-api-access-vdhdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:10:18.434906 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:18.434885    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d12791-97ff-4d65-9cf6-716ce2463a4f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "27d12791-97ff-4d65-9cf6-716ce2463a4f" (UID: "27d12791-97ff-4d65-9cf6-716ce2463a4f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:10:18.533710 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:18.533682    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d12791-97ff-4d65-9cf6-716ce2463a4f-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:10:18.533710 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:18.533708    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27d12791-97ff-4d65-9cf6-716ce2463a4f-webhook-cert\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:10:18.533889 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:18.533717    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdhdl\" (UniqueName: \"kubernetes.io/projected/27d12791-97ff-4d65-9cf6-716ce2463a4f-kube-api-access-vdhdl\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:10:19.012748 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:19.012648    2567 generic.go:356] "Generic (PLEG): container finished" podID="27d12791-97ff-4d65-9cf6-716ce2463a4f" containerID="f51bdd7244b640755ea48b85e564c52790ea19632ef5105e40a075c3511f3c56" exitCode=0
Sep 07 13:10:19.012748 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:19.012708    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fbc95c744-rd55g"
Sep 07 13:10:19.012748 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:19.012718    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fbc95c744-rd55g" event={"ID":"27d12791-97ff-4d65-9cf6-716ce2463a4f","Type":"ContainerDied","Data":"f51bdd7244b640755ea48b85e564c52790ea19632ef5105e40a075c3511f3c56"}
Sep 07 13:10:19.013347 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:19.012775    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fbc95c744-rd55g" event={"ID":"27d12791-97ff-4d65-9cf6-716ce2463a4f","Type":"ContainerDied","Data":"ca9404e855d6dc1a61f8fa737d9e64c9ba2116d74210a86fe8945a823a45206f"}
Sep 07 13:10:19.013347 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:19.012797    2567 scope.go:117] "RemoveContainer" containerID="f51bdd7244b640755ea48b85e564c52790ea19632ef5105e40a075c3511f3c56"
Sep 07 13:10:19.024393 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:19.024368    2567 scope.go:117] "RemoveContainer" containerID="f51bdd7244b640755ea48b85e564c52790ea19632ef5105e40a075c3511f3c56"
Sep 07 13:10:19.024689 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:10:19.024662    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51bdd7244b640755ea48b85e564c52790ea19632ef5105e40a075c3511f3c56\": container with ID starting with f51bdd7244b640755ea48b85e564c52790ea19632ef5105e40a075c3511f3c56 not found: ID does not exist" containerID="f51bdd7244b640755ea48b85e564c52790ea19632ef5105e40a075c3511f3c56"
Sep 07 13:10:19.024803 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:19.024695    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51bdd7244b640755ea48b85e564c52790ea19632ef5105e40a075c3511f3c56"} err="failed to get container status \"f51bdd7244b640755ea48b85e564c52790ea19632ef5105e40a075c3511f3c56\": rpc error: code = NotFound desc = could not find container \"f51bdd7244b640755ea48b85e564c52790ea19632ef5105e40a075c3511f3c56\": container with ID starting with f51bdd7244b640755ea48b85e564c52790ea19632ef5105e40a075c3511f3c56 not found: ID does not exist"
Sep 07 13:10:19.041868 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:19.041835    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fbc95c744-rd55g"]
Sep 07 13:10:19.047075 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:19.047037    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-fbc95c744-rd55g"]
Sep 07 13:10:20.074508 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:20.074470    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27d12791-97ff-4d65-9cf6-716ce2463a4f" path="/var/lib/kubelet/pods/27d12791-97ff-4d65-9cf6-716ce2463a4f/volumes"
Sep 07 13:10:41.461987 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.461946    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"]
Sep 07 13:10:41.462466 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.462285    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d12791-97ff-4d65-9cf6-716ce2463a4f" containerName="manager"
Sep 07 13:10:41.462466 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.462298    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d12791-97ff-4d65-9cf6-716ce2463a4f" containerName="manager"
Sep 07 13:10:41.462466 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.462346    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d12791-97ff-4d65-9cf6-716ce2463a4f" containerName="manager"
Sep 07 13:10:41.465022 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.464986    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:41.467951 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.467908    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 13:10:41.468209 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.467989    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fswx9\""
Sep 07 13:10:41.468209 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.468035    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 07 13:10:41.488461 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.488427    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"]
Sep 07 13:10:41.511290 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.511262    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-maas-api-tls\") pod \"maas-api-849b74d9cb-2s4lr\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") " pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:41.511452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.511299    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-openshift-service-ca\") pod \"maas-api-849b74d9cb-2s4lr\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") " pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:41.511452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.511320    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-metrics-certs\") pod \"maas-api-849b74d9cb-2s4lr\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") " pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:41.511452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.511410    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4xg\" (UniqueName: \"kubernetes.io/projected/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-kube-api-access-tn4xg\") pod \"maas-api-849b74d9cb-2s4lr\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") " pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:41.612745 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.612699    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-openshift-service-ca\") pod \"maas-api-849b74d9cb-2s4lr\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") " pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:41.612745 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.612746    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-metrics-certs\") pod \"maas-api-849b74d9cb-2s4lr\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") " pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:41.613001 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.612807    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4xg\" (UniqueName: \"kubernetes.io/projected/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-kube-api-access-tn4xg\") pod \"maas-api-849b74d9cb-2s4lr\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") " pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:41.613001 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.612967    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-maas-api-tls\") pod \"maas-api-849b74d9cb-2s4lr\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") " pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:41.613435 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.613414    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-openshift-service-ca\") pod \"maas-api-849b74d9cb-2s4lr\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") " pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:41.615321 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.615299    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-metrics-certs\") pod \"maas-api-849b74d9cb-2s4lr\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") " pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:41.615412 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.615329    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-maas-api-tls\") pod \"maas-api-849b74d9cb-2s4lr\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") " pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:41.623394 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.623366    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4xg\" (UniqueName: \"kubernetes.io/projected/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-kube-api-access-tn4xg\") pod \"maas-api-849b74d9cb-2s4lr\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") " pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:41.777552 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:41.777515    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:42.487904 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:42.487876    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"]
Sep 07 13:10:42.490160 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:10:42.490133    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b40a9f5_ebe9_4883_8a6e_1cf9c1420d98.slice/crio-3c00534177303a17a9edf33b31afc3edcb68def9d7d0b9b8a4e2c1b8fac7c235 WatchSource:0}: Error finding container 3c00534177303a17a9edf33b31afc3edcb68def9d7d0b9b8a4e2c1b8fac7c235: Status 404 returned error can't find the container with id 3c00534177303a17a9edf33b31afc3edcb68def9d7d0b9b8a4e2c1b8fac7c235
Sep 07 13:10:42.491597 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:42.491578    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 13:10:42.938701 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:42.938667    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lkvfr"]
Sep 07 13:10:42.942310 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:42.942293    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-lkvfr"
Sep 07 13:10:42.947280 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:42.947258    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 07 13:10:42.956491 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:42.956471    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cxmkd\""
Sep 07 13:10:43.023360 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:43.023323    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8cf259-4a5d-4e00-9f43-7bee665cbd7e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lkvfr\" (UID: \"0f8cf259-4a5d-4e00-9f43-7bee665cbd7e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lkvfr"
Sep 07 13:10:43.023605 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:43.023585    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2pfw\" (UniqueName: \"kubernetes.io/projected/0f8cf259-4a5d-4e00-9f43-7bee665cbd7e-kube-api-access-l2pfw\") pod \"payload-processing-5b54b7fbb7-lkvfr\" (UID: \"0f8cf259-4a5d-4e00-9f43-7bee665cbd7e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lkvfr"
Sep 07 13:10:43.027328 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:43.027281    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lkvfr"]
Sep 07 13:10:43.124685 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:43.124647    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8cf259-4a5d-4e00-9f43-7bee665cbd7e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lkvfr\" (UID: \"0f8cf259-4a5d-4e00-9f43-7bee665cbd7e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lkvfr"
Sep 07 13:10:43.124862 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:43.124702    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2pfw\" (UniqueName: \"kubernetes.io/projected/0f8cf259-4a5d-4e00-9f43-7bee665cbd7e-kube-api-access-l2pfw\") pod \"payload-processing-5b54b7fbb7-lkvfr\" (UID: \"0f8cf259-4a5d-4e00-9f43-7bee665cbd7e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lkvfr"
Sep 07 13:10:43.125556 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:43.125524    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8cf259-4a5d-4e00-9f43-7bee665cbd7e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lkvfr\" (UID: \"0f8cf259-4a5d-4e00-9f43-7bee665cbd7e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lkvfr"
Sep 07 13:10:43.138729 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:43.138679    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr" event={"ID":"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98","Type":"ContainerStarted","Data":"3c00534177303a17a9edf33b31afc3edcb68def9d7d0b9b8a4e2c1b8fac7c235"}
Sep 07 13:10:43.152620 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:43.152579    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2pfw\" (UniqueName: \"kubernetes.io/projected/0f8cf259-4a5d-4e00-9f43-7bee665cbd7e-kube-api-access-l2pfw\") pod \"payload-processing-5b54b7fbb7-lkvfr\" (UID: \"0f8cf259-4a5d-4e00-9f43-7bee665cbd7e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lkvfr"
Sep 07 13:10:43.252571 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:43.252484    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-lkvfr"
Sep 07 13:10:43.548911 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:43.548887    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lkvfr"]
Sep 07 13:10:43.551318 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:10:43.551278    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8cf259_4a5d_4e00_9f43_7bee665cbd7e.slice/crio-034ac767457a0b81f640ee121a720269753b64603670e22abaf223a5b85323e4 WatchSource:0}: Error finding container 034ac767457a0b81f640ee121a720269753b64603670e22abaf223a5b85323e4: Status 404 returned error can't find the container with id 034ac767457a0b81f640ee121a720269753b64603670e22abaf223a5b85323e4
Sep 07 13:10:44.144463 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:44.144430    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-lkvfr" event={"ID":"0f8cf259-4a5d-4e00-9f43-7bee665cbd7e","Type":"ContainerStarted","Data":"034ac767457a0b81f640ee121a720269753b64603670e22abaf223a5b85323e4"}
Sep 07 13:10:45.153430 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:45.153389    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr" event={"ID":"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98","Type":"ContainerStarted","Data":"65c7053dae7abd4d98db91e3794eff3a52074590e5bcfe374d45bd98c747c761"}
Sep 07 13:10:45.153889 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:45.153459    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:46.161037 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:46.160992    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-lkvfr" event={"ID":"0f8cf259-4a5d-4e00-9f43-7bee665cbd7e","Type":"ContainerStarted","Data":"54ee9495671dfab8a1307f895c6e6856d6c2b5818a76572055c68e03ba4d2eeb"}
Sep 07 13:10:46.161545 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:46.161151    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-lkvfr"
Sep 07 13:10:46.184539 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:46.181291    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr" podStartSLOduration=3.611103972 podStartE2EDuration="5.181270206s" totalImagesPullingTime="1.570166234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:10:45.20001612 +0000 UTC m=+689.628314394" watchObservedRunningTime="2026-09-07 13:10:46.181270206 +0000 UTC m=+690.609568480"
Sep 07 13:10:51.167226 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:51.167193    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:10:51.189464 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:51.189401    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-lkvfr" podStartSLOduration=7.195987498 podStartE2EDuration="9.189379743s" totalImagesPullingTime="1.993392245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:10:46.184890131 +0000 UTC m=+690.613188431" watchObservedRunningTime="2026-09-07 13:10:51.189379743 +0000 UTC m=+695.617678032"
Sep 07 13:10:53.201567 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:53.201528    2567 generic.go:356] "Generic (PLEG): container finished" podID="03a5b4a5-e033-43a7-bf17-a35a40587f00" containerID="d8e77ad756f3584653559726d02320401ce5be937dc35b2b317d0c0b7ede5afd" exitCode=0
Sep 07 13:10:53.201567 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:53.201567    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg" event={"ID":"03a5b4a5-e033-43a7-bf17-a35a40587f00","Type":"ContainerDied","Data":"d8e77ad756f3584653559726d02320401ce5be937dc35b2b317d0c0b7ede5afd"}
Sep 07 13:10:53.202018 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:53.201885    2567 scope.go:117] "RemoveContainer" containerID="d8e77ad756f3584653559726d02320401ce5be937dc35b2b317d0c0b7ede5afd"
Sep 07 13:10:53.202900 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:53.202798    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:10:53.202900 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:53.202864    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:10:53.202900 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:53.202898    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:10:54.208456 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:54.208422    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg" event={"ID":"03a5b4a5-e033-43a7-bf17-a35a40587f00","Type":"ContainerStarted","Data":"c83c822bd3fee1e8d2ee6e6f36f5ebc337bede0074ab20e822f3d2907fe2fcec"}
Sep 07 13:10:54.518068 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:54.518034    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:10:54.519171 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:54.519134    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6vglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 07 13:10:54.519284 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:54.519195    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg" podUID="03a5b4a5-e033-43a7-bf17-a35a40587f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 07 13:10:55.518006 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:55.517963    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6vglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 07 13:10:55.518427 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:55.518026    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg" podUID="03a5b4a5-e033-43a7-bf17-a35a40587f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 07 13:10:55.716666 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:55.716606    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:10:56.518779 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:56.518739    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6vglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 07 13:10:56.519191 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:56.518802    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg" podUID="03a5b4a5-e033-43a7-bf17-a35a40587f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 07 13:10:57.169241 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:57.169202    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-lkvfr"
Sep 07 13:10:57.518504 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:57.518474    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6vglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 07 13:10:57.518666 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:57.518539    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg" podUID="03a5b4a5-e033-43a7-bf17-a35a40587f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 07 13:10:58.518135 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:58.518082    2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6vglg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Sep 07 13:10:58.518542 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:58.518155    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg" podUID="03a5b4a5-e033-43a7-bf17-a35a40587f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Sep 07 13:10:59.522131 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:59.522099    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:10:59.522921 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:10:59.522901    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6vglg"
Sep 07 13:11:01.459758 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.459728    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"]
Sep 07 13:11:01.462146 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.462127    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.466095 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.466070    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 07 13:11:01.466221 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.466129    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 07 13:11:01.466407 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.466389    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 07 13:11:01.466407 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.466400    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-45cgz\""
Sep 07 13:11:01.478926 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.478903    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"]
Sep 07 13:11:01.614583 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.614552    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25d1f400-6375-4c37-b4fa-629a376674a7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.614750 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.614597    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d1f400-6375-4c37-b4fa-629a376674a7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.614750 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.614627    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25d1f400-6375-4c37-b4fa-629a376674a7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.614750 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.614701    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lg2f\" (UniqueName: \"kubernetes.io/projected/25d1f400-6375-4c37-b4fa-629a376674a7-kube-api-access-2lg2f\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.614867 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.614758    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25d1f400-6375-4c37-b4fa-629a376674a7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.614867 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.614784    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25d1f400-6375-4c37-b4fa-629a376674a7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.715935 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.715845    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d1f400-6375-4c37-b4fa-629a376674a7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.715935 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.715898    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25d1f400-6375-4c37-b4fa-629a376674a7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.715935 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.715930    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lg2f\" (UniqueName: \"kubernetes.io/projected/25d1f400-6375-4c37-b4fa-629a376674a7-kube-api-access-2lg2f\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.716251 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.715970    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25d1f400-6375-4c37-b4fa-629a376674a7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.716251 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.715984    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25d1f400-6375-4c37-b4fa-629a376674a7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.716251 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.716007    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25d1f400-6375-4c37-b4fa-629a376674a7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.716404 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.716331    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d1f400-6375-4c37-b4fa-629a376674a7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.716404 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.716352    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25d1f400-6375-4c37-b4fa-629a376674a7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.716499 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.716401    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25d1f400-6375-4c37-b4fa-629a376674a7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.718251 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.718223    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25d1f400-6375-4c37-b4fa-629a376674a7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.718432 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.718414    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25d1f400-6375-4c37-b4fa-629a376674a7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.726750 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.726722    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lg2f\" (UniqueName: \"kubernetes.io/projected/25d1f400-6375-4c37-b4fa-629a376674a7-kube-api-access-2lg2f\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk\" (UID: \"25d1f400-6375-4c37-b4fa-629a376674a7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:01.773875 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:01.773851    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:02.055407 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:02.055379    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"]
Sep 07 13:11:02.056402 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:11:02.056377    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d1f400_6375_4c37_b4fa_629a376674a7.slice/crio-2399ed467aec5a336f82e4a33308dec9c00a6dbcb9193ff6aec150a77f0161e7 WatchSource:0}: Error finding container 2399ed467aec5a336f82e4a33308dec9c00a6dbcb9193ff6aec150a77f0161e7: Status 404 returned error can't find the container with id 2399ed467aec5a336f82e4a33308dec9c00a6dbcb9193ff6aec150a77f0161e7
Sep 07 13:11:02.251873 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:02.251832    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk" event={"ID":"25d1f400-6375-4c37-b4fa-629a376674a7","Type":"ContainerStarted","Data":"2399ed467aec5a336f82e4a33308dec9c00a6dbcb9193ff6aec150a77f0161e7"}
Sep 07 13:11:03.259137 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:03.259101    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk" event={"ID":"25d1f400-6375-4c37-b4fa-629a376674a7","Type":"ContainerStarted","Data":"4fc18cfdd76a0265a20e6b90fa5c2dfbbb2570130c33dd5f15d86ea2af5dfadb"}
Sep 07 13:11:03.281731 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:03.281672    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk" podStartSLOduration=1.148136117 podStartE2EDuration="2.281649704s" totalImagesPullingTime="1.133513587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:11:03.281190638 +0000 UTC m=+707.709488939" watchObservedRunningTime="2026-09-07 13:11:03.281649704 +0000 UTC m=+707.709947981"
Sep 07 13:11:11.774311 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:11.774215    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:11.774311 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:11.774298    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:11.786419 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:11.786386    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:12.317729 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:12.317698    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk"
Sep 07 13:11:36.606382 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.606350    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"]
Sep 07 13:11:36.609673 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.609654    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.617994 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.617975    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 07 13:11:36.712913 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.712864    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14eb704-fc90-4a37-8835-069ea0fe87bf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.713132 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.712918    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f14eb704-fc90-4a37-8835-069ea0fe87bf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.713132 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.713031    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14eb704-fc90-4a37-8835-069ea0fe87bf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.713132 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.713092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f14eb704-fc90-4a37-8835-069ea0fe87bf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.713286 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.713204    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg95k\" (UniqueName: \"kubernetes.io/projected/f14eb704-fc90-4a37-8835-069ea0fe87bf-kube-api-access-cg95k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.713286 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.713242    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f14eb704-fc90-4a37-8835-069ea0fe87bf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.795863 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.795822    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"]
Sep 07 13:11:36.814671 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.814638    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg95k\" (UniqueName: \"kubernetes.io/projected/f14eb704-fc90-4a37-8835-069ea0fe87bf-kube-api-access-cg95k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.814835 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.814675    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f14eb704-fc90-4a37-8835-069ea0fe87bf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.814835 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.814702    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14eb704-fc90-4a37-8835-069ea0fe87bf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.814835 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.814721    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f14eb704-fc90-4a37-8835-069ea0fe87bf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.814835 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.814748    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14eb704-fc90-4a37-8835-069ea0fe87bf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.814835 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.814767    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f14eb704-fc90-4a37-8835-069ea0fe87bf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.815166 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.815140    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f14eb704-fc90-4a37-8835-069ea0fe87bf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.815166 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.815159    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f14eb704-fc90-4a37-8835-069ea0fe87bf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.815281 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.815250    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14eb704-fc90-4a37-8835-069ea0fe87bf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.817172 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.817145    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f14eb704-fc90-4a37-8835-069ea0fe87bf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.817356 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.817337    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14eb704-fc90-4a37-8835-069ea0fe87bf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.829679 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.829649    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg95k\" (UniqueName: \"kubernetes.io/projected/f14eb704-fc90-4a37-8835-069ea0fe87bf-kube-api-access-cg95k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4\" (UID: \"f14eb704-fc90-4a37-8835-069ea0fe87bf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:36.919308 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:36.919225    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:37.224502 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:37.224473    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"]
Sep 07 13:11:37.225419 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:11:37.225393    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14eb704_fc90_4a37_8835_069ea0fe87bf.slice/crio-9dbbcbd6c67bb6873d400ad3e8d1e52c162514e06d58df1eccba25f32038bdbd WatchSource:0}: Error finding container 9dbbcbd6c67bb6873d400ad3e8d1e52c162514e06d58df1eccba25f32038bdbd: Status 404 returned error can't find the container with id 9dbbcbd6c67bb6873d400ad3e8d1e52c162514e06d58df1eccba25f32038bdbd
Sep 07 13:11:37.440098 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:37.440045    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4" event={"ID":"f14eb704-fc90-4a37-8835-069ea0fe87bf","Type":"ContainerStarted","Data":"9dbbcbd6c67bb6873d400ad3e8d1e52c162514e06d58df1eccba25f32038bdbd"}
Sep 07 13:11:38.447314 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:38.447276    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4" event={"ID":"f14eb704-fc90-4a37-8835-069ea0fe87bf","Type":"ContainerStarted","Data":"de47e0164b466459c49a119b7e1ec3c13ba1171830ab3a57b03742d75939eec0"}
Sep 07 13:11:38.501400 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:38.501345    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4" podStartSLOduration=2.311992408 podStartE2EDuration="2.501325312s" totalImagesPullingTime="189.332904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:11:38.499565883 +0000 UTC m=+742.927864156" watchObservedRunningTime="2026-09-07 13:11:38.501325312 +0000 UTC m=+742.929623585"
Sep 07 13:11:46.919763 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:46.919723    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:46.920170 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:46.919784    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:46.932266 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:46.932241    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:11:47.510476 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:11:47.510444    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4"
Sep 07 13:13:56.080307 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.080273    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"]
Sep 07 13:13:56.082781 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.082761    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.085166 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.085139    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ebfa46-openshift-default-dockercfg-rjgtc\""
Sep 07 13:13:56.101989 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.101958    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"]
Sep 07 13:13:56.196136 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.196102    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-workload-socket\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.196319 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.196148    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-data\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.196319 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.196176    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istiod-ca-cert\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.196319 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.196251    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-credential-socket\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.196319 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.196302    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-podinfo\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.196539 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.196370    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-envoy\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.196539 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.196431    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-workload-certs\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.196539 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.196454    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stjtv\" (UniqueName: \"kubernetes.io/projected/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-kube-api-access-stjtv\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.196539 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.196482    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-token\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.297884 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.297841    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-workload-socket\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.297884 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.297889    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-data\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.298134 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.298107    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istiod-ca-cert\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.298201 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.298164    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-credential-socket\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.298263 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.298217    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-podinfo\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.298318 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.298286    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-envoy\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.298375 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.298336    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-workload-socket\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.298375 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.298340    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-workload-certs\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.298473 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.298385    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stjtv\" (UniqueName: \"kubernetes.io/projected/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-kube-api-access-stjtv\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.298473 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.298418    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-token\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.298584 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.298481    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-data\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.298584 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.298572    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-credential-socket\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.298715 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.298683    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-workload-certs\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.298908 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.298886    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istiod-ca-cert\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.300909 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.300876    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-envoy\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.301323 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.301302    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-podinfo\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.306853 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.306784    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-token\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.307284 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.307251    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stjtv\" (UniqueName: \"kubernetes.io/projected/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-kube-api-access-stjtv\") pod \"e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") " pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.395532 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.395446    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:56.693014 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.692988    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"]
Sep 07 13:13:56.695294 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:13:56.695262    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f4c689_aeda_4c34_9ffd_0f7ee0a24063.slice/crio-3159b4e3831a02f2dcaf1df5307fe3257f02f92d6fb022ed3685b1fe8a1aef27 WatchSource:0}: Error finding container 3159b4e3831a02f2dcaf1df5307fe3257f02f92d6fb022ed3685b1fe8a1aef27: Status 404 returned error can't find the container with id 3159b4e3831a02f2dcaf1df5307fe3257f02f92d6fb022ed3685b1fe8a1aef27
Sep 07 13:13:56.697668 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.697632    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:13:56.697736 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.697720    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:13:56.697773 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:56.697762    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:13:57.185599 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:57.185556    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" event={"ID":"99f4c689-aeda-4c34-9ffd-0f7ee0a24063","Type":"ContainerStarted","Data":"b8db830900a1f6c1889b08eb6e4ab94647ed93c699b92bdfda972fcd6951490e"}
Sep 07 13:13:57.186016 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:57.185619    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" event={"ID":"99f4c689-aeda-4c34-9ffd-0f7ee0a24063","Type":"ContainerStarted","Data":"3159b4e3831a02f2dcaf1df5307fe3257f02f92d6fb022ed3685b1fe8a1aef27"}
Sep 07 13:13:57.220356 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:57.220303    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" podStartSLOduration=1.220287019 podStartE2EDuration="1.220287019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:13:57.216491855 +0000 UTC m=+881.644790141" watchObservedRunningTime="2026-09-07 13:13:57.220287019 +0000 UTC m=+881.648585291"
Sep 07 13:13:57.396323 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:57.396236    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:57.401966 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:57.401940    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:58.192107 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:58.192070    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:13:58.192997 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:13:58.192976    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:14:00.387414 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.387372    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"]
Sep 07 13:14:00.390430 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.390408    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.392554 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.392535    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ebfa46-serving-cert\""
Sep 07 13:14:00.404108 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.404085    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"]
Sep 07 13:14:00.436888 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.436861    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.437041 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.436908    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-metrics-certs\") pod \"maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.437041 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.436941    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xt8m\" (UniqueName: \"kubernetes.io/projected/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-kube-api-access-5xt8m\") pod \"maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.437167 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.437031    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-maas-api-tls\") pod \"maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.538099 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.538041    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.538301 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.538126    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-metrics-certs\") pod \"maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.538301 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.538150    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xt8m\" (UniqueName: \"kubernetes.io/projected/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-kube-api-access-5xt8m\") pod \"maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.538301 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.538181    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-maas-api-tls\") pod \"maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.538818 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.538786    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.540848 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.540823    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-metrics-certs\") pod \"maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.540952 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.540899    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-maas-api-tls\") pod \"maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.551219 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.551188    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xt8m\" (UniqueName: \"kubernetes.io/projected/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-kube-api-access-5xt8m\") pod \"maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.702803 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.702695    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:00.984713 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:00.984326    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"]
Sep 07 13:14:00.986708 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:14:00.986676    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c6da54_7486_4cfe_83d3_7e421e3c55d6.slice/crio-515a5232bb50547fecd7ef4a4f7b6a0578ca5c9684232b936c600222feaad385 WatchSource:0}: Error finding container 515a5232bb50547fecd7ef4a4f7b6a0578ca5c9684232b936c600222feaad385: Status 404 returned error can't find the container with id 515a5232bb50547fecd7ef4a4f7b6a0578ca5c9684232b936c600222feaad385
Sep 07 13:14:01.209701 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:01.209661    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b" event={"ID":"d8c6da54-7486-4cfe-83d3-7e421e3c55d6","Type":"ContainerStarted","Data":"515a5232bb50547fecd7ef4a4f7b6a0578ca5c9684232b936c600222feaad385"}
Sep 07 13:14:02.217445 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:02.217411    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b" event={"ID":"d8c6da54-7486-4cfe-83d3-7e421e3c55d6","Type":"ContainerStarted","Data":"93e072dc9700cfba2a3085559cfcfa1e850bfcede04d24340b58eebee4539dbc"}
Sep 07 13:14:02.217874 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:02.217511    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:02.237451 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:02.237392    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b" podStartSLOduration=1.680485617 podStartE2EDuration="2.237371129s" totalImagesPullingTime="556.885512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:02.2354759 +0000 UTC m=+886.663774197" watchObservedRunningTime="2026-09-07 13:14:02.237371129 +0000 UTC m=+886.665669403"
Sep 07 13:14:08.229335 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:08.229303    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:14:09.136873 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.136833    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"]
Sep 07 13:14:09.142395 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.142368    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:09.144806 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.144783    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e07a5db5-serving-cert\""
Sep 07 13:14:09.164318 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.164291    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"]
Sep 07 13:14:09.230160 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.230122    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-maas-api-tls\") pod \"maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:09.230160 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.230166    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6dw6\" (UniqueName: \"kubernetes.io/projected/be444a8a-893e-41ee-a407-c24675280fad-kube-api-access-z6dw6\") pod \"maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:09.230651 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.230186    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-metrics-certs\") pod \"maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:09.230651 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.230347    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be444a8a-893e-41ee-a407-c24675280fad-openshift-service-ca\") pod \"maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:09.331621 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.331574    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-maas-api-tls\") pod \"maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:09.331820 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.331632    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6dw6\" (UniqueName: \"kubernetes.io/projected/be444a8a-893e-41ee-a407-c24675280fad-kube-api-access-z6dw6\") pod \"maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:09.331820 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.331662    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-metrics-certs\") pod \"maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:09.331820 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.331745    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be444a8a-893e-41ee-a407-c24675280fad-openshift-service-ca\") pod \"maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:09.332007 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:14:09.331985    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-serving-cert: secret "maas-api-e2e-ait-e07a5db5-serving-cert" not found
Sep 07 13:14:09.332091 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:14:09.332042    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-maas-api-tls podName:be444a8a-893e-41ee-a407-c24675280fad nodeName:}" failed. No retries permitted until 2026-09-07 13:14:09.832027619 +0000 UTC m=+894.260325875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-maas-api-tls") pod "maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk" (UID: "be444a8a-893e-41ee-a407-c24675280fad") : secret "maas-api-e2e-ait-e07a5db5-serving-cert" not found
Sep 07 13:14:09.332530 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.332506    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be444a8a-893e-41ee-a407-c24675280fad-openshift-service-ca\") pod \"maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:09.334842 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.334815    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-metrics-certs\") pod \"maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:09.356478 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.356442    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6dw6\" (UniqueName: \"kubernetes.io/projected/be444a8a-893e-41ee-a407-c24675280fad-kube-api-access-z6dw6\") pod \"maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:09.836734 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.836696    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-maas-api-tls\") pod \"maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:09.839140 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:09.839114    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-maas-api-tls\") pod \"maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:10.054437 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.054399    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:10.402109 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.391940    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"]
Sep 07 13:14:10.404869 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:14:10.404818    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe444a8a_893e_41ee_a407_c24675280fad.slice/crio-cb859ab41f1dc5a53ca863c8e2153c6c284acb5bb715fa14efc4ba99f6c7b86c WatchSource:0}: Error finding container cb859ab41f1dc5a53ca863c8e2153c6c284acb5bb715fa14efc4ba99f6c7b86c: Status 404 returned error can't find the container with id cb859ab41f1dc5a53ca863c8e2153c6c284acb5bb715fa14efc4ba99f6c7b86c
Sep 07 13:14:10.799343 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.799303    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"]
Sep 07 13:14:10.819196 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.819157    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"]
Sep 07 13:14:10.819380 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.819293    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:10.822105 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.821869    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-adacb1-serving-cert\""
Sep 07 13:14:10.847144 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.847112    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd02188-3e10-4701-8805-02a77ab50447-openshift-service-ca\") pod \"maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:10.847288 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.847157    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84xj6\" (UniqueName: \"kubernetes.io/projected/dbd02188-3e10-4701-8805-02a77ab50447-kube-api-access-84xj6\") pod \"maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:10.847339 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.847278    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-maas-api-tls\") pod \"maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:10.847339 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.847322    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-metrics-certs\") pod \"maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:10.948514 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.948485    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd02188-3e10-4701-8805-02a77ab50447-openshift-service-ca\") pod \"maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:10.948683 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.948528    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84xj6\" (UniqueName: \"kubernetes.io/projected/dbd02188-3e10-4701-8805-02a77ab50447-kube-api-access-84xj6\") pod \"maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:10.948683 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.948564    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-maas-api-tls\") pod \"maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:10.948683 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.948583    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-metrics-certs\") pod \"maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:10.949404 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.949281    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd02188-3e10-4701-8805-02a77ab50447-openshift-service-ca\") pod \"maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:10.949612 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:14:10.949575    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-serving-cert: secret "maas-api-e2e-autores-adacb1-serving-cert" not found
Sep 07 13:14:10.949859 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:14:10.949835    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-maas-api-tls podName:dbd02188-3e10-4701-8805-02a77ab50447 nodeName:}" failed. No retries permitted until 2026-09-07 13:14:11.449783953 +0000 UTC m=+895.878082210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-maas-api-tls") pod "maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw" (UID: "dbd02188-3e10-4701-8805-02a77ab50447") : secret "maas-api-e2e-autores-adacb1-serving-cert" not found
Sep 07 13:14:10.951124 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.951087    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-metrics-certs\") pod \"maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:10.962552 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:10.962526    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84xj6\" (UniqueName: \"kubernetes.io/projected/dbd02188-3e10-4701-8805-02a77ab50447-kube-api-access-84xj6\") pod \"maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:11.270085 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.270035    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk" event={"ID":"be444a8a-893e-41ee-a407-c24675280fad","Type":"ContainerStarted","Data":"dec53bbdc69dfa5a12d82672e0275fad1a5048b43675ec28382dd61e9f88f495"}
Sep 07 13:14:11.270266 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.270121    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:11.270266 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.270145    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk" event={"ID":"be444a8a-893e-41ee-a407-c24675280fad","Type":"ContainerStarted","Data":"cb859ab41f1dc5a53ca863c8e2153c6c284acb5bb715fa14efc4ba99f6c7b86c"}
Sep 07 13:14:11.453863 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.453826    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-maas-api-tls\") pod \"maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:11.456304 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.456271    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-maas-api-tls\") pod \"maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:11.665217 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.665150    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk" podStartSLOduration=2.216089859 podStartE2EDuration="2.665123505s" totalImagesPullingTime="449.033646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:11.294439946 +0000 UTC m=+895.722738231" watchObservedRunningTime="2026-09-07 13:14:11.665123505 +0000 UTC m=+896.093421780"
Sep 07 13:14:11.665420 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.665409    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"]
Sep 07 13:14:11.670667 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.670646    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"
Sep 07 13:14:11.672920 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.672894    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-adacb1-dockercfg-vjx9j\""
Sep 07 13:14:11.673367 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.673347    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-adacb1\""
Sep 07 13:14:11.681384 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.681356    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"]
Sep 07 13:14:11.733760 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.733729    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:11.757189 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.757155    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zhm9\" (UniqueName: \"kubernetes.io/projected/966d0a64-7ed0-4cab-9e32-febcd7a04bdc-kube-api-access-2zhm9\") pod \"payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll\" (UID: \"966d0a64-7ed0-4cab-9e32-febcd7a04bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"
Sep 07 13:14:11.757360 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.757227    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966d0a64-7ed0-4cab-9e32-febcd7a04bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll\" (UID: \"966d0a64-7ed0-4cab-9e32-febcd7a04bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"
Sep 07 13:14:11.858517 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.858478    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zhm9\" (UniqueName: \"kubernetes.io/projected/966d0a64-7ed0-4cab-9e32-febcd7a04bdc-kube-api-access-2zhm9\") pod \"payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll\" (UID: \"966d0a64-7ed0-4cab-9e32-febcd7a04bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"
Sep 07 13:14:11.858517 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.858528    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966d0a64-7ed0-4cab-9e32-febcd7a04bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll\" (UID: \"966d0a64-7ed0-4cab-9e32-febcd7a04bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"
Sep 07 13:14:11.859284 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.859258    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966d0a64-7ed0-4cab-9e32-febcd7a04bdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll\" (UID: \"966d0a64-7ed0-4cab-9e32-febcd7a04bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"
Sep 07 13:14:11.866740 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.866676    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zhm9\" (UniqueName: \"kubernetes.io/projected/966d0a64-7ed0-4cab-9e32-febcd7a04bdc-kube-api-access-2zhm9\") pod \"payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll\" (UID: \"966d0a64-7ed0-4cab-9e32-febcd7a04bdc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"
Sep 07 13:14:11.984176 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:11.984137    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"
Sep 07 13:14:12.022278 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:12.022000    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"]
Sep 07 13:14:12.022497 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:14:12.022465    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd02188_3e10_4701_8805_02a77ab50447.slice/crio-2176609474adbd0ea32d9d1ab076123f90e57dbabcb05814d58973bc8e22788a WatchSource:0}: Error finding container 2176609474adbd0ea32d9d1ab076123f90e57dbabcb05814d58973bc8e22788a: Status 404 returned error can't find the container with id 2176609474adbd0ea32d9d1ab076123f90e57dbabcb05814d58973bc8e22788a
Sep 07 13:14:12.266250 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:12.266163    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"]
Sep 07 13:14:12.269841 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:14:12.269795    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966d0a64_7ed0_4cab_9e32_febcd7a04bdc.slice/crio-e6d474afca1cb4a91e844ae6666796683c6c36ad7add6f01b6692015743f7d36 WatchSource:0}: Error finding container e6d474afca1cb4a91e844ae6666796683c6c36ad7add6f01b6692015743f7d36: Status 404 returned error can't find the container with id e6d474afca1cb4a91e844ae6666796683c6c36ad7add6f01b6692015743f7d36
Sep 07 13:14:12.279549 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:12.279516    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll" event={"ID":"966d0a64-7ed0-4cab-9e32-febcd7a04bdc","Type":"ContainerStarted","Data":"e6d474afca1cb4a91e844ae6666796683c6c36ad7add6f01b6692015743f7d36"}
Sep 07 13:14:12.284109 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:12.284031    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw" event={"ID":"dbd02188-3e10-4701-8805-02a77ab50447","Type":"ContainerStarted","Data":"2176609474adbd0ea32d9d1ab076123f90e57dbabcb05814d58973bc8e22788a"}
Sep 07 13:14:13.292276 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:13.292236    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw" event={"ID":"dbd02188-3e10-4701-8805-02a77ab50447","Type":"ContainerStarted","Data":"65bb10804f78c7c6edd40bc5bcd9147a6360a692b2295d137f02492a762a3d79"}
Sep 07 13:14:13.292846 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:13.292815    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:13.294430 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:13.294399    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll" event={"ID":"966d0a64-7ed0-4cab-9e32-febcd7a04bdc","Type":"ContainerStarted","Data":"3967b55e35cae889cbf12e77e76a9f271986f88b51b9c6be4c1b36281e9d1b84"}
Sep 07 13:14:13.294542 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:13.294479    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"
Sep 07 13:14:13.315999 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:13.315923    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw" podStartSLOduration=2.605128051 podStartE2EDuration="3.315899504s" totalImagesPullingTime="710.771453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:13.310774941 +0000 UTC m=+897.739073240" watchObservedRunningTime="2026-09-07 13:14:13.315899504 +0000 UTC m=+897.744197779"
Sep 07 13:14:13.332137 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:13.332012    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll" podStartSLOduration=2.331988456 podStartE2EDuration="2.331988456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:13.32741807 +0000 UTC m=+897.755716368" watchObservedRunningTime="2026-09-07 13:14:13.331988456 +0000 UTC m=+897.760286730"
Sep 07 13:14:14.234299 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:14.234252    2567 patch_prober.go:28] interesting pod/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 13:14:14.234632 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:14.234598    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 13:14:15.308740 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:15.308696    2567 generic.go:356] "Generic (PLEG): container finished" podID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerID="b8db830900a1f6c1889b08eb6e4ab94647ed93c699b92bdfda972fcd6951490e" exitCode=0
Sep 07 13:14:15.309205 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:15.308757    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" event={"ID":"99f4c689-aeda-4c34-9ffd-0f7ee0a24063","Type":"ContainerDied","Data":"b8db830900a1f6c1889b08eb6e4ab94647ed93c699b92bdfda972fcd6951490e"}
Sep 07 13:14:15.309256 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:15.309203    2567 scope.go:117] "RemoveContainer" containerID="b8db830900a1f6c1889b08eb6e4ab94647ed93c699b92bdfda972fcd6951490e"
Sep 07 13:14:15.311292 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:15.310602    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:14:15.311292 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:15.310828    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:14:15.311292 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:15.310956    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:14:16.316248 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:16.316209    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" event={"ID":"99f4c689-aeda-4c34-9ffd-0f7ee0a24063","Type":"ContainerStarted","Data":"5ce8f3900d86fde20626b1a47dbe3e75eb354e86e45de6be39f5c89d53d0a521"}
Sep 07 13:14:16.396356 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:16.396309    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:14:16.396560 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:16.396463    2567 patch_prober.go:28] interesting pod/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 07 13:14:16.396560 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:16.396524    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 07 13:14:17.295242 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:17.295202    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:17.403132 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:17.401457    2567 patch_prober.go:28] interesting pod/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 07 13:14:17.403132 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:17.401527    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 07 13:14:18.396236 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:18.396187    2567 patch_prober.go:28] interesting pod/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 07 13:14:18.396533 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:18.396261    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 07 13:14:19.307251 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:19.307217    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:14:19.402092 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:19.399522    2567 patch_prober.go:28] interesting pod/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 07 13:14:19.402092 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:19.399590    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 07 13:14:20.396882 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:20.396833    2567 patch_prober.go:28] interesting pod/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 07 13:14:20.397421 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:20.396907    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 07 13:14:21.482531 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:21.482493    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:14:21.483136 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:21.483088    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:14:21.484198 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:21.484169    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:14:24.303677 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:24.303636    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"
Sep 07 13:14:36.382508 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.382469    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"]
Sep 07 13:14:36.386666 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.386640    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:36.389160 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.389134    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-824a79-serving-cert\""
Sep 07 13:14:36.399965 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.399932    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"]
Sep 07 13:14:36.408458 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.408424    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxv8g\" (UniqueName: \"kubernetes.io/projected/06ae1771-be89-425c-9bd9-79fb318663fa-kube-api-access-wxv8g\") pod \"maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:36.408675 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.408542    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-maas-api-tls\") pod \"maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:36.408675 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.408649    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-metrics-certs\") pod \"maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:36.408825 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.408690    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ae1771-be89-425c-9bd9-79fb318663fa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:36.509746 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.509704    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv8g\" (UniqueName: \"kubernetes.io/projected/06ae1771-be89-425c-9bd9-79fb318663fa-kube-api-access-wxv8g\") pod \"maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:36.509936 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.509751    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-maas-api-tls\") pod \"maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:36.509936 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.509923    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-metrics-certs\") pod \"maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:36.510031 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:14:36.509945    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-serving-cert: secret "maas-api-e2e-inf-b-824a79-serving-cert" not found
Sep 07 13:14:36.510031 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:14:36.510022    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-maas-api-tls podName:06ae1771-be89-425c-9bd9-79fb318663fa nodeName:}" failed. No retries permitted until 2026-09-07 13:14:37.010000526 +0000 UTC m=+921.438298784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-maas-api-tls") pod "maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk" (UID: "06ae1771-be89-425c-9bd9-79fb318663fa") : secret "maas-api-e2e-inf-b-824a79-serving-cert" not found
Sep 07 13:14:36.510151 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.509952    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ae1771-be89-425c-9bd9-79fb318663fa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:36.510563 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.510543    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ae1771-be89-425c-9bd9-79fb318663fa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:36.512671 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.512647    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-metrics-certs\") pod \"maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:36.520023 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:36.519996    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxv8g\" (UniqueName: \"kubernetes.io/projected/06ae1771-be89-425c-9bd9-79fb318663fa-kube-api-access-wxv8g\") pod \"maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:37.014628 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:37.014587    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-maas-api-tls\") pod \"maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:37.017163 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:37.017136    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-maas-api-tls\") pod \"maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:37.300410 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:37.300376    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:37.575270 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:37.575238    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"]
Sep 07 13:14:37.576668 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:14:37.576637    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ae1771_be89_425c_9bd9_79fb318663fa.slice/crio-e55d3cc42826c484b99175fba347d377391ee7eb92befb477d2c1b0c35852f89 WatchSource:0}: Error finding container e55d3cc42826c484b99175fba347d377391ee7eb92befb477d2c1b0c35852f89: Status 404 returned error can't find the container with id e55d3cc42826c484b99175fba347d377391ee7eb92befb477d2c1b0c35852f89
Sep 07 13:14:38.451655 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:38.451607    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk" event={"ID":"06ae1771-be89-425c-9bd9-79fb318663fa","Type":"ContainerStarted","Data":"befe62669bb82ba1dd8230b8e9ad57bb14c883502f0e2dffd272752b31eff67f"}
Sep 07 13:14:38.451842 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:38.451672    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk" event={"ID":"06ae1771-be89-425c-9bd9-79fb318663fa","Type":"ContainerStarted","Data":"e55d3cc42826c484b99175fba347d377391ee7eb92befb477d2c1b0c35852f89"}
Sep 07 13:14:38.451842 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:38.451719    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:38.471527 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:38.471477    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk" podStartSLOduration=2.039675284 podStartE2EDuration="2.471460279s" totalImagesPullingTime="431.784995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:38.469395054 +0000 UTC m=+922.897693335" watchObservedRunningTime="2026-09-07 13:14:38.471460279 +0000 UTC m=+922.899758552"
Sep 07 13:14:44.463555 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:44.463525    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:14:50.014782 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.014743    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"]
Sep 07 13:14:50.015203 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.014990    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk" podUID="be444a8a-893e-41ee-a407-c24675280fad" containerName="maas-api" containerID="cri-o://dec53bbdc69dfa5a12d82672e0275fad1a5048b43675ec28382dd61e9f88f495" gracePeriod=30
Sep 07 13:14:50.273515 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.273488    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:50.336482 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.336437    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-maas-api-tls\") pod \"be444a8a-893e-41ee-a407-c24675280fad\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") "
Sep 07 13:14:50.336666 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.336519    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-metrics-certs\") pod \"be444a8a-893e-41ee-a407-c24675280fad\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") "
Sep 07 13:14:50.336666 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.336614    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be444a8a-893e-41ee-a407-c24675280fad-openshift-service-ca\") pod \"be444a8a-893e-41ee-a407-c24675280fad\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") "
Sep 07 13:14:50.336666 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.336656    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6dw6\" (UniqueName: \"kubernetes.io/projected/be444a8a-893e-41ee-a407-c24675280fad-kube-api-access-z6dw6\") pod \"be444a8a-893e-41ee-a407-c24675280fad\" (UID: \"be444a8a-893e-41ee-a407-c24675280fad\") "
Sep 07 13:14:50.337019 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.336979    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be444a8a-893e-41ee-a407-c24675280fad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be444a8a-893e-41ee-a407-c24675280fad" (UID: "be444a8a-893e-41ee-a407-c24675280fad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:14:50.338749 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.338720    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be444a8a-893e-41ee-a407-c24675280fad" (UID: "be444a8a-893e-41ee-a407-c24675280fad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:14:50.338827 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.338773    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "be444a8a-893e-41ee-a407-c24675280fad" (UID: "be444a8a-893e-41ee-a407-c24675280fad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:14:50.338827 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.338798    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be444a8a-893e-41ee-a407-c24675280fad-kube-api-access-z6dw6" (OuterVolumeSpecName: "kube-api-access-z6dw6") pod "be444a8a-893e-41ee-a407-c24675280fad" (UID: "be444a8a-893e-41ee-a407-c24675280fad"). InnerVolumeSpecName "kube-api-access-z6dw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:14:50.437603 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.437559    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6dw6\" (UniqueName: \"kubernetes.io/projected/be444a8a-893e-41ee-a407-c24675280fad-kube-api-access-z6dw6\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:14:50.437603 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.437595    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:14:50.437603 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.437608    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be444a8a-893e-41ee-a407-c24675280fad-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:14:50.437841 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.437620    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be444a8a-893e-41ee-a407-c24675280fad-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:14:50.536153 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.536046    2567 generic.go:356] "Generic (PLEG): container finished" podID="be444a8a-893e-41ee-a407-c24675280fad" containerID="dec53bbdc69dfa5a12d82672e0275fad1a5048b43675ec28382dd61e9f88f495" exitCode=0
Sep 07 13:14:50.536153 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.536109    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk" event={"ID":"be444a8a-893e-41ee-a407-c24675280fad","Type":"ContainerDied","Data":"dec53bbdc69dfa5a12d82672e0275fad1a5048b43675ec28382dd61e9f88f495"}
Sep 07 13:14:50.536388 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.536159    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk" event={"ID":"be444a8a-893e-41ee-a407-c24675280fad","Type":"ContainerDied","Data":"cb859ab41f1dc5a53ca863c8e2153c6c284acb5bb715fa14efc4ba99f6c7b86c"}
Sep 07 13:14:50.536388 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.536115    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"
Sep 07 13:14:50.536388 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.536188    2567 scope.go:117] "RemoveContainer" containerID="dec53bbdc69dfa5a12d82672e0275fad1a5048b43675ec28382dd61e9f88f495"
Sep 07 13:14:50.548792 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.548751    2567 scope.go:117] "RemoveContainer" containerID="dec53bbdc69dfa5a12d82672e0275fad1a5048b43675ec28382dd61e9f88f495"
Sep 07 13:14:50.549688 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:14:50.549167    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec53bbdc69dfa5a12d82672e0275fad1a5048b43675ec28382dd61e9f88f495\": container with ID starting with dec53bbdc69dfa5a12d82672e0275fad1a5048b43675ec28382dd61e9f88f495 not found: ID does not exist" containerID="dec53bbdc69dfa5a12d82672e0275fad1a5048b43675ec28382dd61e9f88f495"
Sep 07 13:14:50.549813 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.549720    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec53bbdc69dfa5a12d82672e0275fad1a5048b43675ec28382dd61e9f88f495"} err="failed to get container status \"dec53bbdc69dfa5a12d82672e0275fad1a5048b43675ec28382dd61e9f88f495\": rpc error: code = NotFound desc = could not find container \"dec53bbdc69dfa5a12d82672e0275fad1a5048b43675ec28382dd61e9f88f495\": container with ID starting with dec53bbdc69dfa5a12d82672e0275fad1a5048b43675ec28382dd61e9f88f495 not found: ID does not exist"
Sep 07 13:14:50.567991 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.567964    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"]
Sep 07 13:14:50.571509 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:50.571481    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e07a5db5-54fdfdd5d-pxgkk"]
Sep 07 13:14:52.076617 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:52.076581    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be444a8a-893e-41ee-a407-c24675280fad" path="/var/lib/kubelet/pods/be444a8a-893e-41ee-a407-c24675280fad/volumes"
Sep 07 13:14:54.238152 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.238118    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"]
Sep 07 13:14:54.238585 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.238568    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be444a8a-893e-41ee-a407-c24675280fad" containerName="maas-api"
Sep 07 13:14:54.238631 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.238591    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="be444a8a-893e-41ee-a407-c24675280fad" containerName="maas-api"
Sep 07 13:14:54.238670 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.238660    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="be444a8a-893e-41ee-a407-c24675280fad" containerName="maas-api"
Sep 07 13:14:54.242175 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.242148    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.245475 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.245453    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-50f0dc8e-serving-cert\""
Sep 07 13:14:54.260997 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.260969    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"]
Sep 07 13:14:54.374283 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.374246    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-465nd\" (UniqueName: \"kubernetes.io/projected/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-kube-api-access-465nd\") pod \"maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.374452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.374373    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-metrics-certs\") pod \"maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.374521 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.374455    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-openshift-service-ca\") pod \"maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.374521 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.374513    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-maas-api-tls\") pod \"maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.475225 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.475180    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-maas-api-tls\") pod \"maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.475425 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.475252    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-465nd\" (UniqueName: \"kubernetes.io/projected/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-kube-api-access-465nd\") pod \"maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.475425 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.475315    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-metrics-certs\") pod \"maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.475425 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.475350    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-openshift-service-ca\") pod \"maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.476133 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.476094    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-openshift-service-ca\") pod \"maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.477852 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.477820    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-metrics-certs\") pod \"maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.477955 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.477826    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-maas-api-tls\") pod \"maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.484213 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.484185    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-465nd\" (UniqueName: \"kubernetes.io/projected/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-kube-api-access-465nd\") pod \"maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.553248 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.553212    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:54.834155 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:54.833656    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"]
Sep 07 13:14:54.835352 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:14:54.835319    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1e36b0_7d8a_4486_9c57_8eefeea61f6b.slice/crio-89184c095dfd32e9bb88aba70d6ea175853025d69e35468eedbaf0ccfe5c9d64 WatchSource:0}: Error finding container 89184c095dfd32e9bb88aba70d6ea175853025d69e35468eedbaf0ccfe5c9d64: Status 404 returned error can't find the container with id 89184c095dfd32e9bb88aba70d6ea175853025d69e35468eedbaf0ccfe5c9d64
Sep 07 13:14:55.571589 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:55.571553    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck" event={"ID":"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b","Type":"ContainerStarted","Data":"f633a2b5cc1aac191053de345dedd2ad7b972e9d8afef9220820d6c58c585461"}
Sep 07 13:14:55.572090 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:55.571612    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck" event={"ID":"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b","Type":"ContainerStarted","Data":"89184c095dfd32e9bb88aba70d6ea175853025d69e35468eedbaf0ccfe5c9d64"}
Sep 07 13:14:55.572090 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:55.571669    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:14:55.589965 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:14:55.589912    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck" podStartSLOduration=1.24740246 podStartE2EDuration="1.589895893s" totalImagesPullingTime="342.493433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:55.587228862 +0000 UTC m=+940.015527155" watchObservedRunningTime="2026-09-07 13:14:55.589895893 +0000 UTC m=+940.018194204"
Sep 07 13:15:01.584114 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:01.584050    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:15:25.760697 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:25.760661    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"]
Sep 07 13:15:25.761248 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:25.760873    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck" podUID="3c1e36b0-7d8a-4486-9c57-8eefeea61f6b" containerName="maas-api" containerID="cri-o://f633a2b5cc1aac191053de345dedd2ad7b972e9d8afef9220820d6c58c585461" gracePeriod=30
Sep 07 13:15:26.026886 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.026862    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:15:26.087635 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.087604    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-maas-api-tls\") pod \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") "
Sep 07 13:15:26.087798 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.087641    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-openshift-service-ca\") pod \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") "
Sep 07 13:15:26.087798 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.087671    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-metrics-certs\") pod \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") "
Sep 07 13:15:26.087798 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.087703    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-465nd\" (UniqueName: \"kubernetes.io/projected/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-kube-api-access-465nd\") pod \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\" (UID: \"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b\") "
Sep 07 13:15:26.088162 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.088131    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3c1e36b0-7d8a-4486-9c57-8eefeea61f6b" (UID: "3c1e36b0-7d8a-4486-9c57-8eefeea61f6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:15:26.089760 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.089733    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3c1e36b0-7d8a-4486-9c57-8eefeea61f6b" (UID: "3c1e36b0-7d8a-4486-9c57-8eefeea61f6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:15:26.089878 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.089774    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3c1e36b0-7d8a-4486-9c57-8eefeea61f6b" (UID: "3c1e36b0-7d8a-4486-9c57-8eefeea61f6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:15:26.089878 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.089789    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-kube-api-access-465nd" (OuterVolumeSpecName: "kube-api-access-465nd") pod "3c1e36b0-7d8a-4486-9c57-8eefeea61f6b" (UID: "3c1e36b0-7d8a-4486-9c57-8eefeea61f6b"). InnerVolumeSpecName "kube-api-access-465nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:15:26.189110 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.189052    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-465nd\" (UniqueName: \"kubernetes.io/projected/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-kube-api-access-465nd\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:15:26.189110 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.189099    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:15:26.189110 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.189108    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:15:26.189333 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.189119    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:15:26.757720 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.757681    2567 generic.go:356] "Generic (PLEG): container finished" podID="3c1e36b0-7d8a-4486-9c57-8eefeea61f6b" containerID="f633a2b5cc1aac191053de345dedd2ad7b972e9d8afef9220820d6c58c585461" exitCode=0
Sep 07 13:15:26.757920 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.757728    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"
Sep 07 13:15:26.757920 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.757728    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck" event={"ID":"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b","Type":"ContainerDied","Data":"f633a2b5cc1aac191053de345dedd2ad7b972e9d8afef9220820d6c58c585461"}
Sep 07 13:15:26.757920 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.757869    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck" event={"ID":"3c1e36b0-7d8a-4486-9c57-8eefeea61f6b","Type":"ContainerDied","Data":"89184c095dfd32e9bb88aba70d6ea175853025d69e35468eedbaf0ccfe5c9d64"}
Sep 07 13:15:26.757920 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.757907    2567 scope.go:117] "RemoveContainer" containerID="f633a2b5cc1aac191053de345dedd2ad7b972e9d8afef9220820d6c58c585461"
Sep 07 13:15:26.774581 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.774468    2567 scope.go:117] "RemoveContainer" containerID="f633a2b5cc1aac191053de345dedd2ad7b972e9d8afef9220820d6c58c585461"
Sep 07 13:15:26.774807 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:15:26.774761    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f633a2b5cc1aac191053de345dedd2ad7b972e9d8afef9220820d6c58c585461\": container with ID starting with f633a2b5cc1aac191053de345dedd2ad7b972e9d8afef9220820d6c58c585461 not found: ID does not exist" containerID="f633a2b5cc1aac191053de345dedd2ad7b972e9d8afef9220820d6c58c585461"
Sep 07 13:15:26.774807 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.774788    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f633a2b5cc1aac191053de345dedd2ad7b972e9d8afef9220820d6c58c585461"} err="failed to get container status \"f633a2b5cc1aac191053de345dedd2ad7b972e9d8afef9220820d6c58c585461\": rpc error: code = NotFound desc = could not find container \"f633a2b5cc1aac191053de345dedd2ad7b972e9d8afef9220820d6c58c585461\": container with ID starting with f633a2b5cc1aac191053de345dedd2ad7b972e9d8afef9220820d6c58c585461 not found: ID does not exist"
Sep 07 13:15:26.785691 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.785661    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"]
Sep 07 13:15:26.789151 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:26.789121    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-50f0dc8e-67f8654557-wbrck"]
Sep 07 13:15:28.075128 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:28.075082    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c1e36b0-7d8a-4486-9c57-8eefeea61f6b" path="/var/lib/kubelet/pods/3c1e36b0-7d8a-4486-9c57-8eefeea61f6b/volumes"
Sep 07 13:15:30.430221 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.430186    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"]
Sep 07 13:15:30.430781 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.430759    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1e36b0-7d8a-4486-9c57-8eefeea61f6b" containerName="maas-api"
Sep 07 13:15:30.430781 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.430779    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1e36b0-7d8a-4486-9c57-8eefeea61f6b" containerName="maas-api"
Sep 07 13:15:30.430993 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.430888    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1e36b0-7d8a-4486-9c57-8eefeea61f6b" containerName="maas-api"
Sep 07 13:15:30.435852 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.435831    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:30.438009 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.437986    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7d32a524-serving-cert\""
Sep 07 13:15:30.446633 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.446611    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"]
Sep 07 13:15:30.528803 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.528760    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tlrb\" (UniqueName: \"kubernetes.io/projected/2c581508-933c-4786-8d50-90fa3d0ef4ae-kube-api-access-5tlrb\") pod \"maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:30.529004 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.528923    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c581508-933c-4786-8d50-90fa3d0ef4ae-openshift-service-ca\") pod \"maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:30.529004 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.528988    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-metrics-certs\") pod \"maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:30.529159 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.529018    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-maas-api-tls\") pod \"maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:30.629765 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.629725    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-metrics-certs\") pod \"maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:30.629765 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.629765    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-maas-api-tls\") pod \"maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:30.630031 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.629792    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tlrb\" (UniqueName: \"kubernetes.io/projected/2c581508-933c-4786-8d50-90fa3d0ef4ae-kube-api-access-5tlrb\") pod \"maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:30.630031 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.629860    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c581508-933c-4786-8d50-90fa3d0ef4ae-openshift-service-ca\") pod \"maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:30.630031 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:15:30.629879    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-serving-cert: secret "maas-api-e2e-ait-7d32a524-serving-cert" not found
Sep 07 13:15:30.630031 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:15:30.629944    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-maas-api-tls podName:2c581508-933c-4786-8d50-90fa3d0ef4ae nodeName:}" failed. No retries permitted until 2026-09-07 13:15:31.129927542 +0000 UTC m=+975.558225793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-maas-api-tls") pod "maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws" (UID: "2c581508-933c-4786-8d50-90fa3d0ef4ae") : secret "maas-api-e2e-ait-7d32a524-serving-cert" not found
Sep 07 13:15:30.630578 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.630557    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c581508-933c-4786-8d50-90fa3d0ef4ae-openshift-service-ca\") pod \"maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:30.632419 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.632387    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-metrics-certs\") pod \"maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:30.640558 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:30.640532    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tlrb\" (UniqueName: \"kubernetes.io/projected/2c581508-933c-4786-8d50-90fa3d0ef4ae-kube-api-access-5tlrb\") pod \"maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:31.135459 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.135418    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-maas-api-tls\") pod \"maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:31.137899 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.137873    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-maas-api-tls\") pod \"maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:31.180647 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.180607    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"]
Sep 07 13:15:31.184238 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.184221    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"
Sep 07 13:15:31.186329 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.186306    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7d32a524-dockercfg-frc6m\""
Sep 07 13:15:31.186441 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.186307    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7d32a524\""
Sep 07 13:15:31.193365 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.193340    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"]
Sep 07 13:15:31.236230 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.236187    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc36d56b-f8be-46fe-aee9-b72c673dc619-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7\" (UID: \"cc36d56b-f8be-46fe-aee9-b72c673dc619\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"
Sep 07 13:15:31.236403 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.236248    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvqf8\" (UniqueName: \"kubernetes.io/projected/cc36d56b-f8be-46fe-aee9-b72c673dc619-kube-api-access-wvqf8\") pod \"payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7\" (UID: \"cc36d56b-f8be-46fe-aee9-b72c673dc619\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"
Sep 07 13:15:31.337083 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.337031    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc36d56b-f8be-46fe-aee9-b72c673dc619-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7\" (UID: \"cc36d56b-f8be-46fe-aee9-b72c673dc619\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"
Sep 07 13:15:31.337251 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.337103    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvqf8\" (UniqueName: \"kubernetes.io/projected/cc36d56b-f8be-46fe-aee9-b72c673dc619-kube-api-access-wvqf8\") pod \"payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7\" (UID: \"cc36d56b-f8be-46fe-aee9-b72c673dc619\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"
Sep 07 13:15:31.337974 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.337949    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc36d56b-f8be-46fe-aee9-b72c673dc619-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7\" (UID: \"cc36d56b-f8be-46fe-aee9-b72c673dc619\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"
Sep 07 13:15:31.351079 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.351036    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:31.356250 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.356229    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvqf8\" (UniqueName: \"kubernetes.io/projected/cc36d56b-f8be-46fe-aee9-b72c673dc619-kube-api-access-wvqf8\") pod \"payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7\" (UID: \"cc36d56b-f8be-46fe-aee9-b72c673dc619\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"
Sep 07 13:15:31.496440 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.496406    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"
Sep 07 13:15:31.642408 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.642374    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"]
Sep 07 13:15:31.769789 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.769762    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"]
Sep 07 13:15:31.770998 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:15:31.770960    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc36d56b_f8be_46fe_aee9_b72c673dc619.slice/crio-4e36459db8c53dfae90a768412b29aa4bfa95289caeae64a72dd0e0892c2bd91 WatchSource:0}: Error finding container 4e36459db8c53dfae90a768412b29aa4bfa95289caeae64a72dd0e0892c2bd91: Status 404 returned error can't find the container with id 4e36459db8c53dfae90a768412b29aa4bfa95289caeae64a72dd0e0892c2bd91
Sep 07 13:15:31.790796 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.790764    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws" event={"ID":"2c581508-933c-4786-8d50-90fa3d0ef4ae","Type":"ContainerStarted","Data":"f6a65873a7341be43a4ea3e1bc60e0c89399e83ce6ebb9507a6ad14b39bc59d4"}
Sep 07 13:15:31.792391 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:31.792354    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7" event={"ID":"cc36d56b-f8be-46fe-aee9-b72c673dc619","Type":"ContainerStarted","Data":"4e36459db8c53dfae90a768412b29aa4bfa95289caeae64a72dd0e0892c2bd91"}
Sep 07 13:15:32.799432 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:32.799386    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7" event={"ID":"cc36d56b-f8be-46fe-aee9-b72c673dc619","Type":"ContainerStarted","Data":"b1e56850cd94c487449eb07d58ceb17537572a706991dd6df301671c705084a1"}
Sep 07 13:15:32.799875 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:32.799454    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"
Sep 07 13:15:32.801238 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:32.801197    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws" event={"ID":"2c581508-933c-4786-8d50-90fa3d0ef4ae","Type":"ContainerStarted","Data":"ba9415a1083ed8973b2a39557037a4e18835e5f78b356dce15b03cd3e838b1a1"}
Sep 07 13:15:32.801357 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:32.801255    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:32.827938 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:32.827893    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7" podStartSLOduration=1.8278778020000002 podStartE2EDuration="1.827877802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:15:32.825687648 +0000 UTC m=+977.253985941" watchObservedRunningTime="2026-09-07 13:15:32.827877802 +0000 UTC m=+977.256176072"
Sep 07 13:15:32.851505 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:32.851447    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws" podStartSLOduration=2.260510088 podStartE2EDuration="2.851427708s" totalImagesPullingTime="590.91762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:15:32.849942796 +0000 UTC m=+977.278241069" watchObservedRunningTime="2026-09-07 13:15:32.851427708 +0000 UTC m=+977.279725982"
Sep 07 13:15:38.811772 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:38.811689    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:15:43.809420 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:15:43.809384    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"
Sep 07 13:16:01.250325 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.250289    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"]
Sep 07 13:16:01.250780 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.250507    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws" podUID="2c581508-933c-4786-8d50-90fa3d0ef4ae" containerName="maas-api" containerID="cri-o://ba9415a1083ed8973b2a39557037a4e18835e5f78b356dce15b03cd3e838b1a1" gracePeriod=30
Sep 07 13:16:01.357115 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.357082    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"]
Sep 07 13:16:01.357316 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.357293    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7" podUID="cc36d56b-f8be-46fe-aee9-b72c673dc619" containerName="payload-pre-processing" containerID="cri-o://b1e56850cd94c487449eb07d58ceb17537572a706991dd6df301671c705084a1" gracePeriod=30
Sep 07 13:16:01.495915 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.495887    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:16:01.525686 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.525652    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c581508-933c-4786-8d50-90fa3d0ef4ae-openshift-service-ca\") pod \"2c581508-933c-4786-8d50-90fa3d0ef4ae\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") "
Sep 07 13:16:01.525996 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.525975    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tlrb\" (UniqueName: \"kubernetes.io/projected/2c581508-933c-4786-8d50-90fa3d0ef4ae-kube-api-access-5tlrb\") pod \"2c581508-933c-4786-8d50-90fa3d0ef4ae\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") "
Sep 07 13:16:01.526309 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.526291    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-metrics-certs\") pod \"2c581508-933c-4786-8d50-90fa3d0ef4ae\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") "
Sep 07 13:16:01.526472 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.526455    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-maas-api-tls\") pod \"2c581508-933c-4786-8d50-90fa3d0ef4ae\" (UID: \"2c581508-933c-4786-8d50-90fa3d0ef4ae\") "
Sep 07 13:16:01.528077 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.527831    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c581508-933c-4786-8d50-90fa3d0ef4ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c581508-933c-4786-8d50-90fa3d0ef4ae" (UID: "2c581508-933c-4786-8d50-90fa3d0ef4ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:16:01.536943 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.536845    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c581508-933c-4786-8d50-90fa3d0ef4ae-kube-api-access-5tlrb" (OuterVolumeSpecName: "kube-api-access-5tlrb") pod "2c581508-933c-4786-8d50-90fa3d0ef4ae" (UID: "2c581508-933c-4786-8d50-90fa3d0ef4ae"). InnerVolumeSpecName "kube-api-access-5tlrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:16:01.537894 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.537854    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2c581508-933c-4786-8d50-90fa3d0ef4ae" (UID: "2c581508-933c-4786-8d50-90fa3d0ef4ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:16:01.538641 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.538605    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c581508-933c-4786-8d50-90fa3d0ef4ae" (UID: "2c581508-933c-4786-8d50-90fa3d0ef4ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:16:01.624905 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.624879    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"
Sep 07 13:16:01.628147 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.628126    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc36d56b-f8be-46fe-aee9-b72c673dc619-plugins-config-volume\") pod \"cc36d56b-f8be-46fe-aee9-b72c673dc619\" (UID: \"cc36d56b-f8be-46fe-aee9-b72c673dc619\") "
Sep 07 13:16:01.628244 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.628172    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvqf8\" (UniqueName: \"kubernetes.io/projected/cc36d56b-f8be-46fe-aee9-b72c673dc619-kube-api-access-wvqf8\") pod \"cc36d56b-f8be-46fe-aee9-b72c673dc619\" (UID: \"cc36d56b-f8be-46fe-aee9-b72c673dc619\") "
Sep 07 13:16:01.628486 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.628467    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:01.628486 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.628486    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c581508-933c-4786-8d50-90fa3d0ef4ae-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:01.628684 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.628500    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c581508-933c-4786-8d50-90fa3d0ef4ae-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:01.628684 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.628495    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc36d56b-f8be-46fe-aee9-b72c673dc619-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc36d56b-f8be-46fe-aee9-b72c673dc619" (UID: "cc36d56b-f8be-46fe-aee9-b72c673dc619"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:16:01.628684 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.628513    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tlrb\" (UniqueName: \"kubernetes.io/projected/2c581508-933c-4786-8d50-90fa3d0ef4ae-kube-api-access-5tlrb\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:01.630397 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.630371    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc36d56b-f8be-46fe-aee9-b72c673dc619-kube-api-access-wvqf8" (OuterVolumeSpecName: "kube-api-access-wvqf8") pod "cc36d56b-f8be-46fe-aee9-b72c673dc619" (UID: "cc36d56b-f8be-46fe-aee9-b72c673dc619"). InnerVolumeSpecName "kube-api-access-wvqf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:16:01.729785 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.729747    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc36d56b-f8be-46fe-aee9-b72c673dc619-plugins-config-volume\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:01.729785 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.729777    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvqf8\" (UniqueName: \"kubernetes.io/projected/cc36d56b-f8be-46fe-aee9-b72c673dc619-kube-api-access-wvqf8\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:01.987443 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.987345    2567 generic.go:356] "Generic (PLEG): container finished" podID="cc36d56b-f8be-46fe-aee9-b72c673dc619" containerID="b1e56850cd94c487449eb07d58ceb17537572a706991dd6df301671c705084a1" exitCode=0
Sep 07 13:16:01.987443 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.987400    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"
Sep 07 13:16:01.987668 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.987439    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7" event={"ID":"cc36d56b-f8be-46fe-aee9-b72c673dc619","Type":"ContainerDied","Data":"b1e56850cd94c487449eb07d58ceb17537572a706991dd6df301671c705084a1"}
Sep 07 13:16:01.987668 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.987519    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7" event={"ID":"cc36d56b-f8be-46fe-aee9-b72c673dc619","Type":"ContainerDied","Data":"4e36459db8c53dfae90a768412b29aa4bfa95289caeae64a72dd0e0892c2bd91"}
Sep 07 13:16:01.987668 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.987560    2567 scope.go:117] "RemoveContainer" containerID="b1e56850cd94c487449eb07d58ceb17537572a706991dd6df301671c705084a1"
Sep 07 13:16:01.989110 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.989086    2567 generic.go:356] "Generic (PLEG): container finished" podID="2c581508-933c-4786-8d50-90fa3d0ef4ae" containerID="ba9415a1083ed8973b2a39557037a4e18835e5f78b356dce15b03cd3e838b1a1" exitCode=0
Sep 07 13:16:01.989247 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.989130    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws" event={"ID":"2c581508-933c-4786-8d50-90fa3d0ef4ae","Type":"ContainerDied","Data":"ba9415a1083ed8973b2a39557037a4e18835e5f78b356dce15b03cd3e838b1a1"}
Sep 07 13:16:01.989247 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.989168    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"
Sep 07 13:16:01.989247 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.989181    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws" event={"ID":"2c581508-933c-4786-8d50-90fa3d0ef4ae","Type":"ContainerDied","Data":"f6a65873a7341be43a4ea3e1bc60e0c89399e83ce6ebb9507a6ad14b39bc59d4"}
Sep 07 13:16:01.997510 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.997494    2567 scope.go:117] "RemoveContainer" containerID="b1e56850cd94c487449eb07d58ceb17537572a706991dd6df301671c705084a1"
Sep 07 13:16:01.997792 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:16:01.997775    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e56850cd94c487449eb07d58ceb17537572a706991dd6df301671c705084a1\": container with ID starting with b1e56850cd94c487449eb07d58ceb17537572a706991dd6df301671c705084a1 not found: ID does not exist" containerID="b1e56850cd94c487449eb07d58ceb17537572a706991dd6df301671c705084a1"
Sep 07 13:16:01.997872 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.997803    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e56850cd94c487449eb07d58ceb17537572a706991dd6df301671c705084a1"} err="failed to get container status \"b1e56850cd94c487449eb07d58ceb17537572a706991dd6df301671c705084a1\": rpc error: code = NotFound desc = could not find container \"b1e56850cd94c487449eb07d58ceb17537572a706991dd6df301671c705084a1\": container with ID starting with b1e56850cd94c487449eb07d58ceb17537572a706991dd6df301671c705084a1 not found: ID does not exist"
Sep 07 13:16:01.997872 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:01.997823    2567 scope.go:117] "RemoveContainer" containerID="ba9415a1083ed8973b2a39557037a4e18835e5f78b356dce15b03cd3e838b1a1"
Sep 07 13:16:02.006742 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:02.006724    2567 scope.go:117] "RemoveContainer" containerID="ba9415a1083ed8973b2a39557037a4e18835e5f78b356dce15b03cd3e838b1a1"
Sep 07 13:16:02.006988 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:16:02.006970    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9415a1083ed8973b2a39557037a4e18835e5f78b356dce15b03cd3e838b1a1\": container with ID starting with ba9415a1083ed8973b2a39557037a4e18835e5f78b356dce15b03cd3e838b1a1 not found: ID does not exist" containerID="ba9415a1083ed8973b2a39557037a4e18835e5f78b356dce15b03cd3e838b1a1"
Sep 07 13:16:02.007029 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:02.006997    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9415a1083ed8973b2a39557037a4e18835e5f78b356dce15b03cd3e838b1a1"} err="failed to get container status \"ba9415a1083ed8973b2a39557037a4e18835e5f78b356dce15b03cd3e838b1a1\": rpc error: code = NotFound desc = could not find container \"ba9415a1083ed8973b2a39557037a4e18835e5f78b356dce15b03cd3e838b1a1\": container with ID starting with ba9415a1083ed8973b2a39557037a4e18835e5f78b356dce15b03cd3e838b1a1 not found: ID does not exist"
Sep 07 13:16:02.019522 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:02.019501    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"]
Sep 07 13:16:02.026448 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:02.026424    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d32a524-595fff4c99-wg4ws"]
Sep 07 13:16:02.043341 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:02.043319    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"]
Sep 07 13:16:02.049183 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:02.049163    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d32a524-747477dd8b-zgff7"]
Sep 07 13:16:02.075206 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:02.075177    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c581508-933c-4786-8d50-90fa3d0ef4ae" path="/var/lib/kubelet/pods/2c581508-933c-4786-8d50-90fa3d0ef4ae/volumes"
Sep 07 13:16:02.075569 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:02.075555    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc36d56b-f8be-46fe-aee9-b72c673dc619" path="/var/lib/kubelet/pods/cc36d56b-f8be-46fe-aee9-b72c673dc619/volumes"
Sep 07 13:16:07.480307 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.480265    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"]
Sep 07 13:16:07.480969 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.480942    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c581508-933c-4786-8d50-90fa3d0ef4ae" containerName="maas-api"
Sep 07 13:16:07.480969 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.480970    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c581508-933c-4786-8d50-90fa3d0ef4ae" containerName="maas-api"
Sep 07 13:16:07.481156 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.481003    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc36d56b-f8be-46fe-aee9-b72c673dc619" containerName="payload-pre-processing"
Sep 07 13:16:07.481156 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.481013    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc36d56b-f8be-46fe-aee9-b72c673dc619" containerName="payload-pre-processing"
Sep 07 13:16:07.481156 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.481141    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c581508-933c-4786-8d50-90fa3d0ef4ae" containerName="maas-api"
Sep 07 13:16:07.481262 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.481158    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc36d56b-f8be-46fe-aee9-b72c673dc619" containerName="payload-pre-processing"
Sep 07 13:16:07.486446 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.486423    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:07.489144 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.489112    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-bc6fa7f3-serving-cert\""
Sep 07 13:16:07.493037 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.492806    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"]
Sep 07 13:16:07.584755 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.584712    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-openshift-service-ca\") pod \"maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:07.584980 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.584787    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-metrics-certs\") pod \"maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:07.584980 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.584894    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-maas-api-tls\") pod \"maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:07.585133 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.584987    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95jgw\" (UniqueName: \"kubernetes.io/projected/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-kube-api-access-95jgw\") pod \"maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:07.686459 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.686424    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-openshift-service-ca\") pod \"maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:07.686661 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.686481    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-metrics-certs\") pod \"maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:07.686661 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.686530    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-maas-api-tls\") pod \"maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:07.686661 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.686590    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95jgw\" (UniqueName: \"kubernetes.io/projected/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-kube-api-access-95jgw\") pod \"maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:07.686829 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:16:07.686700    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-serving-cert: secret "maas-api-e2e-derive-bc6fa7f3-serving-cert" not found
Sep 07 13:16:07.686829 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:16:07.686807    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-maas-api-tls podName:68d22492-6aa8-4aa0-8aba-d08ced40a5e9 nodeName:}" failed. No retries permitted until 2026-09-07 13:16:08.186785264 +0000 UTC m=+1012.615083521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-maas-api-tls") pod "maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8" (UID: "68d22492-6aa8-4aa0-8aba-d08ced40a5e9") : secret "maas-api-e2e-derive-bc6fa7f3-serving-cert" not found
Sep 07 13:16:07.687464 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.687430    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-openshift-service-ca\") pod \"maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:07.689013 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.688981    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-metrics-certs\") pod \"maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:07.695018 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:07.694987    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95jgw\" (UniqueName: \"kubernetes.io/projected/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-kube-api-access-95jgw\") pod \"maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:08.171018 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.170977    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"]
Sep 07 13:16:08.174909 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.174886    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"
Sep 07 13:16:08.178766 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.178734    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-bc6fa7f3\""
Sep 07 13:16:08.178906 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.178819    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-bc6fa7f3-dockercfg-pjp47\""
Sep 07 13:16:08.185619 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.185596    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"]
Sep 07 13:16:08.191928 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.191899    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-maas-api-tls\") pod \"maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:08.194668 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.194645    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-maas-api-tls\") pod \"maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:08.292963 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.292925    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee05016-f823-41b2-8b8d-ecdb1bb0ce84-plugins-config-volume\") pod \"payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s\" (UID: \"bee05016-f823-41b2-8b8d-ecdb1bb0ce84\") " pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"
Sep 07 13:16:08.293165 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.293070    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf5m7\" (UniqueName: \"kubernetes.io/projected/bee05016-f823-41b2-8b8d-ecdb1bb0ce84-kube-api-access-nf5m7\") pod \"payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s\" (UID: \"bee05016-f823-41b2-8b8d-ecdb1bb0ce84\") " pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"
Sep 07 13:16:08.394461 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.394424    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf5m7\" (UniqueName: \"kubernetes.io/projected/bee05016-f823-41b2-8b8d-ecdb1bb0ce84-kube-api-access-nf5m7\") pod \"payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s\" (UID: \"bee05016-f823-41b2-8b8d-ecdb1bb0ce84\") " pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"
Sep 07 13:16:08.394648 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.394526    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee05016-f823-41b2-8b8d-ecdb1bb0ce84-plugins-config-volume\") pod \"payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s\" (UID: \"bee05016-f823-41b2-8b8d-ecdb1bb0ce84\") " pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"
Sep 07 13:16:08.395294 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.395268    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee05016-f823-41b2-8b8d-ecdb1bb0ce84-plugins-config-volume\") pod \"payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s\" (UID: \"bee05016-f823-41b2-8b8d-ecdb1bb0ce84\") " pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"
Sep 07 13:16:08.403363 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.403337    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:08.408252 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.408224    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf5m7\" (UniqueName: \"kubernetes.io/projected/bee05016-f823-41b2-8b8d-ecdb1bb0ce84-kube-api-access-nf5m7\") pod \"payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s\" (UID: \"bee05016-f823-41b2-8b8d-ecdb1bb0ce84\") " pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"
Sep 07 13:16:08.489303 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.489250    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"
Sep 07 13:16:08.682423 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.682322    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"]
Sep 07 13:16:08.682695 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:16:08.682665    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d22492_6aa8_4aa0_8aba_d08ced40a5e9.slice/crio-f0a0c7e4dafed75ea44a3a4faac5554c838b2daec9620e2a89a81d4889a00291 WatchSource:0}: Error finding container f0a0c7e4dafed75ea44a3a4faac5554c838b2daec9620e2a89a81d4889a00291: Status 404 returned error can't find the container with id f0a0c7e4dafed75ea44a3a4faac5554c838b2daec9620e2a89a81d4889a00291
Sep 07 13:16:08.683805 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.683781    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 13:16:08.763287 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:08.763261    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"]
Sep 07 13:16:08.764976 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:16:08.764940    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee05016_f823_41b2_8b8d_ecdb1bb0ce84.slice/crio-d39ccdd592c93204e2a28f8d93874446986c926e223a464a8c0b9e681393e8ad WatchSource:0}: Error finding container d39ccdd592c93204e2a28f8d93874446986c926e223a464a8c0b9e681393e8ad: Status 404 returned error can't find the container with id d39ccdd592c93204e2a28f8d93874446986c926e223a464a8c0b9e681393e8ad
Sep 07 13:16:09.037723 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:09.037682    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8" event={"ID":"68d22492-6aa8-4aa0-8aba-d08ced40a5e9","Type":"ContainerStarted","Data":"f0a0c7e4dafed75ea44a3a4faac5554c838b2daec9620e2a89a81d4889a00291"}
Sep 07 13:16:09.039498 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:09.039463    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s" event={"ID":"bee05016-f823-41b2-8b8d-ecdb1bb0ce84","Type":"ContainerStarted","Data":"91ea91aa231cd6b43c5acb51118aaaa7038f648ad18d3b1f0b6489935387f191"}
Sep 07 13:16:09.039672 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:09.039514    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s" event={"ID":"bee05016-f823-41b2-8b8d-ecdb1bb0ce84","Type":"ContainerStarted","Data":"d39ccdd592c93204e2a28f8d93874446986c926e223a464a8c0b9e681393e8ad"}
Sep 07 13:16:09.039672 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:09.039545    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"
Sep 07 13:16:09.060405 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:09.060339    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s" podStartSLOduration=1.060321955 podStartE2EDuration="1.060321955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:16:09.05756243 +0000 UTC m=+1013.485860728" watchObservedRunningTime="2026-09-07 13:16:09.060321955 +0000 UTC m=+1013.488620258"
Sep 07 13:16:10.046540 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:10.046501    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8" event={"ID":"68d22492-6aa8-4aa0-8aba-d08ced40a5e9","Type":"ContainerStarted","Data":"3d275ee3a1980eec6c328b2a4baff3ff6f837477c50011e4d68184a4c66e2050"}
Sep 07 13:16:10.046945 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:10.046597    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:10.067402 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:10.067351    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8" podStartSLOduration=2.3229646170000002 podStartE2EDuration="3.067334102s" totalImagesPullingTime="744.369485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:16:10.065550837 +0000 UTC m=+1014.493849110" watchObservedRunningTime="2026-09-07 13:16:10.067334102 +0000 UTC m=+1014.495632375"
Sep 07 13:16:16.057820 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:16.057790    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:20.049175 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:20.049130    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"
Sep 07 13:16:25.355303 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.355261    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"]
Sep 07 13:16:25.355804 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.355541    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw" podUID="dbd02188-3e10-4701-8805-02a77ab50447" containerName="maas-api" containerID="cri-o://65bb10804f78c7c6edd40bc5bcd9147a6360a692b2295d137f02492a762a3d79" gracePeriod=30
Sep 07 13:16:25.425418 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.425379    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"]
Sep 07 13:16:25.425661 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.425631    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll" podUID="966d0a64-7ed0-4cab-9e32-febcd7a04bdc" containerName="payload-pre-processing" containerID="cri-o://3967b55e35cae889cbf12e77e76a9f271986f88b51b9c6be4c1b36281e9d1b84" gracePeriod=30
Sep 07 13:16:25.747289 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.747262    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:16:25.754246 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.754220    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"
Sep 07 13:16:25.867822 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.867791    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966d0a64-7ed0-4cab-9e32-febcd7a04bdc-plugins-config-volume\") pod \"966d0a64-7ed0-4cab-9e32-febcd7a04bdc\" (UID: \"966d0a64-7ed0-4cab-9e32-febcd7a04bdc\") "
Sep 07 13:16:25.868007 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.867897    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-maas-api-tls\") pod \"dbd02188-3e10-4701-8805-02a77ab50447\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") "
Sep 07 13:16:25.868007 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.867964    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84xj6\" (UniqueName: \"kubernetes.io/projected/dbd02188-3e10-4701-8805-02a77ab50447-kube-api-access-84xj6\") pod \"dbd02188-3e10-4701-8805-02a77ab50447\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") "
Sep 07 13:16:25.868007 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.867994    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-metrics-certs\") pod \"dbd02188-3e10-4701-8805-02a77ab50447\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") "
Sep 07 13:16:25.868170 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.868031    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zhm9\" (UniqueName: \"kubernetes.io/projected/966d0a64-7ed0-4cab-9e32-febcd7a04bdc-kube-api-access-2zhm9\") pod \"966d0a64-7ed0-4cab-9e32-febcd7a04bdc\" (UID: \"966d0a64-7ed0-4cab-9e32-febcd7a04bdc\") "
Sep 07 13:16:25.868170 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.868092    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd02188-3e10-4701-8805-02a77ab50447-openshift-service-ca\") pod \"dbd02188-3e10-4701-8805-02a77ab50447\" (UID: \"dbd02188-3e10-4701-8805-02a77ab50447\") "
Sep 07 13:16:25.868306 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.868281    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966d0a64-7ed0-4cab-9e32-febcd7a04bdc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "966d0a64-7ed0-4cab-9e32-febcd7a04bdc" (UID: "966d0a64-7ed0-4cab-9e32-febcd7a04bdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:16:25.868477 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.868452    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966d0a64-7ed0-4cab-9e32-febcd7a04bdc-plugins-config-volume\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:25.868568 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.868534    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd02188-3e10-4701-8805-02a77ab50447-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dbd02188-3e10-4701-8805-02a77ab50447" (UID: "dbd02188-3e10-4701-8805-02a77ab50447"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:16:25.870268 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.870235    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dbd02188-3e10-4701-8805-02a77ab50447" (UID: "dbd02188-3e10-4701-8805-02a77ab50447"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:16:25.870268 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.870257    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dbd02188-3e10-4701-8805-02a77ab50447" (UID: "dbd02188-3e10-4701-8805-02a77ab50447"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:16:25.870548 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.870527    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd02188-3e10-4701-8805-02a77ab50447-kube-api-access-84xj6" (OuterVolumeSpecName: "kube-api-access-84xj6") pod "dbd02188-3e10-4701-8805-02a77ab50447" (UID: "dbd02188-3e10-4701-8805-02a77ab50447"). InnerVolumeSpecName "kube-api-access-84xj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:16:25.870591 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.870557    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966d0a64-7ed0-4cab-9e32-febcd7a04bdc-kube-api-access-2zhm9" (OuterVolumeSpecName: "kube-api-access-2zhm9") pod "966d0a64-7ed0-4cab-9e32-febcd7a04bdc" (UID: "966d0a64-7ed0-4cab-9e32-febcd7a04bdc"). InnerVolumeSpecName "kube-api-access-2zhm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:16:25.969518 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.969418    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:25.969518 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.969458    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84xj6\" (UniqueName: \"kubernetes.io/projected/dbd02188-3e10-4701-8805-02a77ab50447-kube-api-access-84xj6\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:25.969518 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.969471    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbd02188-3e10-4701-8805-02a77ab50447-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:25.969518 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.969481    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zhm9\" (UniqueName: \"kubernetes.io/projected/966d0a64-7ed0-4cab-9e32-febcd7a04bdc-kube-api-access-2zhm9\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:25.969518 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:25.969489    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd02188-3e10-4701-8805-02a77ab50447-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:26.148822 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.148785    2567 generic.go:356] "Generic (PLEG): container finished" podID="966d0a64-7ed0-4cab-9e32-febcd7a04bdc" containerID="3967b55e35cae889cbf12e77e76a9f271986f88b51b9c6be4c1b36281e9d1b84" exitCode=0
Sep 07 13:16:26.148990 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.148844    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"
Sep 07 13:16:26.148990 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.148872    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll" event={"ID":"966d0a64-7ed0-4cab-9e32-febcd7a04bdc","Type":"ContainerDied","Data":"3967b55e35cae889cbf12e77e76a9f271986f88b51b9c6be4c1b36281e9d1b84"}
Sep 07 13:16:26.148990 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.148939    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll" event={"ID":"966d0a64-7ed0-4cab-9e32-febcd7a04bdc","Type":"ContainerDied","Data":"e6d474afca1cb4a91e844ae6666796683c6c36ad7add6f01b6692015743f7d36"}
Sep 07 13:16:26.148990 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.148960    2567 scope.go:117] "RemoveContainer" containerID="3967b55e35cae889cbf12e77e76a9f271986f88b51b9c6be4c1b36281e9d1b84"
Sep 07 13:16:26.150626 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.150596    2567 generic.go:356] "Generic (PLEG): container finished" podID="dbd02188-3e10-4701-8805-02a77ab50447" containerID="65bb10804f78c7c6edd40bc5bcd9147a6360a692b2295d137f02492a762a3d79" exitCode=0
Sep 07 13:16:26.150754 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.150658    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"
Sep 07 13:16:26.150754 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.150690    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw" event={"ID":"dbd02188-3e10-4701-8805-02a77ab50447","Type":"ContainerDied","Data":"65bb10804f78c7c6edd40bc5bcd9147a6360a692b2295d137f02492a762a3d79"}
Sep 07 13:16:26.150754 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.150746    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw" event={"ID":"dbd02188-3e10-4701-8805-02a77ab50447","Type":"ContainerDied","Data":"2176609474adbd0ea32d9d1ab076123f90e57dbabcb05814d58973bc8e22788a"}
Sep 07 13:16:26.158438 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.158423    2567 scope.go:117] "RemoveContainer" containerID="3967b55e35cae889cbf12e77e76a9f271986f88b51b9c6be4c1b36281e9d1b84"
Sep 07 13:16:26.158735 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:16:26.158714    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3967b55e35cae889cbf12e77e76a9f271986f88b51b9c6be4c1b36281e9d1b84\": container with ID starting with 3967b55e35cae889cbf12e77e76a9f271986f88b51b9c6be4c1b36281e9d1b84 not found: ID does not exist" containerID="3967b55e35cae889cbf12e77e76a9f271986f88b51b9c6be4c1b36281e9d1b84"
Sep 07 13:16:26.158799 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.158747    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3967b55e35cae889cbf12e77e76a9f271986f88b51b9c6be4c1b36281e9d1b84"} err="failed to get container status \"3967b55e35cae889cbf12e77e76a9f271986f88b51b9c6be4c1b36281e9d1b84\": rpc error: code = NotFound desc = could not find container \"3967b55e35cae889cbf12e77e76a9f271986f88b51b9c6be4c1b36281e9d1b84\": container with ID starting with 3967b55e35cae889cbf12e77e76a9f271986f88b51b9c6be4c1b36281e9d1b84 not found: ID does not exist"
Sep 07 13:16:26.158799 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.158766    2567 scope.go:117] "RemoveContainer" containerID="65bb10804f78c7c6edd40bc5bcd9147a6360a692b2295d137f02492a762a3d79"
Sep 07 13:16:26.167661 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.167639    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"]
Sep 07 13:16:26.168412 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.168399    2567 scope.go:117] "RemoveContainer" containerID="65bb10804f78c7c6edd40bc5bcd9147a6360a692b2295d137f02492a762a3d79"
Sep 07 13:16:26.168701 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:16:26.168681    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65bb10804f78c7c6edd40bc5bcd9147a6360a692b2295d137f02492a762a3d79\": container with ID starting with 65bb10804f78c7c6edd40bc5bcd9147a6360a692b2295d137f02492a762a3d79 not found: ID does not exist" containerID="65bb10804f78c7c6edd40bc5bcd9147a6360a692b2295d137f02492a762a3d79"
Sep 07 13:16:26.168764 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.168714    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65bb10804f78c7c6edd40bc5bcd9147a6360a692b2295d137f02492a762a3d79"} err="failed to get container status \"65bb10804f78c7c6edd40bc5bcd9147a6360a692b2295d137f02492a762a3d79\": rpc error: code = NotFound desc = could not find container \"65bb10804f78c7c6edd40bc5bcd9147a6360a692b2295d137f02492a762a3d79\": container with ID starting with 65bb10804f78c7c6edd40bc5bcd9147a6360a692b2295d137f02492a762a3d79 not found: ID does not exist"
Sep 07 13:16:26.174084 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.174049    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-adacb1-846dd575f8-txcll"]
Sep 07 13:16:26.184148 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.184125    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"]
Sep 07 13:16:26.187721 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:26.187699    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-adacb1-7fc55bbb4d-c8nnw"]
Sep 07 13:16:28.077869 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:28.077820    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="966d0a64-7ed0-4cab-9e32-febcd7a04bdc" path="/var/lib/kubelet/pods/966d0a64-7ed0-4cab-9e32-febcd7a04bdc/volumes"
Sep 07 13:16:28.078638 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:28.078613    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbd02188-3e10-4701-8805-02a77ab50447" path="/var/lib/kubelet/pods/dbd02188-3e10-4701-8805-02a77ab50447/volumes"
Sep 07 13:16:37.467132 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.467082    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"]
Sep 07 13:16:37.467787 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.467761    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd02188-3e10-4701-8805-02a77ab50447" containerName="maas-api"
Sep 07 13:16:37.467857 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.467792    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd02188-3e10-4701-8805-02a77ab50447" containerName="maas-api"
Sep 07 13:16:37.467857 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.467811    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966d0a64-7ed0-4cab-9e32-febcd7a04bdc" containerName="payload-pre-processing"
Sep 07 13:16:37.467857 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.467820    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="966d0a64-7ed0-4cab-9e32-febcd7a04bdc" containerName="payload-pre-processing"
Sep 07 13:16:37.468017 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.467944    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd02188-3e10-4701-8805-02a77ab50447" containerName="maas-api"
Sep 07 13:16:37.468017 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.467961    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="966d0a64-7ed0-4cab-9e32-febcd7a04bdc" containerName="payload-pre-processing"
Sep 07 13:16:37.473086 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.473042    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:37.475735 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.475708    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-f5e4c4-serving-cert\""
Sep 07 13:16:37.495682 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.495651    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"]
Sep 07 13:16:37.587367 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.587320    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a9e421-c525-43df-8bf6-a42110b293a4-openshift-service-ca\") pod \"maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:37.587631 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.587402    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a9e421-c525-43df-8bf6-a42110b293a4-maas-api-tls\") pod \"maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:37.587631 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.587485    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55t68\" (UniqueName: \"kubernetes.io/projected/16a9e421-c525-43df-8bf6-a42110b293a4-kube-api-access-55t68\") pod \"maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:37.587631 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.587529    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a9e421-c525-43df-8bf6-a42110b293a4-metrics-certs\") pod \"maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:37.688880 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.688840    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a9e421-c525-43df-8bf6-a42110b293a4-openshift-service-ca\") pod \"maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:37.689133 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.688909    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a9e421-c525-43df-8bf6-a42110b293a4-maas-api-tls\") pod \"maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:37.689133 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.688966    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55t68\" (UniqueName: \"kubernetes.io/projected/16a9e421-c525-43df-8bf6-a42110b293a4-kube-api-access-55t68\") pod \"maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:37.689133 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.689004    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a9e421-c525-43df-8bf6-a42110b293a4-metrics-certs\") pod \"maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:37.689592 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.689567    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a9e421-c525-43df-8bf6-a42110b293a4-openshift-service-ca\") pod \"maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:37.691602 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.691570    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a9e421-c525-43df-8bf6-a42110b293a4-metrics-certs\") pod \"maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:37.691699 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.691614    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a9e421-c525-43df-8bf6-a42110b293a4-maas-api-tls\") pod \"maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:37.699412 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.699384    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55t68\" (UniqueName: \"kubernetes.io/projected/16a9e421-c525-43df-8bf6-a42110b293a4-kube-api-access-55t68\") pod \"maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:37.786025 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:37.785990    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:38.079856 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:38.079823    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"]
Sep 07 13:16:38.081544 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:16:38.081498    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a9e421_c525_43df_8bf6_a42110b293a4.slice/crio-0ad2cca883d15f26a03442dea606abc451ca585bd1af94ae9f14658dc7d62208 WatchSource:0}: Error finding container 0ad2cca883d15f26a03442dea606abc451ca585bd1af94ae9f14658dc7d62208: Status 404 returned error can't find the container with id 0ad2cca883d15f26a03442dea606abc451ca585bd1af94ae9f14658dc7d62208
Sep 07 13:16:38.226854 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:38.226817    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm" event={"ID":"16a9e421-c525-43df-8bf6-a42110b293a4","Type":"ContainerStarted","Data":"0ad2cca883d15f26a03442dea606abc451ca585bd1af94ae9f14658dc7d62208"}
Sep 07 13:16:38.515666 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:38.514353    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"]
Sep 07 13:16:38.515666 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:38.514589    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8" podUID="68d22492-6aa8-4aa0-8aba-d08ced40a5e9" containerName="maas-api" containerID="cri-o://3d275ee3a1980eec6c328b2a4baff3ff6f837477c50011e4d68184a4c66e2050" gracePeriod=30
Sep 07 13:16:38.584002 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:38.583961    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"]
Sep 07 13:16:38.584479 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:38.584438    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s" podUID="bee05016-f823-41b2-8b8d-ecdb1bb0ce84" containerName="payload-processing" containerID="cri-o://91ea91aa231cd6b43c5acb51118aaaa7038f648ad18d3b1f0b6489935387f191" gracePeriod=30
Sep 07 13:16:38.859234 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:38.859210    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:38.897026 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:38.896995    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"
Sep 07 13:16:39.000801 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.000762    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-metrics-certs\") pod \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") "
Sep 07 13:16:39.000801 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.000804    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-maas-api-tls\") pod \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") "
Sep 07 13:16:39.001055 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.000830    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf5m7\" (UniqueName: \"kubernetes.io/projected/bee05016-f823-41b2-8b8d-ecdb1bb0ce84-kube-api-access-nf5m7\") pod \"bee05016-f823-41b2-8b8d-ecdb1bb0ce84\" (UID: \"bee05016-f823-41b2-8b8d-ecdb1bb0ce84\") "
Sep 07 13:16:39.001055 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.000874    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee05016-f823-41b2-8b8d-ecdb1bb0ce84-plugins-config-volume\") pod \"bee05016-f823-41b2-8b8d-ecdb1bb0ce84\" (UID: \"bee05016-f823-41b2-8b8d-ecdb1bb0ce84\") "
Sep 07 13:16:39.001055 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.000942    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-openshift-service-ca\") pod \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") "
Sep 07 13:16:39.001055 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.000978    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95jgw\" (UniqueName: \"kubernetes.io/projected/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-kube-api-access-95jgw\") pod \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\" (UID: \"68d22492-6aa8-4aa0-8aba-d08ced40a5e9\") "
Sep 07 13:16:39.001322 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.001276    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee05016-f823-41b2-8b8d-ecdb1bb0ce84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bee05016-f823-41b2-8b8d-ecdb1bb0ce84" (UID: "bee05016-f823-41b2-8b8d-ecdb1bb0ce84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:16:39.001393 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.001339    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee05016-f823-41b2-8b8d-ecdb1bb0ce84-plugins-config-volume\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:39.001393 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.001360    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68d22492-6aa8-4aa0-8aba-d08ced40a5e9" (UID: "68d22492-6aa8-4aa0-8aba-d08ced40a5e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:16:39.003175 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.003144    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68d22492-6aa8-4aa0-8aba-d08ced40a5e9" (UID: "68d22492-6aa8-4aa0-8aba-d08ced40a5e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:16:39.003175 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.003157    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "68d22492-6aa8-4aa0-8aba-d08ced40a5e9" (UID: "68d22492-6aa8-4aa0-8aba-d08ced40a5e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:16:39.003326 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.003224    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-kube-api-access-95jgw" (OuterVolumeSpecName: "kube-api-access-95jgw") pod "68d22492-6aa8-4aa0-8aba-d08ced40a5e9" (UID: "68d22492-6aa8-4aa0-8aba-d08ced40a5e9"). InnerVolumeSpecName "kube-api-access-95jgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:16:39.003326 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.003253    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee05016-f823-41b2-8b8d-ecdb1bb0ce84-kube-api-access-nf5m7" (OuterVolumeSpecName: "kube-api-access-nf5m7") pod "bee05016-f823-41b2-8b8d-ecdb1bb0ce84" (UID: "bee05016-f823-41b2-8b8d-ecdb1bb0ce84"). InnerVolumeSpecName "kube-api-access-nf5m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:16:39.101852 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.101816    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:39.101852 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.101844    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:39.101852 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.101853    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf5m7\" (UniqueName: \"kubernetes.io/projected/bee05016-f823-41b2-8b8d-ecdb1bb0ce84-kube-api-access-nf5m7\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:39.102225 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.101863    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:39.102225 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.101875    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95jgw\" (UniqueName: \"kubernetes.io/projected/68d22492-6aa8-4aa0-8aba-d08ced40a5e9-kube-api-access-95jgw\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:16:39.237254 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.237130    2567 generic.go:356] "Generic (PLEG): container finished" podID="68d22492-6aa8-4aa0-8aba-d08ced40a5e9" containerID="3d275ee3a1980eec6c328b2a4baff3ff6f837477c50011e4d68184a4c66e2050" exitCode=0
Sep 07 13:16:39.237254 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.237252    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"
Sep 07 13:16:39.237515 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.237294    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8" event={"ID":"68d22492-6aa8-4aa0-8aba-d08ced40a5e9","Type":"ContainerDied","Data":"3d275ee3a1980eec6c328b2a4baff3ff6f837477c50011e4d68184a4c66e2050"}
Sep 07 13:16:39.237515 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.237358    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8" event={"ID":"68d22492-6aa8-4aa0-8aba-d08ced40a5e9","Type":"ContainerDied","Data":"f0a0c7e4dafed75ea44a3a4faac5554c838b2daec9620e2a89a81d4889a00291"}
Sep 07 13:16:39.237515 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.237390    2567 scope.go:117] "RemoveContainer" containerID="3d275ee3a1980eec6c328b2a4baff3ff6f837477c50011e4d68184a4c66e2050"
Sep 07 13:16:39.239267 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.239239    2567 generic.go:356] "Generic (PLEG): container finished" podID="bee05016-f823-41b2-8b8d-ecdb1bb0ce84" containerID="91ea91aa231cd6b43c5acb51118aaaa7038f648ad18d3b1f0b6489935387f191" exitCode=0
Sep 07 13:16:39.239399 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.239297    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"
Sep 07 13:16:39.239399 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.239328    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s" event={"ID":"bee05016-f823-41b2-8b8d-ecdb1bb0ce84","Type":"ContainerDied","Data":"91ea91aa231cd6b43c5acb51118aaaa7038f648ad18d3b1f0b6489935387f191"}
Sep 07 13:16:39.239399 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.239372    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s" event={"ID":"bee05016-f823-41b2-8b8d-ecdb1bb0ce84","Type":"ContainerDied","Data":"d39ccdd592c93204e2a28f8d93874446986c926e223a464a8c0b9e681393e8ad"}
Sep 07 13:16:39.241388 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.241362    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm" event={"ID":"16a9e421-c525-43df-8bf6-a42110b293a4","Type":"ContainerStarted","Data":"4fb9a2efb32cbbbcbf462ab3963748d3d57cb95334afee2f8d940879090a99f1"}
Sep 07 13:16:39.241512 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.241460    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:16:39.250439 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.250407    2567 scope.go:117] "RemoveContainer" containerID="3d275ee3a1980eec6c328b2a4baff3ff6f837477c50011e4d68184a4c66e2050"
Sep 07 13:16:39.250797 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:16:39.250773    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d275ee3a1980eec6c328b2a4baff3ff6f837477c50011e4d68184a4c66e2050\": container with ID starting with 3d275ee3a1980eec6c328b2a4baff3ff6f837477c50011e4d68184a4c66e2050 not found: ID does not exist" containerID="3d275ee3a1980eec6c328b2a4baff3ff6f837477c50011e4d68184a4c66e2050"
Sep 07 13:16:39.250881 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.250807    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d275ee3a1980eec6c328b2a4baff3ff6f837477c50011e4d68184a4c66e2050"} err="failed to get container status \"3d275ee3a1980eec6c328b2a4baff3ff6f837477c50011e4d68184a4c66e2050\": rpc error: code = NotFound desc = could not find container \"3d275ee3a1980eec6c328b2a4baff3ff6f837477c50011e4d68184a4c66e2050\": container with ID starting with 3d275ee3a1980eec6c328b2a4baff3ff6f837477c50011e4d68184a4c66e2050 not found: ID does not exist"
Sep 07 13:16:39.250881 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.250834    2567 scope.go:117] "RemoveContainer" containerID="91ea91aa231cd6b43c5acb51118aaaa7038f648ad18d3b1f0b6489935387f191"
Sep 07 13:16:39.261214 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.261186    2567 scope.go:117] "RemoveContainer" containerID="91ea91aa231cd6b43c5acb51118aaaa7038f648ad18d3b1f0b6489935387f191"
Sep 07 13:16:39.261506 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:16:39.261488    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ea91aa231cd6b43c5acb51118aaaa7038f648ad18d3b1f0b6489935387f191\": container with ID starting with 91ea91aa231cd6b43c5acb51118aaaa7038f648ad18d3b1f0b6489935387f191 not found: ID does not exist" containerID="91ea91aa231cd6b43c5acb51118aaaa7038f648ad18d3b1f0b6489935387f191"
Sep 07 13:16:39.261560 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.261514    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ea91aa231cd6b43c5acb51118aaaa7038f648ad18d3b1f0b6489935387f191"} err="failed to get container status \"91ea91aa231cd6b43c5acb51118aaaa7038f648ad18d3b1f0b6489935387f191\": rpc error: code = NotFound desc = could not find container \"91ea91aa231cd6b43c5acb51118aaaa7038f648ad18d3b1f0b6489935387f191\": container with ID starting with 91ea91aa231cd6b43c5acb51118aaaa7038f648ad18d3b1f0b6489935387f191 not found: ID does not exist"
Sep 07 13:16:39.298767 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.298712    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm" podStartSLOduration=1.880132008 podStartE2EDuration="2.29869256s" totalImagesPullingTime="418.560552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:16:39.296726932 +0000 UTC m=+1043.725025218" watchObservedRunningTime="2026-09-07 13:16:39.29869256 +0000 UTC m=+1043.726990833"
Sep 07 13:16:39.322638 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.322606    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"]
Sep 07 13:16:39.331124 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.331088    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc6fa7f3-db5bdf6f7-75g8s"]
Sep 07 13:16:39.343092 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.343051    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"]
Sep 07 13:16:39.347770 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:39.347740    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc6fa7f3-fc9687dcb-z8lc8"]
Sep 07 13:16:40.076888 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:40.076849    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68d22492-6aa8-4aa0-8aba-d08ced40a5e9" path="/var/lib/kubelet/pods/68d22492-6aa8-4aa0-8aba-d08ced40a5e9/volumes"
Sep 07 13:16:40.077311 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:40.077292    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bee05016-f823-41b2-8b8d-ecdb1bb0ce84" path="/var/lib/kubelet/pods/bee05016-f823-41b2-8b8d-ecdb1bb0ce84/volumes"
Sep 07 13:16:45.255204 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:16:45.255171    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:17:35.268255 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.268223    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"]
Sep 07 13:17:35.268701 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.268627    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee05016-f823-41b2-8b8d-ecdb1bb0ce84" containerName="payload-processing"
Sep 07 13:17:35.268701 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.268640    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee05016-f823-41b2-8b8d-ecdb1bb0ce84" containerName="payload-processing"
Sep 07 13:17:35.268701 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.268655    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d22492-6aa8-4aa0-8aba-d08ced40a5e9" containerName="maas-api"
Sep 07 13:17:35.268701 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.268661    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d22492-6aa8-4aa0-8aba-d08ced40a5e9" containerName="maas-api"
Sep 07 13:17:35.268836 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.268740    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee05016-f823-41b2-8b8d-ecdb1bb0ce84" containerName="payload-processing"
Sep 07 13:17:35.268836 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.268752    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d22492-6aa8-4aa0-8aba-d08ced40a5e9" containerName="maas-api"
Sep 07 13:17:35.273681 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.273655    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.282984 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.282951    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f5e4c4\"/\"openshift-service-ca.crt\""
Sep 07 13:17:35.283299 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.283281    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f5e4c4\"/\"explicit-model-f5e4c4-kserve-self-signed-certs\""
Sep 07 13:17:35.283400 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.283340    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f5e4c4\"/\"kube-root-ca.crt\""
Sep 07 13:17:35.283607 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.283590    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f5e4c4\"/\"default-dockercfg-v6mg5\""
Sep 07 13:17:35.292896 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.292866    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"]
Sep 07 13:17:35.318449 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.318417    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-home\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.318449 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.318452    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-dshm\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.318622 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.318533    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-model-cache\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.318622 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.318608    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd254cab-b58d-4e73-b511-3b05e5c81849-tls-certs\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.318700 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.318628    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnq82\" (UniqueName: \"kubernetes.io/projected/dd254cab-b58d-4e73-b511-3b05e5c81849-kube-api-access-bnq82\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.318700 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.318668    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-tmp-dir\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.420172 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.420135    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd254cab-b58d-4e73-b511-3b05e5c81849-tls-certs\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.420172 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.420175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnq82\" (UniqueName: \"kubernetes.io/projected/dd254cab-b58d-4e73-b511-3b05e5c81849-kube-api-access-bnq82\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.420401 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.420211    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-tmp-dir\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.420401 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.420273    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-home\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.420401 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.420287    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-dshm\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.420401 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.420306    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-model-cache\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.420731 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.420702    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-home\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.420870 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.420735    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-tmp-dir\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.420870 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.420766    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-model-cache\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.422582 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.422557    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-dshm\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.422880 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.422851    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd254cab-b58d-4e73-b511-3b05e5c81849-tls-certs\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.429932 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.429908    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnq82\" (UniqueName: \"kubernetes.io/projected/dd254cab-b58d-4e73-b511-3b05e5c81849-kube-api-access-bnq82\") pod \"explicit-model-f5e4c4-kserve-659799c5cf-rqk5m\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") " pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.582843 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.582734    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:35.868769 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:35.868743    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"]
Sep 07 13:17:35.870825 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:17:35.870789    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd254cab_b58d_4e73_b511_3b05e5c81849.slice/crio-c385f6b0c75e74f2398e835da6f66b1bb7a843dff8c176fdd7d9d53af7f78048 WatchSource:0}: Error finding container c385f6b0c75e74f2398e835da6f66b1bb7a843dff8c176fdd7d9d53af7f78048: Status 404 returned error can't find the container with id c385f6b0c75e74f2398e835da6f66b1bb7a843dff8c176fdd7d9d53af7f78048
Sep 07 13:17:36.571973 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:36.571927    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m" event={"ID":"dd254cab-b58d-4e73-b511-3b05e5c81849","Type":"ContainerStarted","Data":"f56279d831f38adac9551559c9cd20053090403edf6ae902de10b447eb1deb3b"}
Sep 07 13:17:36.572392 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:36.571983    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m" event={"ID":"dd254cab-b58d-4e73-b511-3b05e5c81849","Type":"ContainerStarted","Data":"c385f6b0c75e74f2398e835da6f66b1bb7a843dff8c176fdd7d9d53af7f78048"}
Sep 07 13:17:36.596933 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:36.596880    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m" podStartSLOduration=1.596863018 podStartE2EDuration="1.596863018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:17:36.595745408 +0000 UTC m=+1101.024043681" watchObservedRunningTime="2026-09-07 13:17:36.596863018 +0000 UTC m=+1101.025161575"
Sep 07 13:17:45.583508 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:45.583458    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:45.584016 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:45.583522    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:45.596205 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:45.596173    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:45.634282 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:45.634252    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:17:53.339604 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:17:53.339573    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"]
Sep 07 13:18:04.046388 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.046349    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"]
Sep 07 13:18:04.047324 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.046605    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm" podUID="16a9e421-c525-43df-8bf6-a42110b293a4" containerName="maas-api" containerID="cri-o://4fb9a2efb32cbbbcbf462ab3963748d3d57cb95334afee2f8d940879090a99f1" gracePeriod=30
Sep 07 13:18:04.321722 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.321681    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:18:04.406979 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.406935    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a9e421-c525-43df-8bf6-a42110b293a4-metrics-certs\") pod \"16a9e421-c525-43df-8bf6-a42110b293a4\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") "
Sep 07 13:18:04.407196 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.407028    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a9e421-c525-43df-8bf6-a42110b293a4-maas-api-tls\") pod \"16a9e421-c525-43df-8bf6-a42110b293a4\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") "
Sep 07 13:18:04.407272 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.407189    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55t68\" (UniqueName: \"kubernetes.io/projected/16a9e421-c525-43df-8bf6-a42110b293a4-kube-api-access-55t68\") pod \"16a9e421-c525-43df-8bf6-a42110b293a4\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") "
Sep 07 13:18:04.407335 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.407275    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a9e421-c525-43df-8bf6-a42110b293a4-openshift-service-ca\") pod \"16a9e421-c525-43df-8bf6-a42110b293a4\" (UID: \"16a9e421-c525-43df-8bf6-a42110b293a4\") "
Sep 07 13:18:04.408134 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.408100    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a9e421-c525-43df-8bf6-a42110b293a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16a9e421-c525-43df-8bf6-a42110b293a4" (UID: "16a9e421-c525-43df-8bf6-a42110b293a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:18:04.409773 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.409746    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16a9e421-c525-43df-8bf6-a42110b293a4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "16a9e421-c525-43df-8bf6-a42110b293a4" (UID: "16a9e421-c525-43df-8bf6-a42110b293a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:18:04.424327 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.424040    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16a9e421-c525-43df-8bf6-a42110b293a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16a9e421-c525-43df-8bf6-a42110b293a4" (UID: "16a9e421-c525-43df-8bf6-a42110b293a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:18:04.424677 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.424635    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a9e421-c525-43df-8bf6-a42110b293a4-kube-api-access-55t68" (OuterVolumeSpecName: "kube-api-access-55t68") pod "16a9e421-c525-43df-8bf6-a42110b293a4" (UID: "16a9e421-c525-43df-8bf6-a42110b293a4"). InnerVolumeSpecName "kube-api-access-55t68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:18:04.508946 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.508904    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16a9e421-c525-43df-8bf6-a42110b293a4-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:04.508946 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.508946    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55t68\" (UniqueName: \"kubernetes.io/projected/16a9e421-c525-43df-8bf6-a42110b293a4-kube-api-access-55t68\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:04.509183 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.508964    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a9e421-c525-43df-8bf6-a42110b293a4-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:04.509183 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.508980    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16a9e421-c525-43df-8bf6-a42110b293a4-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:04.739005 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.738923    2567 generic.go:356] "Generic (PLEG): container finished" podID="16a9e421-c525-43df-8bf6-a42110b293a4" containerID="4fb9a2efb32cbbbcbf462ab3963748d3d57cb95334afee2f8d940879090a99f1" exitCode=0
Sep 07 13:18:04.739005 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.738973    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"
Sep 07 13:18:04.739005 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.738988    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm" event={"ID":"16a9e421-c525-43df-8bf6-a42110b293a4","Type":"ContainerDied","Data":"4fb9a2efb32cbbbcbf462ab3963748d3d57cb95334afee2f8d940879090a99f1"}
Sep 07 13:18:04.739288 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.739044    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm" event={"ID":"16a9e421-c525-43df-8bf6-a42110b293a4","Type":"ContainerDied","Data":"0ad2cca883d15f26a03442dea606abc451ca585bd1af94ae9f14658dc7d62208"}
Sep 07 13:18:04.739288 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.739116    2567 scope.go:117] "RemoveContainer" containerID="4fb9a2efb32cbbbcbf462ab3963748d3d57cb95334afee2f8d940879090a99f1"
Sep 07 13:18:04.748987 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.748965    2567 scope.go:117] "RemoveContainer" containerID="4fb9a2efb32cbbbcbf462ab3963748d3d57cb95334afee2f8d940879090a99f1"
Sep 07 13:18:04.749274 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:18:04.749251    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb9a2efb32cbbbcbf462ab3963748d3d57cb95334afee2f8d940879090a99f1\": container with ID starting with 4fb9a2efb32cbbbcbf462ab3963748d3d57cb95334afee2f8d940879090a99f1 not found: ID does not exist" containerID="4fb9a2efb32cbbbcbf462ab3963748d3d57cb95334afee2f8d940879090a99f1"
Sep 07 13:18:04.749330 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.749283    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb9a2efb32cbbbcbf462ab3963748d3d57cb95334afee2f8d940879090a99f1"} err="failed to get container status \"4fb9a2efb32cbbbcbf462ab3963748d3d57cb95334afee2f8d940879090a99f1\": rpc error: code = NotFound desc = could not find container \"4fb9a2efb32cbbbcbf462ab3963748d3d57cb95334afee2f8d940879090a99f1\": container with ID starting with 4fb9a2efb32cbbbcbf462ab3963748d3d57cb95334afee2f8d940879090a99f1 not found: ID does not exist"
Sep 07 13:18:04.806816 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.806784    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"]
Sep 07 13:18:04.817771 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:04.817741    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f5e4c4-59db48d66b-4jdfm"]
Sep 07 13:18:06.076248 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:06.076205    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16a9e421-c525-43df-8bf6-a42110b293a4" path="/var/lib/kubelet/pods/16a9e421-c525-43df-8bf6-a42110b293a4/volumes"
Sep 07 13:18:08.359596 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.359552    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m" podUID="dd254cab-b58d-4e73-b511-3b05e5c81849" containerName="main" containerID="cri-o://f56279d831f38adac9551559c9cd20053090403edf6ae902de10b447eb1deb3b" gracePeriod=45
Sep 07 13:18:08.610908 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.610854    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:18:08.645771 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.645744    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd254cab-b58d-4e73-b511-3b05e5c81849-tls-certs\") pod \"dd254cab-b58d-4e73-b511-3b05e5c81849\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") "
Sep 07 13:18:08.645904 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.645779    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-dshm\") pod \"dd254cab-b58d-4e73-b511-3b05e5c81849\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") "
Sep 07 13:18:08.645904 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.645840    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-model-cache\") pod \"dd254cab-b58d-4e73-b511-3b05e5c81849\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") "
Sep 07 13:18:08.645904 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.645858    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnq82\" (UniqueName: \"kubernetes.io/projected/dd254cab-b58d-4e73-b511-3b05e5c81849-kube-api-access-bnq82\") pod \"dd254cab-b58d-4e73-b511-3b05e5c81849\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") "
Sep 07 13:18:08.645904 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.645879    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-home\") pod \"dd254cab-b58d-4e73-b511-3b05e5c81849\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") "
Sep 07 13:18:08.646145 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.645952    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-tmp-dir\") pod \"dd254cab-b58d-4e73-b511-3b05e5c81849\" (UID: \"dd254cab-b58d-4e73-b511-3b05e5c81849\") "
Sep 07 13:18:08.646249 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.646159    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd254cab-b58d-4e73-b511-3b05e5c81849" (UID: "dd254cab-b58d-4e73-b511-3b05e5c81849"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:08.646377 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.646356    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-model-cache\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:08.646443 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.646391    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-home" (OuterVolumeSpecName: "home") pod "dd254cab-b58d-4e73-b511-3b05e5c81849" (UID: "dd254cab-b58d-4e73-b511-3b05e5c81849"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:08.646443 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.646406    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd254cab-b58d-4e73-b511-3b05e5c81849" (UID: "dd254cab-b58d-4e73-b511-3b05e5c81849"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:08.648323 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.648284    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-dshm" (OuterVolumeSpecName: "dshm") pod "dd254cab-b58d-4e73-b511-3b05e5c81849" (UID: "dd254cab-b58d-4e73-b511-3b05e5c81849"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:08.648416 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.648353    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd254cab-b58d-4e73-b511-3b05e5c81849-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd254cab-b58d-4e73-b511-3b05e5c81849" (UID: "dd254cab-b58d-4e73-b511-3b05e5c81849"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:18:08.648416 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.648366    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd254cab-b58d-4e73-b511-3b05e5c81849-kube-api-access-bnq82" (OuterVolumeSpecName: "kube-api-access-bnq82") pod "dd254cab-b58d-4e73-b511-3b05e5c81849" (UID: "dd254cab-b58d-4e73-b511-3b05e5c81849"). InnerVolumeSpecName "kube-api-access-bnq82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:18:08.747189 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.747152    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd254cab-b58d-4e73-b511-3b05e5c81849-tls-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:08.747189 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.747180    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-dshm\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:08.747189 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.747190    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnq82\" (UniqueName: \"kubernetes.io/projected/dd254cab-b58d-4e73-b511-3b05e5c81849-kube-api-access-bnq82\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:08.747189 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.747198    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-home\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:08.747451 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.747207    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd254cab-b58d-4e73-b511-3b05e5c81849-tmp-dir\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:08.772847 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.772813    2567 generic.go:356] "Generic (PLEG): container finished" podID="dd254cab-b58d-4e73-b511-3b05e5c81849" containerID="f56279d831f38adac9551559c9cd20053090403edf6ae902de10b447eb1deb3b" exitCode=0
Sep 07 13:18:08.773000 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.772874    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"
Sep 07 13:18:08.773000 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.772901    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m" event={"ID":"dd254cab-b58d-4e73-b511-3b05e5c81849","Type":"ContainerDied","Data":"f56279d831f38adac9551559c9cd20053090403edf6ae902de10b447eb1deb3b"}
Sep 07 13:18:08.773000 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.772952    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m" event={"ID":"dd254cab-b58d-4e73-b511-3b05e5c81849","Type":"ContainerDied","Data":"c385f6b0c75e74f2398e835da6f66b1bb7a843dff8c176fdd7d9d53af7f78048"}
Sep 07 13:18:08.773000 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.772975    2567 scope.go:117] "RemoveContainer" containerID="f56279d831f38adac9551559c9cd20053090403edf6ae902de10b447eb1deb3b"
Sep 07 13:18:08.781934 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.781909    2567 scope.go:117] "RemoveContainer" containerID="f56279d831f38adac9551559c9cd20053090403edf6ae902de10b447eb1deb3b"
Sep 07 13:18:08.782232 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:18:08.782211    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56279d831f38adac9551559c9cd20053090403edf6ae902de10b447eb1deb3b\": container with ID starting with f56279d831f38adac9551559c9cd20053090403edf6ae902de10b447eb1deb3b not found: ID does not exist" containerID="f56279d831f38adac9551559c9cd20053090403edf6ae902de10b447eb1deb3b"
Sep 07 13:18:08.782357 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.782243    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56279d831f38adac9551559c9cd20053090403edf6ae902de10b447eb1deb3b"} err="failed to get container status \"f56279d831f38adac9551559c9cd20053090403edf6ae902de10b447eb1deb3b\": rpc error: code = NotFound desc = could not find container \"f56279d831f38adac9551559c9cd20053090403edf6ae902de10b447eb1deb3b\": container with ID starting with f56279d831f38adac9551559c9cd20053090403edf6ae902de10b447eb1deb3b not found: ID does not exist"
Sep 07 13:18:08.810020 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.809999    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"]
Sep 07 13:18:08.812506 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:08.812479    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-f5e4c4/explicit-model-f5e4c4-kserve-659799c5cf-rqk5m"]
Sep 07 13:18:10.075360 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:10.075325    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd254cab-b58d-4e73-b511-3b05e5c81849" path="/var/lib/kubelet/pods/dd254cab-b58d-4e73-b511-3b05e5c81849/volumes"
Sep 07 13:18:12.226342 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.226310    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"]
Sep 07 13:18:12.226696 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.226531    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b" podUID="d8c6da54-7486-4cfe-83d3-7e421e3c55d6" containerName="maas-api" containerID="cri-o://93e072dc9700cfba2a3085559cfcfa1e850bfcede04d24340b58eebee4539dbc" gracePeriod=30
Sep 07 13:18:12.483574 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.483514    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:18:12.586260 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.586228    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-metrics-certs\") pod \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") "
Sep 07 13:18:12.586407 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.586267    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-openshift-service-ca\") pod \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") "
Sep 07 13:18:12.586407 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.586349    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xt8m\" (UniqueName: \"kubernetes.io/projected/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-kube-api-access-5xt8m\") pod \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") "
Sep 07 13:18:12.586407 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.586380    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-maas-api-tls\") pod \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\" (UID: \"d8c6da54-7486-4cfe-83d3-7e421e3c55d6\") "
Sep 07 13:18:12.586746 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.586713    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d8c6da54-7486-4cfe-83d3-7e421e3c55d6" (UID: "d8c6da54-7486-4cfe-83d3-7e421e3c55d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:18:12.588651 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.588614    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d8c6da54-7486-4cfe-83d3-7e421e3c55d6" (UID: "d8c6da54-7486-4cfe-83d3-7e421e3c55d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:18:12.588651 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.588637    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d8c6da54-7486-4cfe-83d3-7e421e3c55d6" (UID: "d8c6da54-7486-4cfe-83d3-7e421e3c55d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:18:12.588651 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.588621    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-kube-api-access-5xt8m" (OuterVolumeSpecName: "kube-api-access-5xt8m") pod "d8c6da54-7486-4cfe-83d3-7e421e3c55d6" (UID: "d8c6da54-7486-4cfe-83d3-7e421e3c55d6"). InnerVolumeSpecName "kube-api-access-5xt8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:18:12.688030 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.687987    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xt8m\" (UniqueName: \"kubernetes.io/projected/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-kube-api-access-5xt8m\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:12.688030 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.688030    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:12.688256 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.688045    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:12.688256 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.688072    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6da54-7486-4cfe-83d3-7e421e3c55d6-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:12.800871 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.800836    2567 generic.go:356] "Generic (PLEG): container finished" podID="d8c6da54-7486-4cfe-83d3-7e421e3c55d6" containerID="93e072dc9700cfba2a3085559cfcfa1e850bfcede04d24340b58eebee4539dbc" exitCode=0
Sep 07 13:18:12.801026 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.800878    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"
Sep 07 13:18:12.801026 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.800913    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b" event={"ID":"d8c6da54-7486-4cfe-83d3-7e421e3c55d6","Type":"ContainerDied","Data":"93e072dc9700cfba2a3085559cfcfa1e850bfcede04d24340b58eebee4539dbc"}
Sep 07 13:18:12.801026 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.800989    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b" event={"ID":"d8c6da54-7486-4cfe-83d3-7e421e3c55d6","Type":"ContainerDied","Data":"515a5232bb50547fecd7ef4a4f7b6a0578ca5c9684232b936c600222feaad385"}
Sep 07 13:18:12.801026 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.801011    2567 scope.go:117] "RemoveContainer" containerID="93e072dc9700cfba2a3085559cfcfa1e850bfcede04d24340b58eebee4539dbc"
Sep 07 13:18:12.810543 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.810528    2567 scope.go:117] "RemoveContainer" containerID="93e072dc9700cfba2a3085559cfcfa1e850bfcede04d24340b58eebee4539dbc"
Sep 07 13:18:12.810806 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:18:12.810788    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e072dc9700cfba2a3085559cfcfa1e850bfcede04d24340b58eebee4539dbc\": container with ID starting with 93e072dc9700cfba2a3085559cfcfa1e850bfcede04d24340b58eebee4539dbc not found: ID does not exist" containerID="93e072dc9700cfba2a3085559cfcfa1e850bfcede04d24340b58eebee4539dbc"
Sep 07 13:18:12.810846 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.810816    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e072dc9700cfba2a3085559cfcfa1e850bfcede04d24340b58eebee4539dbc"} err="failed to get container status \"93e072dc9700cfba2a3085559cfcfa1e850bfcede04d24340b58eebee4539dbc\": rpc error: code = NotFound desc = could not find container \"93e072dc9700cfba2a3085559cfcfa1e850bfcede04d24340b58eebee4539dbc\": container with ID starting with 93e072dc9700cfba2a3085559cfcfa1e850bfcede04d24340b58eebee4539dbc not found: ID does not exist"
Sep 07 13:18:12.844003 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.843975    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"]
Sep 07 13:18:12.878500 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:12.878472    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ebfa46-895d7f6f4-nvz6b"]
Sep 07 13:18:14.075669 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.075638    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8c6da54-7486-4cfe-83d3-7e421e3c55d6" path="/var/lib/kubelet/pods/d8c6da54-7486-4cfe-83d3-7e421e3c55d6/volumes"
Sep 07 13:18:14.884119 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.883645    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"]
Sep 07 13:18:14.887433 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.884421    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c6da54-7486-4cfe-83d3-7e421e3c55d6" containerName="maas-api"
Sep 07 13:18:14.887433 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.884448    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c6da54-7486-4cfe-83d3-7e421e3c55d6" containerName="maas-api"
Sep 07 13:18:14.887433 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.884487    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16a9e421-c525-43df-8bf6-a42110b293a4" containerName="maas-api"
Sep 07 13:18:14.887433 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.884498    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="16a9e421-c525-43df-8bf6-a42110b293a4" containerName="maas-api"
Sep 07 13:18:14.887433 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.884507    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd254cab-b58d-4e73-b511-3b05e5c81849" containerName="main"
Sep 07 13:18:14.887433 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.884514    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd254cab-b58d-4e73-b511-3b05e5c81849" containerName="main"
Sep 07 13:18:14.887433 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.884616    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c6da54-7486-4cfe-83d3-7e421e3c55d6" containerName="maas-api"
Sep 07 13:18:14.887433 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.884633    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="16a9e421-c525-43df-8bf6-a42110b293a4" containerName="maas-api"
Sep 07 13:18:14.887433 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.884641    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd254cab-b58d-4e73-b511-3b05e5c81849" containerName="main"
Sep 07 13:18:14.889192 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.889168    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:14.891859 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.891835    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-067a48-openshift-default-dockercfg-gmp2x\""
Sep 07 13:18:14.903870 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:14.903836    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"]
Sep 07 13:18:15.009629 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.009590    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-token\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.009820 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.009646    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-workload-socket\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.009820 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.009686    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e767f6-7128-4b09-9ec4-28ecd555a97f-istiod-ca-cert\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.009820 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.009717    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-podinfo\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.009820 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.009797    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d8tb\" (UniqueName: \"kubernetes.io/projected/12e767f6-7128-4b09-9ec4-28ecd555a97f-kube-api-access-4d8tb\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.009992 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.009830    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-workload-certs\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.009992 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.009888    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-credential-socket\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.009992 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.009944    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-data\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.010134 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.009999    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-envoy\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.110513 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.110465    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-token\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.110513 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.110521    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-workload-socket\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.111047 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.110540    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e767f6-7128-4b09-9ec4-28ecd555a97f-istiod-ca-cert\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.111047 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.110558    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-podinfo\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.111047 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.110588    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8tb\" (UniqueName: \"kubernetes.io/projected/12e767f6-7128-4b09-9ec4-28ecd555a97f-kube-api-access-4d8tb\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.111047 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.110617    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-workload-certs\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.111047 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.110675    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-credential-socket\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.111047 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.110705    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-data\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.111047 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.110740    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-envoy\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.111471 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.111140    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-workload-certs\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.111471 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.111198    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-data\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.111471 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.111245    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-credential-socket\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.111471 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.111384    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e767f6-7128-4b09-9ec4-28ecd555a97f-istiod-ca-cert\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.111645 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.111552    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-workload-socket\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.113097 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.113072    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-envoy\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.113195 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.113153    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-podinfo\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.120770 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.120729    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-token\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.120979 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.120958    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d8tb\" (UniqueName: \"kubernetes.io/projected/12e767f6-7128-4b09-9ec4-28ecd555a97f-kube-api-access-4d8tb\") pod \"e2e-noten-067a48-openshift-default-768b86f464-vvk9t\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") " pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.204741 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.204661    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:15.498584 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.498492    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"]
Sep 07 13:18:15.504835 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:18:15.504801    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e767f6_7128_4b09_9ec4_28ecd555a97f.slice/crio-bbc0d6de6424cf7d92431d9066395c4bbe422c30e79f35fd6b612d9749805f44 WatchSource:0}: Error finding container bbc0d6de6424cf7d92431d9066395c4bbe422c30e79f35fd6b612d9749805f44: Status 404 returned error can't find the container with id bbc0d6de6424cf7d92431d9066395c4bbe422c30e79f35fd6b612d9749805f44
Sep 07 13:18:15.507117 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.507054    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:18:15.507221 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.507159    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:18:15.507221 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.507193    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:18:15.820015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.819968    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t" event={"ID":"12e767f6-7128-4b09-9ec4-28ecd555a97f","Type":"ContainerStarted","Data":"94b135b0cb6eea31d46c82cb6e643a8e1bf7d6e53342f6f06c97d47128dc8dcc"}
Sep 07 13:18:15.820214 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.820042    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t" event={"ID":"12e767f6-7128-4b09-9ec4-28ecd555a97f","Type":"ContainerStarted","Data":"bbc0d6de6424cf7d92431d9066395c4bbe422c30e79f35fd6b612d9749805f44"}
Sep 07 13:18:15.846724 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:15.846670    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t" podStartSLOduration=1.846654031 podStartE2EDuration="1.846654031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:18:15.845200536 +0000 UTC m=+1140.273498809" watchObservedRunningTime="2026-09-07 13:18:15.846654031 +0000 UTC m=+1140.274952305"
Sep 07 13:18:16.205515 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:16.205478    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:16.210475 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:16.210452    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:16.826544 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:16.826502    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:16.827527 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:16.827502    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:18:23.681366 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:23.681328    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"]
Sep 07 13:18:23.681848 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:23.681585    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerName="istio-proxy" containerID="cri-o://5ce8f3900d86fde20626b1a47dbe3e75eb354e86e45de6be39f5c89d53d0a521" gracePeriod=30
Sep 07 13:18:27.835565 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:27.835530    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"]
Sep 07 13:18:27.840530 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:27.840507    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:27.851425 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:27.851404    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3dd3e936-serving-cert\""
Sep 07 13:18:27.907001 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:27.906964    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"]
Sep 07 13:18:27.934989 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:27.934956    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5992306f-a972-4231-a79c-6ccb291b5663-openshift-service-ca\") pod \"maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:27.935241 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:27.935220    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlb5p\" (UniqueName: \"kubernetes.io/projected/5992306f-a972-4231-a79c-6ccb291b5663-kube-api-access-jlb5p\") pod \"maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:27.935327 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:27.935275    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-maas-api-tls\") pod \"maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:27.935393 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:27.935372    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-metrics-certs\") pod \"maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:28.036661 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.036622    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5992306f-a972-4231-a79c-6ccb291b5663-openshift-service-ca\") pod \"maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:28.036846 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.036702    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlb5p\" (UniqueName: \"kubernetes.io/projected/5992306f-a972-4231-a79c-6ccb291b5663-kube-api-access-jlb5p\") pod \"maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:28.036846 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.036729    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-maas-api-tls\") pod \"maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:28.036846 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.036774    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-metrics-certs\") pod \"maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:28.037116 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:18:28.036880    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-serving-cert: secret "maas-api-e2e-mt-3dd3e936-serving-cert" not found
Sep 07 13:18:28.037116 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:18:28.036972    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-maas-api-tls podName:5992306f-a972-4231-a79c-6ccb291b5663 nodeName:}" failed. No retries permitted until 2026-09-07 13:18:28.5369506 +0000 UTC m=+1152.965248851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-maas-api-tls") pod "maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz" (UID: "5992306f-a972-4231-a79c-6ccb291b5663") : secret "maas-api-e2e-mt-3dd3e936-serving-cert" not found
Sep 07 13:18:28.037423 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.037399    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5992306f-a972-4231-a79c-6ccb291b5663-openshift-service-ca\") pod \"maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:28.039119 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.039095    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-metrics-certs\") pod \"maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:28.067946 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.067922    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlb5p\" (UniqueName: \"kubernetes.io/projected/5992306f-a972-4231-a79c-6ccb291b5663-kube-api-access-jlb5p\") pod \"maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:28.540755 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.540715    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-maas-api-tls\") pod \"maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:28.543078 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.543039    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-maas-api-tls\") pod \"maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:28.751762 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.751732    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:28.897707 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.897669    2567 generic.go:356] "Generic (PLEG): container finished" podID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerID="5ce8f3900d86fde20626b1a47dbe3e75eb354e86e45de6be39f5c89d53d0a521" exitCode=0
Sep 07 13:18:28.898102 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.897739    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" event={"ID":"99f4c689-aeda-4c34-9ffd-0f7ee0a24063","Type":"ContainerDied","Data":"5ce8f3900d86fde20626b1a47dbe3e75eb354e86e45de6be39f5c89d53d0a521"}
Sep 07 13:18:28.898102 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.897803    2567 scope.go:117] "RemoveContainer" containerID="b8db830900a1f6c1889b08eb6e4ab94647ed93c699b92bdfda972fcd6951490e"
Sep 07 13:18:28.941235 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:28.941209    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:18:29.044952 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.044912    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-workload-socket\") pod \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") "
Sep 07 13:18:29.045148 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.044972    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-podinfo\") pod \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") "
Sep 07 13:18:29.045148 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045015    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-credential-socket\") pod \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") "
Sep 07 13:18:29.045148 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045052    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-token\") pod \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") "
Sep 07 13:18:29.045148 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045092    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-workload-certs\") pod \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") "
Sep 07 13:18:29.045148 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045121    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stjtv\" (UniqueName: \"kubernetes.io/projected/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-kube-api-access-stjtv\") pod \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") "
Sep 07 13:18:29.045452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045164    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istiod-ca-cert\") pod \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") "
Sep 07 13:18:29.045452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045219    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-envoy\") pod \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") "
Sep 07 13:18:29.045452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045234    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "99f4c689-aeda-4c34-9ffd-0f7ee0a24063" (UID: "99f4c689-aeda-4c34-9ffd-0f7ee0a24063"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:29.045452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045277    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-data\") pod \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\" (UID: \"99f4c689-aeda-4c34-9ffd-0f7ee0a24063\") "
Sep 07 13:18:29.045452 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045393    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "99f4c689-aeda-4c34-9ffd-0f7ee0a24063" (UID: "99f4c689-aeda-4c34-9ffd-0f7ee0a24063"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:29.045971 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045610    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-data" (OuterVolumeSpecName: "istio-data") pod "99f4c689-aeda-4c34-9ffd-0f7ee0a24063" (UID: "99f4c689-aeda-4c34-9ffd-0f7ee0a24063"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:29.045971 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045649    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-workload-socket\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.045971 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045672    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-credential-socket\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.045971 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045823    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "99f4c689-aeda-4c34-9ffd-0f7ee0a24063" (UID: "99f4c689-aeda-4c34-9ffd-0f7ee0a24063"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:29.045971 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.045876    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "99f4c689-aeda-4c34-9ffd-0f7ee0a24063" (UID: "99f4c689-aeda-4c34-9ffd-0f7ee0a24063"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:18:29.047790 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.047762    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "99f4c689-aeda-4c34-9ffd-0f7ee0a24063" (UID: "99f4c689-aeda-4c34-9ffd-0f7ee0a24063"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:18:29.047901 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.047788    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-kube-api-access-stjtv" (OuterVolumeSpecName: "kube-api-access-stjtv") pod "99f4c689-aeda-4c34-9ffd-0f7ee0a24063" (UID: "99f4c689-aeda-4c34-9ffd-0f7ee0a24063"). InnerVolumeSpecName "kube-api-access-stjtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:18:29.048034 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.048006    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "99f4c689-aeda-4c34-9ffd-0f7ee0a24063" (UID: "99f4c689-aeda-4c34-9ffd-0f7ee0a24063"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:29.048210 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.048195    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-token" (OuterVolumeSpecName: "istio-token") pod "99f4c689-aeda-4c34-9ffd-0f7ee0a24063" (UID: "99f4c689-aeda-4c34-9ffd-0f7ee0a24063"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:18:29.054433 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.054378    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"]
Sep 07 13:18:29.066070 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:18:29.066036    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5992306f_a972_4231_a79c_6ccb291b5663.slice/crio-a8931931c8e56cde1e7067d1fee383ed9ff966b6385a5917e4ed56f7c4b94c47 WatchSource:0}: Error finding container a8931931c8e56cde1e7067d1fee383ed9ff966b6385a5917e4ed56f7c4b94c47: Status 404 returned error can't find the container with id a8931931c8e56cde1e7067d1fee383ed9ff966b6385a5917e4ed56f7c4b94c47
Sep 07 13:18:29.146311 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.146279    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istiod-ca-cert\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.146311 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.146304    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-envoy\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.146311 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.146313    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-data\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.146525 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.146321    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-podinfo\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.146525 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.146329    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-istio-token\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.146525 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.146337    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-workload-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.146525 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.146345    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stjtv\" (UniqueName: \"kubernetes.io/projected/99f4c689-aeda-4c34-9ffd-0f7ee0a24063-kube-api-access-stjtv\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.906029 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.905990    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz" event={"ID":"5992306f-a972-4231-a79c-6ccb291b5663","Type":"ContainerStarted","Data":"afd35e05c27981ad4f3e7c84d94ff75b3ab1906bc7d6251fd309da26b65b5a43"}
Sep 07 13:18:29.906499 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.906086    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:29.906499 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.906114    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz" event={"ID":"5992306f-a972-4231-a79c-6ccb291b5663","Type":"ContainerStarted","Data":"a8931931c8e56cde1e7067d1fee383ed9ff966b6385a5917e4ed56f7c4b94c47"}
Sep 07 13:18:29.907467 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.907448    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"
Sep 07 13:18:29.907564 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.907447    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk" event={"ID":"99f4c689-aeda-4c34-9ffd-0f7ee0a24063","Type":"ContainerDied","Data":"3159b4e3831a02f2dcaf1df5307fe3257f02f92d6fb022ed3685b1fe8a1aef27"}
Sep 07 13:18:29.907603 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.907581    2567 scope.go:117] "RemoveContainer" containerID="5ce8f3900d86fde20626b1a47dbe3e75eb354e86e45de6be39f5c89d53d0a521"
Sep 07 13:18:29.946463 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:29.946419    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz" podStartSLOduration=2.498207295 podStartE2EDuration="2.946403264s" totalImagesPullingTime="448.195969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:18:29.941644524 +0000 UTC m=+1154.369942796" watchObservedRunningTime="2026-09-07 13:18:29.946403264 +0000 UTC m=+1154.374701549"
Sep 07 13:18:30.040393 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:30.040352    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"]
Sep 07 13:18:30.050967 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:30.050935    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ebfa46-openshift-default-5fbbb4b666-lrszk"]
Sep 07 13:18:30.075951 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:30.075921    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" path="/var/lib/kubelet/pods/99f4c689-aeda-4c34-9ffd-0f7ee0a24063/volumes"
Sep 07 13:18:35.574844 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:35.574807    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"]
Sep 07 13:18:35.575311 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:35.575011    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk" podUID="06ae1771-be89-425c-9bd9-79fb318663fa" containerName="maas-api" containerID="cri-o://befe62669bb82ba1dd8230b8e9ad57bb14c883502f0e2dffd272752b31eff67f" gracePeriod=30
Sep 07 13:18:35.851133 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:35.851109    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:18:35.918726 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:35.918694    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:18:35.947671 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:35.947635    2567 generic.go:356] "Generic (PLEG): container finished" podID="06ae1771-be89-425c-9bd9-79fb318663fa" containerID="befe62669bb82ba1dd8230b8e9ad57bb14c883502f0e2dffd272752b31eff67f" exitCode=0
Sep 07 13:18:35.947833 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:35.947684    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"
Sep 07 13:18:35.947833 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:35.947736    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk" event={"ID":"06ae1771-be89-425c-9bd9-79fb318663fa","Type":"ContainerDied","Data":"befe62669bb82ba1dd8230b8e9ad57bb14c883502f0e2dffd272752b31eff67f"}
Sep 07 13:18:35.947833 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:35.947791    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk" event={"ID":"06ae1771-be89-425c-9bd9-79fb318663fa","Type":"ContainerDied","Data":"e55d3cc42826c484b99175fba347d377391ee7eb92befb477d2c1b0c35852f89"}
Sep 07 13:18:35.947833 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:35.947804    2567 scope.go:117] "RemoveContainer" containerID="befe62669bb82ba1dd8230b8e9ad57bb14c883502f0e2dffd272752b31eff67f"
Sep 07 13:18:35.956883 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:35.956865    2567 scope.go:117] "RemoveContainer" containerID="befe62669bb82ba1dd8230b8e9ad57bb14c883502f0e2dffd272752b31eff67f"
Sep 07 13:18:35.957214 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:18:35.957182    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"befe62669bb82ba1dd8230b8e9ad57bb14c883502f0e2dffd272752b31eff67f\": container with ID starting with befe62669bb82ba1dd8230b8e9ad57bb14c883502f0e2dffd272752b31eff67f not found: ID does not exist" containerID="befe62669bb82ba1dd8230b8e9ad57bb14c883502f0e2dffd272752b31eff67f"
Sep 07 13:18:35.957279 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:35.957228    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befe62669bb82ba1dd8230b8e9ad57bb14c883502f0e2dffd272752b31eff67f"} err="failed to get container status \"befe62669bb82ba1dd8230b8e9ad57bb14c883502f0e2dffd272752b31eff67f\": rpc error: code = NotFound desc = could not find container \"befe62669bb82ba1dd8230b8e9ad57bb14c883502f0e2dffd272752b31eff67f\": container with ID starting with befe62669bb82ba1dd8230b8e9ad57bb14c883502f0e2dffd272752b31eff67f not found: ID does not exist"
Sep 07 13:18:36.007366 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.007332    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxv8g\" (UniqueName: \"kubernetes.io/projected/06ae1771-be89-425c-9bd9-79fb318663fa-kube-api-access-wxv8g\") pod \"06ae1771-be89-425c-9bd9-79fb318663fa\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") "
Sep 07 13:18:36.007513 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.007457    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ae1771-be89-425c-9bd9-79fb318663fa-openshift-service-ca\") pod \"06ae1771-be89-425c-9bd9-79fb318663fa\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") "
Sep 07 13:18:36.007599 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.007533    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-metrics-certs\") pod \"06ae1771-be89-425c-9bd9-79fb318663fa\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") "
Sep 07 13:18:36.007599 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.007561    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-maas-api-tls\") pod \"06ae1771-be89-425c-9bd9-79fb318663fa\" (UID: \"06ae1771-be89-425c-9bd9-79fb318663fa\") "
Sep 07 13:18:36.007861 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.007834    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ae1771-be89-425c-9bd9-79fb318663fa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06ae1771-be89-425c-9bd9-79fb318663fa" (UID: "06ae1771-be89-425c-9bd9-79fb318663fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:18:36.007966 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.007933    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ae1771-be89-425c-9bd9-79fb318663fa-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:36.009816 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.009776    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "06ae1771-be89-425c-9bd9-79fb318663fa" (UID: "06ae1771-be89-425c-9bd9-79fb318663fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:18:36.010021 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.010002    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06ae1771-be89-425c-9bd9-79fb318663fa" (UID: "06ae1771-be89-425c-9bd9-79fb318663fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:18:36.010172 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.010026    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ae1771-be89-425c-9bd9-79fb318663fa-kube-api-access-wxv8g" (OuterVolumeSpecName: "kube-api-access-wxv8g") pod "06ae1771-be89-425c-9bd9-79fb318663fa" (UID: "06ae1771-be89-425c-9bd9-79fb318663fa"). InnerVolumeSpecName "kube-api-access-wxv8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:18:36.109105 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.108945    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:36.109105 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.108980    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ae1771-be89-425c-9bd9-79fb318663fa-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:36.109105 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.108993    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxv8g\" (UniqueName: \"kubernetes.io/projected/06ae1771-be89-425c-9bd9-79fb318663fa-kube-api-access-wxv8g\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:18:36.331645 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.331609    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"]
Sep 07 13:18:36.352339 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:36.352309    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-824a79-9d44b8b4-9fjtk"]
Sep 07 13:18:38.075476 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:38.075432    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06ae1771-be89-425c-9bd9-79fb318663fa" path="/var/lib/kubelet/pods/06ae1771-be89-425c-9bd9-79fb318663fa/volumes"
Sep 07 13:18:45.407154 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.407116    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"]
Sep 07 13:18:45.407541 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.407516    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ae1771-be89-425c-9bd9-79fb318663fa" containerName="maas-api"
Sep 07 13:18:45.407541 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.407530    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ae1771-be89-425c-9bd9-79fb318663fa" containerName="maas-api"
Sep 07 13:18:45.407616 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.407552    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerName="istio-proxy"
Sep 07 13:18:45.407616 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.407558    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerName="istio-proxy"
Sep 07 13:18:45.407616 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.407571    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerName="istio-proxy"
Sep 07 13:18:45.407616 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.407576    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerName="istio-proxy"
Sep 07 13:18:45.407731 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.407633    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerName="istio-proxy"
Sep 07 13:18:45.407731 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.407640    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ae1771-be89-425c-9bd9-79fb318663fa" containerName="maas-api"
Sep 07 13:18:45.410584 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.410567    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.426323 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.426300    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-5910bb-openshift-default-dockercfg-2tzkp\""
Sep 07 13:18:45.449893 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.449866    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"]
Sep 07 13:18:45.490286 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.490254    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863eb763-4af8-41b7-b579-2652c3351065-istio-podinfo\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.490479 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.490304    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-workload-certs\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.490479 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.490353    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-istio-data\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.490479 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.490416    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-credential-socket\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.490479 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.490458    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-workload-socket\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.490702 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.490485    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr5cm\" (UniqueName: \"kubernetes.io/projected/863eb763-4af8-41b7-b579-2652c3351065-kube-api-access-mr5cm\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.490702 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.490507    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863eb763-4af8-41b7-b579-2652c3351065-istiod-ca-cert\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.490702 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.490530    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863eb763-4af8-41b7-b579-2652c3351065-istio-token\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.490702 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.490578    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-istio-envoy\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.591891 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.591848    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-workload-socket\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.592121 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.591898    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr5cm\" (UniqueName: \"kubernetes.io/projected/863eb763-4af8-41b7-b579-2652c3351065-kube-api-access-mr5cm\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.592121 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.591927    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863eb763-4af8-41b7-b579-2652c3351065-istiod-ca-cert\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.592121 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.591963    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863eb763-4af8-41b7-b579-2652c3351065-istio-token\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.592121 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.591999    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-istio-envoy\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.592121 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.592046    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863eb763-4af8-41b7-b579-2652c3351065-istio-podinfo\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.592415 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.592182    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-workload-certs\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.592415 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.592224    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-istio-data\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.592415 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.592245    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-workload-socket\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.592415 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.592308    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-credential-socket\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.592627 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.592531    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-istio-data\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.592627 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.592619    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-credential-socket\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.592754 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.592731    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-workload-certs\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.593120 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.593101    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863eb763-4af8-41b7-b579-2652c3351065-istiod-ca-cert\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.594931 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.594909    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-istio-envoy\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.595186 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.595169    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863eb763-4af8-41b7-b579-2652c3351065-istio-podinfo\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.610696 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.610667    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863eb763-4af8-41b7-b579-2652c3351065-istio-token\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.613682 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.613664    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr5cm\" (UniqueName: \"kubernetes.io/projected/863eb763-4af8-41b7-b579-2652c3351065-kube-api-access-mr5cm\") pod \"e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") " pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:45.722214 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:45.722118    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:46.030144 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:46.030119    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"]
Sep 07 13:18:46.030514 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:18:46.030491    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863eb763_4af8_41b7_b579_2652c3351065.slice/crio-bb1422e656828fbf0e2ccac3878563ad0d080b05330cd1532a14dd625d99a561 WatchSource:0}: Error finding container bb1422e656828fbf0e2ccac3878563ad0d080b05330cd1532a14dd625d99a561: Status 404 returned error can't find the container with id bb1422e656828fbf0e2ccac3878563ad0d080b05330cd1532a14dd625d99a561
Sep 07 13:18:46.032732 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:46.032699    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:18:46.032814 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:46.032759    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:18:46.032814 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:46.032789    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:18:47.010748 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:47.010709    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" event={"ID":"863eb763-4af8-41b7-b579-2652c3351065","Type":"ContainerStarted","Data":"69afb87447f753de05c38b47d4b219fae103047c099386fbb574f23b2b794701"}
Sep 07 13:18:47.011226 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:47.010777    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" event={"ID":"863eb763-4af8-41b7-b579-2652c3351065","Type":"ContainerStarted","Data":"bb1422e656828fbf0e2ccac3878563ad0d080b05330cd1532a14dd625d99a561"}
Sep 07 13:18:47.054093 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:47.054025    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" podStartSLOduration=2.05400856 podStartE2EDuration="2.05400856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:18:47.051167128 +0000 UTC m=+1171.479465414" watchObservedRunningTime="2026-09-07 13:18:47.05400856 +0000 UTC m=+1171.482306832"
Sep 07 13:18:47.723237 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:47.723179    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:47.728309 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:47.728282    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:48.016525 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:48.016421    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:48.017286 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:48.017263    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:18:50.323535 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.323482    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"]
Sep 07 13:18:50.324207 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.324155    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="99f4c689-aeda-4c34-9ffd-0f7ee0a24063" containerName="istio-proxy"
Sep 07 13:18:50.327698 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.327674    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:50.331561 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.331534    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-5910bb-serving-cert\""
Sep 07 13:18:50.340699 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.340677    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"]
Sep 07 13:18:50.440905 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.440866    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:50.441130 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.440923    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g4jz\" (UniqueName: \"kubernetes.io/projected/e5e7fa90-f906-4ee5-af46-2b942481bd11-kube-api-access-9g4jz\") pod \"maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:50.441130 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.440999    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e7fa90-f906-4ee5-af46-2b942481bd11-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:50.441130 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.441105    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:50.542108 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.542038    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e7fa90-f906-4ee5-af46-2b942481bd11-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:50.542292 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.542164    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:50.542292 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.542200    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:50.542292 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.542254    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g4jz\" (UniqueName: \"kubernetes.io/projected/e5e7fa90-f906-4ee5-af46-2b942481bd11-kube-api-access-9g4jz\") pod \"maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:50.542451 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:18:50.542339    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-serving-cert: secret "maas-api-e2e-shared-a-w3-5910bb-serving-cert" not found
Sep 07 13:18:50.542451 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:18:50.542420    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-maas-api-tls podName:e5e7fa90-f906-4ee5-af46-2b942481bd11 nodeName:}" failed. No retries permitted until 2026-09-07 13:18:51.042395746 +0000 UTC m=+1175.470694012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-maas-api-tls") pod "maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr" (UID: "e5e7fa90-f906-4ee5-af46-2b942481bd11") : secret "maas-api-e2e-shared-a-w3-5910bb-serving-cert" not found
Sep 07 13:18:50.542695 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.542677    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e7fa90-f906-4ee5-af46-2b942481bd11-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:50.544813 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.544789    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:50.554379 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:50.554354    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g4jz\" (UniqueName: \"kubernetes.io/projected/e5e7fa90-f906-4ee5-af46-2b942481bd11-kube-api-access-9g4jz\") pod \"maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:51.046655 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:51.046617    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:51.049208 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:51.049178    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:51.240711 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:51.240676    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:51.569722 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:18:51.563224    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e7fa90_f906_4ee5_af46_2b942481bd11.slice/crio-b2b5f7ab37035ad17773c3a84fbb664635c3cb1b1169c2df8e2588e645588d30 WatchSource:0}: Error finding container b2b5f7ab37035ad17773c3a84fbb664635c3cb1b1169c2df8e2588e645588d30: Status 404 returned error can't find the container with id b2b5f7ab37035ad17773c3a84fbb664635c3cb1b1169c2df8e2588e645588d30
Sep 07 13:18:51.569722 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:51.569611    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"]
Sep 07 13:18:52.039562 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:52.039527    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr" event={"ID":"e5e7fa90-f906-4ee5-af46-2b942481bd11","Type":"ContainerStarted","Data":"68b9c339ecc9f1fd18b1b5f5a53ee614ea51c6dda6fed0a8020c1ec60e3d2786"}
Sep 07 13:18:52.039730 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:52.039581    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr" event={"ID":"e5e7fa90-f906-4ee5-af46-2b942481bd11","Type":"ContainerStarted","Data":"b2b5f7ab37035ad17773c3a84fbb664635c3cb1b1169c2df8e2588e645588d30"}
Sep 07 13:18:52.039730 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:52.039631    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:18:52.061716 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:52.061660    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr" podStartSLOduration=1.732050742 podStartE2EDuration="2.061643934s" totalImagesPullingTime="329.593192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:18:52.058525236 +0000 UTC m=+1176.486823532" watchObservedRunningTime="2026-09-07 13:18:52.061643934 +0000 UTC m=+1176.489942224"
Sep 07 13:18:58.051302 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:18:58.051272    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:19:04.058235 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:04.058190    2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 13:19:04.058660 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:04.058268    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 13:19:05.124317 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:05.124236    2567 generic.go:356] "Generic (PLEG): container finished" podID="863eb763-4af8-41b7-b579-2652c3351065" containerID="69afb87447f753de05c38b47d4b219fae103047c099386fbb574f23b2b794701" exitCode=0
Sep 07 13:19:05.124717 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:05.124317    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" event={"ID":"863eb763-4af8-41b7-b579-2652c3351065","Type":"ContainerDied","Data":"69afb87447f753de05c38b47d4b219fae103047c099386fbb574f23b2b794701"}
Sep 07 13:19:05.124771 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:05.124730    2567 scope.go:117] "RemoveContainer" containerID="69afb87447f753de05c38b47d4b219fae103047c099386fbb574f23b2b794701"
Sep 07 13:19:05.125967 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:05.125937    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:19:05.126042 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:05.126012    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:19:05.126102 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:05.126041    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:19:06.132751 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:06.132711    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" event={"ID":"863eb763-4af8-41b7-b579-2652c3351065","Type":"ContainerStarted","Data":"29680f742e73ba27d8a70a01228ba672e149ee260985f3c948689d4cec066e13"}
Sep 07 13:19:06.723320 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:06.723281    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:19:06.723619 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:06.723583    2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 07 13:19:06.723758 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:06.723655    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 07 13:19:07.723311 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:07.723278    2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 07 13:19:07.723697 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:07.723331    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 07 13:19:08.723005 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:08.722961    2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 07 13:19:08.723205 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:08.723041    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 07 13:19:09.722906 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:09.722869    2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 07 13:19:09.723412 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:09.722929    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 07 13:19:10.722832 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:10.722792    2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 07 13:19:10.723012 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:10.722851    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 07 13:19:11.726557 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:11.726523    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:19:11.727037 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:11.726868    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:19:11.727642 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:11.727624    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:19:15.816018 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:15.815978    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"]
Sep 07 13:19:15.816450 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:15.816213    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz" podUID="5992306f-a972-4231-a79c-6ccb291b5663" containerName="maas-api" containerID="cri-o://afd35e05c27981ad4f3e7c84d94ff75b3ab1906bc7d6251fd309da26b65b5a43" gracePeriod=30
Sep 07 13:19:15.914127 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:15.914079    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz" podUID="5992306f-a972-4231-a79c-6ccb291b5663" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.59:8443/health\": dial tcp 10.132.0.59:8443: connect: connection refused"
Sep 07 13:19:15.993977 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:19:15.993949    2567 info.go:109] Failed to get network devices: open /sys/class/net/a8931931c8e56cd/address: no such file or directory
Sep 07 13:19:16.072424 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.072357    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:19:16.191340 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.191303    2567 generic.go:356] "Generic (PLEG): container finished" podID="5992306f-a972-4231-a79c-6ccb291b5663" containerID="afd35e05c27981ad4f3e7c84d94ff75b3ab1906bc7d6251fd309da26b65b5a43" exitCode=0
Sep 07 13:19:16.191536 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.191347    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz" event={"ID":"5992306f-a972-4231-a79c-6ccb291b5663","Type":"ContainerDied","Data":"afd35e05c27981ad4f3e7c84d94ff75b3ab1906bc7d6251fd309da26b65b5a43"}
Sep 07 13:19:16.191536 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.191426    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz" event={"ID":"5992306f-a972-4231-a79c-6ccb291b5663","Type":"ContainerDied","Data":"a8931931c8e56cde1e7067d1fee383ed9ff966b6385a5917e4ed56f7c4b94c47"}
Sep 07 13:19:16.191536 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.191459    2567 scope.go:117] "RemoveContainer" containerID="afd35e05c27981ad4f3e7c84d94ff75b3ab1906bc7d6251fd309da26b65b5a43"
Sep 07 13:19:16.191536 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.191486    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"
Sep 07 13:19:16.191775 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.191750    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-maas-api-tls\") pod \"5992306f-a972-4231-a79c-6ccb291b5663\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") "
Sep 07 13:19:16.191838 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.191784    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlb5p\" (UniqueName: \"kubernetes.io/projected/5992306f-a972-4231-a79c-6ccb291b5663-kube-api-access-jlb5p\") pod \"5992306f-a972-4231-a79c-6ccb291b5663\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") "
Sep 07 13:19:16.191896 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.191884    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5992306f-a972-4231-a79c-6ccb291b5663-openshift-service-ca\") pod \"5992306f-a972-4231-a79c-6ccb291b5663\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") "
Sep 07 13:19:16.191950 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.191922    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-metrics-certs\") pod \"5992306f-a972-4231-a79c-6ccb291b5663\" (UID: \"5992306f-a972-4231-a79c-6ccb291b5663\") "
Sep 07 13:19:16.192446 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.192414    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5992306f-a972-4231-a79c-6ccb291b5663-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5992306f-a972-4231-a79c-6ccb291b5663" (UID: "5992306f-a972-4231-a79c-6ccb291b5663"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:19:16.194997 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.194939    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5992306f-a972-4231-a79c-6ccb291b5663" (UID: "5992306f-a972-4231-a79c-6ccb291b5663"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:19:16.194997 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.194952    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5992306f-a972-4231-a79c-6ccb291b5663-kube-api-access-jlb5p" (OuterVolumeSpecName: "kube-api-access-jlb5p") pod "5992306f-a972-4231-a79c-6ccb291b5663" (UID: "5992306f-a972-4231-a79c-6ccb291b5663"). InnerVolumeSpecName "kube-api-access-jlb5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:19:16.194997 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.194964    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5992306f-a972-4231-a79c-6ccb291b5663" (UID: "5992306f-a972-4231-a79c-6ccb291b5663"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:19:16.214933 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.214911    2567 scope.go:117] "RemoveContainer" containerID="afd35e05c27981ad4f3e7c84d94ff75b3ab1906bc7d6251fd309da26b65b5a43"
Sep 07 13:19:16.215280 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:19:16.215254    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd35e05c27981ad4f3e7c84d94ff75b3ab1906bc7d6251fd309da26b65b5a43\": container with ID starting with afd35e05c27981ad4f3e7c84d94ff75b3ab1906bc7d6251fd309da26b65b5a43 not found: ID does not exist" containerID="afd35e05c27981ad4f3e7c84d94ff75b3ab1906bc7d6251fd309da26b65b5a43"
Sep 07 13:19:16.215383 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.215288    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd35e05c27981ad4f3e7c84d94ff75b3ab1906bc7d6251fd309da26b65b5a43"} err="failed to get container status \"afd35e05c27981ad4f3e7c84d94ff75b3ab1906bc7d6251fd309da26b65b5a43\": rpc error: code = NotFound desc = could not find container \"afd35e05c27981ad4f3e7c84d94ff75b3ab1906bc7d6251fd309da26b65b5a43\": container with ID starting with afd35e05c27981ad4f3e7c84d94ff75b3ab1906bc7d6251fd309da26b65b5a43 not found: ID does not exist"
Sep 07 13:19:16.293740 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.293692    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5992306f-a972-4231-a79c-6ccb291b5663-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:19:16.293740 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.293734    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:19:16.293740 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.293748    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5992306f-a972-4231-a79c-6ccb291b5663-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:19:16.293996 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.293760    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlb5p\" (UniqueName: \"kubernetes.io/projected/5992306f-a972-4231-a79c-6ccb291b5663-kube-api-access-jlb5p\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:19:16.517456 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.517422    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"]
Sep 07 13:19:16.524479 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:16.524450    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3dd3e936-7946fc9665-5r2pz"]
Sep 07 13:19:18.074876 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:18.074837    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5992306f-a972-4231-a79c-6ccb291b5663" path="/var/lib/kubelet/pods/5992306f-a972-4231-a79c-6ccb291b5663/volumes"
Sep 07 13:19:24.534929 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.534895    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"]
Sep 07 13:19:24.535422 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.535354    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5992306f-a972-4231-a79c-6ccb291b5663" containerName="maas-api"
Sep 07 13:19:24.535422 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.535375    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5992306f-a972-4231-a79c-6ccb291b5663" containerName="maas-api"
Sep 07 13:19:24.535589 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.535505    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5992306f-a972-4231-a79c-6ccb291b5663" containerName="maas-api"
Sep 07 13:19:24.540346 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.540322    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:24.543800 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.543776    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-c7c371-serving-cert\""
Sep 07 13:19:24.563819 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.563780    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"]
Sep 07 13:19:24.572814 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.572788    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:24.572948 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.572822    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f02309-4419-4035-999e-4eff749cc77b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:24.572948 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.572855    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:24.572948 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.572941    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2q8\" (UniqueName: \"kubernetes.io/projected/41f02309-4419-4035-999e-4eff749cc77b-kube-api-access-tk2q8\") pod \"maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:24.673882 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.673834    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:24.674085 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.673888    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f02309-4419-4035-999e-4eff749cc77b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:24.674085 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.673931    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:24.674085 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:19:24.674003    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-serving-cert: secret "maas-api-e2e-shared-b-w3-c7c371-serving-cert" not found
Sep 07 13:19:24.674265 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:19:24.674097    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-maas-api-tls podName:41f02309-4419-4035-999e-4eff749cc77b nodeName:}" failed. No retries permitted until 2026-09-07 13:19:25.174073781 +0000 UTC m=+1209.602372047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-maas-api-tls") pod "maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44" (UID: "41f02309-4419-4035-999e-4eff749cc77b") : secret "maas-api-e2e-shared-b-w3-c7c371-serving-cert" not found
Sep 07 13:19:24.674265 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.674012    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2q8\" (UniqueName: \"kubernetes.io/projected/41f02309-4419-4035-999e-4eff749cc77b-kube-api-access-tk2q8\") pod \"maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:24.674690 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.674666    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f02309-4419-4035-999e-4eff749cc77b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:24.677319 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.677294    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:24.696990 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:24.696956    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2q8\" (UniqueName: \"kubernetes.io/projected/41f02309-4419-4035-999e-4eff749cc77b-kube-api-access-tk2q8\") pod \"maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:25.180701 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:25.180658    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:25.183135 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:25.183108    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:25.451228 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:25.451138    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:25.765185 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:25.765140    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"]
Sep 07 13:19:25.768115 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:19:25.768050    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f02309_4419_4035_999e_4eff749cc77b.slice/crio-be7d83e8bc0a23654dbc1c63886ba42880be9b510f02286fdd9d73bdd89ad19c WatchSource:0}: Error finding container be7d83e8bc0a23654dbc1c63886ba42880be9b510f02286fdd9d73bdd89ad19c: Status 404 returned error can't find the container with id be7d83e8bc0a23654dbc1c63886ba42880be9b510f02286fdd9d73bdd89ad19c
Sep 07 13:19:26.253913 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:26.253884    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44" event={"ID":"41f02309-4419-4035-999e-4eff749cc77b","Type":"ContainerStarted","Data":"be7d83e8bc0a23654dbc1c63886ba42880be9b510f02286fdd9d73bdd89ad19c"}
Sep 07 13:19:27.261103 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:27.261040    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44" event={"ID":"41f02309-4419-4035-999e-4eff749cc77b","Type":"ContainerStarted","Data":"824786c296df8184d29a3283adbc4b29ae69dd244a1b20343de7857067902f8d"}
Sep 07 13:19:27.261514 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:27.261257    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:27.324442 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:27.324380    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44" podStartSLOduration=2.8757443240000002 podStartE2EDuration="3.324364655s" totalImagesPullingTime="448.620331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:19:27.322492728 +0000 UTC m=+1211.750791001" watchObservedRunningTime="2026-09-07 13:19:27.324364655 +0000 UTC m=+1211.752662952"
Sep 07 13:19:33.272310 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:33.272278    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:19:36.833441 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:36.833405    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"]
Sep 07 13:19:36.833896 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:36.833673    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t" podUID="12e767f6-7128-4b09-9ec4-28ecd555a97f" containerName="istio-proxy" containerID="cri-o://94b135b0cb6eea31d46c82cb6e643a8e1bf7d6e53342f6f06c97d47128dc8dcc" gracePeriod=30
Sep 07 13:19:42.078992 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.078958    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:19:42.124149 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.124050    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-data\") pod \"12e767f6-7128-4b09-9ec4-28ecd555a97f\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") "
Sep 07 13:19:42.124149 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.124123    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d8tb\" (UniqueName: \"kubernetes.io/projected/12e767f6-7128-4b09-9ec4-28ecd555a97f-kube-api-access-4d8tb\") pod \"12e767f6-7128-4b09-9ec4-28ecd555a97f\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") "
Sep 07 13:19:42.124329 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.124184    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-podinfo\") pod \"12e767f6-7128-4b09-9ec4-28ecd555a97f\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") "
Sep 07 13:19:42.124329 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.124243    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-workload-certs\") pod \"12e767f6-7128-4b09-9ec4-28ecd555a97f\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") "
Sep 07 13:19:42.124329 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.124292    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e767f6-7128-4b09-9ec4-28ecd555a97f-istiod-ca-cert\") pod \"12e767f6-7128-4b09-9ec4-28ecd555a97f\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") "
Sep 07 13:19:42.124329 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.124322    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-token\") pod \"12e767f6-7128-4b09-9ec4-28ecd555a97f\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") "
Sep 07 13:19:42.124524 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.124349    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-workload-socket\") pod \"12e767f6-7128-4b09-9ec4-28ecd555a97f\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") "
Sep 07 13:19:42.124524 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.124347    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-data" (OuterVolumeSpecName: "istio-data") pod "12e767f6-7128-4b09-9ec4-28ecd555a97f" (UID: "12e767f6-7128-4b09-9ec4-28ecd555a97f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:19:42.124524 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.124384    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-credential-socket\") pod \"12e767f6-7128-4b09-9ec4-28ecd555a97f\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") "
Sep 07 13:19:42.124524 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.124419    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-envoy\") pod \"12e767f6-7128-4b09-9ec4-28ecd555a97f\" (UID: \"12e767f6-7128-4b09-9ec4-28ecd555a97f\") "
Sep 07 13:19:42.124743 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.124533    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "12e767f6-7128-4b09-9ec4-28ecd555a97f" (UID: "12e767f6-7128-4b09-9ec4-28ecd555a97f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:19:42.124802 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.124735    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "12e767f6-7128-4b09-9ec4-28ecd555a97f" (UID: "12e767f6-7128-4b09-9ec4-28ecd555a97f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:19:42.125144 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.125085    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "12e767f6-7128-4b09-9ec4-28ecd555a97f" (UID: "12e767f6-7128-4b09-9ec4-28ecd555a97f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:19:42.125144 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.124941    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-workload-socket\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:19:42.125337 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.125159    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-data\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:19:42.125337 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.125176    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-workload-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:19:42.125644 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.125612    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e767f6-7128-4b09-9ec4-28ecd555a97f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "12e767f6-7128-4b09-9ec4-28ecd555a97f" (UID: "12e767f6-7128-4b09-9ec4-28ecd555a97f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:19:42.126848 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.126820    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-token" (OuterVolumeSpecName: "istio-token") pod "12e767f6-7128-4b09-9ec4-28ecd555a97f" (UID: "12e767f6-7128-4b09-9ec4-28ecd555a97f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:19:42.126985 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.126961    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e767f6-7128-4b09-9ec4-28ecd555a97f-kube-api-access-4d8tb" (OuterVolumeSpecName: "kube-api-access-4d8tb") pod "12e767f6-7128-4b09-9ec4-28ecd555a97f" (UID: "12e767f6-7128-4b09-9ec4-28ecd555a97f"). InnerVolumeSpecName "kube-api-access-4d8tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:19:42.126985 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.126964    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "12e767f6-7128-4b09-9ec4-28ecd555a97f" (UID: "12e767f6-7128-4b09-9ec4-28ecd555a97f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:19:42.127204 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.127098    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "12e767f6-7128-4b09-9ec4-28ecd555a97f" (UID: "12e767f6-7128-4b09-9ec4-28ecd555a97f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:19:42.226328 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.226285    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-envoy\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:19:42.226328 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.226322    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d8tb\" (UniqueName: \"kubernetes.io/projected/12e767f6-7128-4b09-9ec4-28ecd555a97f-kube-api-access-4d8tb\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:19:42.226328 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.226333    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-podinfo\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:19:42.226570 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.226341    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e767f6-7128-4b09-9ec4-28ecd555a97f-istiod-ca-cert\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:19:42.226570 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.226349    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e767f6-7128-4b09-9ec4-28ecd555a97f-istio-token\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:19:42.226570 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.226357    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e767f6-7128-4b09-9ec4-28ecd555a97f-credential-socket\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:19:42.355292 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.355246    2567 generic.go:356] "Generic (PLEG): container finished" podID="12e767f6-7128-4b09-9ec4-28ecd555a97f" containerID="94b135b0cb6eea31d46c82cb6e643a8e1bf7d6e53342f6f06c97d47128dc8dcc" exitCode=0
Sep 07 13:19:42.355292 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.355299    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"
Sep 07 13:19:42.355500 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.355338    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t" event={"ID":"12e767f6-7128-4b09-9ec4-28ecd555a97f","Type":"ContainerDied","Data":"94b135b0cb6eea31d46c82cb6e643a8e1bf7d6e53342f6f06c97d47128dc8dcc"}
Sep 07 13:19:42.355500 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.355394    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t" event={"ID":"12e767f6-7128-4b09-9ec4-28ecd555a97f","Type":"ContainerDied","Data":"bbc0d6de6424cf7d92431d9066395c4bbe422c30e79f35fd6b612d9749805f44"}
Sep 07 13:19:42.355500 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.355416    2567 scope.go:117] "RemoveContainer" containerID="94b135b0cb6eea31d46c82cb6e643a8e1bf7d6e53342f6f06c97d47128dc8dcc"
Sep 07 13:19:42.365153 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.365135    2567 scope.go:117] "RemoveContainer" containerID="94b135b0cb6eea31d46c82cb6e643a8e1bf7d6e53342f6f06c97d47128dc8dcc"
Sep 07 13:19:42.365420 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:19:42.365400    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b135b0cb6eea31d46c82cb6e643a8e1bf7d6e53342f6f06c97d47128dc8dcc\": container with ID starting with 94b135b0cb6eea31d46c82cb6e643a8e1bf7d6e53342f6f06c97d47128dc8dcc not found: ID does not exist" containerID="94b135b0cb6eea31d46c82cb6e643a8e1bf7d6e53342f6f06c97d47128dc8dcc"
Sep 07 13:19:42.365468 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.365430    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b135b0cb6eea31d46c82cb6e643a8e1bf7d6e53342f6f06c97d47128dc8dcc"} err="failed to get container status \"94b135b0cb6eea31d46c82cb6e643a8e1bf7d6e53342f6f06c97d47128dc8dcc\": rpc error: code = NotFound desc = could not find container \"94b135b0cb6eea31d46c82cb6e643a8e1bf7d6e53342f6f06c97d47128dc8dcc\": container with ID starting with 94b135b0cb6eea31d46c82cb6e643a8e1bf7d6e53342f6f06c97d47128dc8dcc not found: ID does not exist"
Sep 07 13:19:42.379264 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.379205    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"]
Sep 07 13:19:42.383799 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:42.383772    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-067a48-openshift-default-768b86f464-vvk9t"]
Sep 07 13:19:44.075266 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.075223    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12e767f6-7128-4b09-9ec4-28ecd555a97f" path="/var/lib/kubelet/pods/12e767f6-7128-4b09-9ec4-28ecd555a97f/volumes"
Sep 07 13:19:44.484544 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.484453    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"]
Sep 07 13:19:44.485096 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.485046    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12e767f6-7128-4b09-9ec4-28ecd555a97f" containerName="istio-proxy"
Sep 07 13:19:44.485096 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.485093    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="12e767f6-7128-4b09-9ec4-28ecd555a97f" containerName="istio-proxy"
Sep 07 13:19:44.485289 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.485206    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="12e767f6-7128-4b09-9ec4-28ecd555a97f" containerName="istio-proxy"
Sep 07 13:19:44.488503 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.488481    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:44.490622 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.490602    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-d5da3f-serving-cert\""
Sep 07 13:19:44.506297 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.506269    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"]
Sep 07 13:19:44.548543 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.548510    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scnhz\" (UniqueName: \"kubernetes.io/projected/3f644a7a-56bd-4026-8df6-c4b86ee680ff-kube-api-access-scnhz\") pod \"maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:44.548693 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.548580    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f644a7a-56bd-4026-8df6-c4b86ee680ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:44.548693 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.548621    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:44.548693 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.548644    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:44.649978 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.649934    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scnhz\" (UniqueName: \"kubernetes.io/projected/3f644a7a-56bd-4026-8df6-c4b86ee680ff-kube-api-access-scnhz\") pod \"maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:44.650199 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.650028    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f644a7a-56bd-4026-8df6-c4b86ee680ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:44.650199 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.650055    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:44.650199 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.650111    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:44.650362 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:19:44.650199    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-serving-cert: secret "maas-api-e2e-shared-a-w2-d5da3f-serving-cert" not found
Sep 07 13:19:44.650362 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:19:44.650268    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-maas-api-tls podName:3f644a7a-56bd-4026-8df6-c4b86ee680ff nodeName:}" failed. No retries permitted until 2026-09-07 13:19:45.150247646 +0000 UTC m=+1229.578545901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-maas-api-tls") pod "maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g" (UID: "3f644a7a-56bd-4026-8df6-c4b86ee680ff") : secret "maas-api-e2e-shared-a-w2-d5da3f-serving-cert" not found
Sep 07 13:19:44.650748 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.650726    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f644a7a-56bd-4026-8df6-c4b86ee680ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:44.652692 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.652668    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:44.660912 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:44.660889    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scnhz\" (UniqueName: \"kubernetes.io/projected/3f644a7a-56bd-4026-8df6-c4b86ee680ff-kube-api-access-scnhz\") pod \"maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:45.155846 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.155798    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:45.158277 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.158255    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:45.399890 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.399854    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:45.505212 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.505169    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"]
Sep 07 13:19:45.509709 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.509688    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"
Sep 07 13:19:45.511836 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.511808    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d5da3f-dockercfg-bh59w\""
Sep 07 13:19:45.511836 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.511828    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d5da3f\""
Sep 07 13:19:45.521668 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.521644    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"]
Sep 07 13:19:45.559550 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.559519    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70883334-79e1-41c2-9668-c9dfddf251c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9\" (UID: \"70883334-79e1-41c2-9668-c9dfddf251c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"
Sep 07 13:19:45.559692 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.559586    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtklx\" (UniqueName: \"kubernetes.io/projected/70883334-79e1-41c2-9668-c9dfddf251c1-kube-api-access-xtklx\") pod \"payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9\" (UID: \"70883334-79e1-41c2-9668-c9dfddf251c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"
Sep 07 13:19:45.661284 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.661204    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtklx\" (UniqueName: \"kubernetes.io/projected/70883334-79e1-41c2-9668-c9dfddf251c1-kube-api-access-xtklx\") pod \"payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9\" (UID: \"70883334-79e1-41c2-9668-c9dfddf251c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"
Sep 07 13:19:45.661438 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.661295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70883334-79e1-41c2-9668-c9dfddf251c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9\" (UID: \"70883334-79e1-41c2-9668-c9dfddf251c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"
Sep 07 13:19:45.661904 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.661884    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70883334-79e1-41c2-9668-c9dfddf251c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9\" (UID: \"70883334-79e1-41c2-9668-c9dfddf251c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"
Sep 07 13:19:45.673880 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.673851    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtklx\" (UniqueName: \"kubernetes.io/projected/70883334-79e1-41c2-9668-c9dfddf251c1-kube-api-access-xtklx\") pod \"payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9\" (UID: \"70883334-79e1-41c2-9668-c9dfddf251c1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"
Sep 07 13:19:45.678484 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.678460    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"]
Sep 07 13:19:45.680382 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:19:45.680354    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f644a7a_56bd_4026_8df6_c4b86ee680ff.slice/crio-ce7ad34a09e0c5a770bdb7204108c350f26bd6e865c542736236637f55dfc0d6 WatchSource:0}: Error finding container ce7ad34a09e0c5a770bdb7204108c350f26bd6e865c542736236637f55dfc0d6: Status 404 returned error can't find the container with id ce7ad34a09e0c5a770bdb7204108c350f26bd6e865c542736236637f55dfc0d6
Sep 07 13:19:45.820991 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:45.820955    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"
Sep 07 13:19:46.105609 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:46.105563    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"]
Sep 07 13:19:46.107319 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:19:46.107278    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70883334_79e1_41c2_9668_c9dfddf251c1.slice/crio-0c4fc33266476dbc37447f3f69c71f02666826b10f3f07cf9d89519a8ab1a409 WatchSource:0}: Error finding container 0c4fc33266476dbc37447f3f69c71f02666826b10f3f07cf9d89519a8ab1a409: Status 404 returned error can't find the container with id 0c4fc33266476dbc37447f3f69c71f02666826b10f3f07cf9d89519a8ab1a409
Sep 07 13:19:46.381557 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:46.381458    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g" event={"ID":"3f644a7a-56bd-4026-8df6-c4b86ee680ff","Type":"ContainerStarted","Data":"ce7ad34a09e0c5a770bdb7204108c350f26bd6e865c542736236637f55dfc0d6"}
Sep 07 13:19:46.383323 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:46.383288    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9" event={"ID":"70883334-79e1-41c2-9668-c9dfddf251c1","Type":"ContainerStarted","Data":"e5dfe890b7653508987d2abc52967d9270497ae162a3c60dedd370d9c9f1b5d7"}
Sep 07 13:19:46.383455 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:46.383347    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9" event={"ID":"70883334-79e1-41c2-9668-c9dfddf251c1","Type":"ContainerStarted","Data":"0c4fc33266476dbc37447f3f69c71f02666826b10f3f07cf9d89519a8ab1a409"}
Sep 07 13:19:46.383527 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:46.383470    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"
Sep 07 13:19:46.405694 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:46.405640    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9" podStartSLOduration=1.405624175 podStartE2EDuration="1.405624175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:19:46.404553903 +0000 UTC m=+1230.832852199" watchObservedRunningTime="2026-09-07 13:19:46.405624175 +0000 UTC m=+1230.833922448"
Sep 07 13:19:47.391161 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:47.391115    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g" event={"ID":"3f644a7a-56bd-4026-8df6-c4b86ee680ff","Type":"ContainerStarted","Data":"24e6b6d747e7ecd76d4fb2de60dd01402c22e811d8356db84f82551e37c90ee8"}
Sep 07 13:19:47.391628 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:47.391336    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:47.413483 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:47.413426    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g" podStartSLOduration=2.479233414 podStartE2EDuration="3.413410087s" totalImagesPullingTime="934.176673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:19:47.409587515 +0000 UTC m=+1231.837885805" watchObservedRunningTime="2026-09-07 13:19:47.413410087 +0000 UTC m=+1231.841708360"
Sep 07 13:19:53.404623 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:53.404592    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:19:57.393502 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:19:57.393418    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"
Sep 07 13:20:13.566806 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.566690    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"]
Sep 07 13:20:13.570991 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.570971    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:13.573503 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.573480    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-8e7ddc-serving-cert\""
Sep 07 13:20:13.584665 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.584638    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"]
Sep 07 13:20:13.621310 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.621274    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:13.621494 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.621328    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:13.621494 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.621394    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw2h8\" (UniqueName: \"kubernetes.io/projected/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-kube-api-access-tw2h8\") pod \"maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:13.621494 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.621450    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:13.722500 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.722458    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw2h8\" (UniqueName: \"kubernetes.io/projected/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-kube-api-access-tw2h8\") pod \"maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:13.722712 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.722521    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:13.722712 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.722567    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:13.722712 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.722592    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:13.722712 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:20:13.722686    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-serving-cert: secret "maas-api-e2e-shared-b-w2-8e7ddc-serving-cert" not found
Sep 07 13:20:13.722945 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:20:13.722780    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-maas-api-tls podName:f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1 nodeName:}" failed. No retries permitted until 2026-09-07 13:20:14.222759089 +0000 UTC m=+1258.651057354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-maas-api-tls") pod "maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b" (UID: "f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1") : secret "maas-api-e2e-shared-b-w2-8e7ddc-serving-cert" not found
Sep 07 13:20:13.723343 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.723323    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:13.725071 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.725042    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:13.749885 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:13.749853    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw2h8\" (UniqueName: \"kubernetes.io/projected/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-kube-api-access-tw2h8\") pod \"maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:14.225815 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:14.225770    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:14.228352 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:14.228325    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:14.484357 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:14.484228    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:14.784741 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:14.784708    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"]
Sep 07 13:20:14.786070 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:20:14.786021    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8962cd5_0271_4a1d_b92e_1c1c43e1e9e1.slice/crio-c7020925927d6222089e06dbb5d1ed746c43dacdac335298acbe95ce9a1de43d WatchSource:0}: Error finding container c7020925927d6222089e06dbb5d1ed746c43dacdac335298acbe95ce9a1de43d: Status 404 returned error can't find the container with id c7020925927d6222089e06dbb5d1ed746c43dacdac335298acbe95ce9a1de43d
Sep 07 13:20:15.573333 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:15.573287    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b" event={"ID":"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1","Type":"ContainerStarted","Data":"0799ebdab5f6a296102c5f164d4b322076796b37868ff4d5b70745fc7fde1e95"}
Sep 07 13:20:15.573511 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:15.573390    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b" event={"ID":"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1","Type":"ContainerStarted","Data":"c7020925927d6222089e06dbb5d1ed746c43dacdac335298acbe95ce9a1de43d"}
Sep 07 13:20:15.573511 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:15.573421    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:15.596581 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:15.596523    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b" podStartSLOduration=2.030361377 podStartE2EDuration="2.596507637s" totalImagesPullingTime="566.14626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:20:15.593323566 +0000 UTC m=+1260.021621863" watchObservedRunningTime="2026-09-07 13:20:15.596507637 +0000 UTC m=+1260.024805910"
Sep 07 13:20:21.585129 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:21.585098    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:20:30.547222 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.547190    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"]
Sep 07 13:20:30.552290 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.552267    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.556360 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.556341    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c7c371\"/\"default-dockercfg-gc9sj\""
Sep 07 13:20:30.556862 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.556837    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c7c371\"/\"openshift-service-ca.crt\""
Sep 07 13:20:30.556919 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.556901    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c7c371\"/\"kube-root-ca.crt\""
Sep 07 13:20:30.557309 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.557290    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c7c371\"/\"auth-test-model-c7c371-kserve-self-signed-certs\""
Sep 07 13:20:30.566573 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.566551    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"]
Sep 07 13:20:30.682837 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.682808    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-tls-certs\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.683012 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.682841    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-home\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.683012 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.682862    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsjcq\" (UniqueName: \"kubernetes.io/projected/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-kube-api-access-xsjcq\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.683012 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.682972    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-tmp-dir\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.683012 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.682988    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-model-cache\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.683175 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.683119    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-dshm\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.784397 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.784365    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-tmp-dir\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.784600 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.784498    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-model-cache\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.784677 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.784623    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-dshm\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.784677 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.784668    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-tls-certs\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.784788 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.784699    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-home\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.784788 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.784729    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsjcq\" (UniqueName: \"kubernetes.io/projected/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-kube-api-access-xsjcq\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.784788 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.784749    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-tmp-dir\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.784788 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.784774    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-model-cache\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.785010 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.784995    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-home\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.786939 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.786918    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-dshm\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.787261 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.787241    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-tls-certs\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.793880 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.793855    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsjcq\" (UniqueName: \"kubernetes.io/projected/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-kube-api-access-xsjcq\") pod \"auth-test-model-c7c371-kserve-694c6d7775-5bhvj\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:30.861992 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:30.861924    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:31.139762 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:31.139729    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"]
Sep 07 13:20:31.141326 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:20:31.141301    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbc38af4_1439_4d9d_87ee_3c6a983bdfa1.slice/crio-3bc2b01f8dd86e1465d5597e12dd9e1bfb816a4ffbd9d8f0578c1844ebdc6526 WatchSource:0}: Error finding container 3bc2b01f8dd86e1465d5597e12dd9e1bfb816a4ffbd9d8f0578c1844ebdc6526: Status 404 returned error can't find the container with id 3bc2b01f8dd86e1465d5597e12dd9e1bfb816a4ffbd9d8f0578c1844ebdc6526
Sep 07 13:20:31.669155 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:31.669118    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj" event={"ID":"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1","Type":"ContainerStarted","Data":"68065d2e6d26e1ae1093b1e5683a3ca099babc7ea0037d82c4ebe26abf287aa1"}
Sep 07 13:20:31.669536 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:31.669167    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj" event={"ID":"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1","Type":"ContainerStarted","Data":"3bc2b01f8dd86e1465d5597e12dd9e1bfb816a4ffbd9d8f0578c1844ebdc6526"}
Sep 07 13:20:31.694369 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:31.694320    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj" podStartSLOduration=1.6943037319999998 podStartE2EDuration="1.694303732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:20:31.692018497 +0000 UTC m=+1276.120316812" watchObservedRunningTime="2026-09-07 13:20:31.694303732 +0000 UTC m=+1276.122602004"
Sep 07 13:20:40.862352 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:40.862307    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:40.862352 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:40.862349    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:40.874885 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:40.874862    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:20:41.737027 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:20:41.736993    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:21:00.910891 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:00.910850    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"]
Sep 07 13:21:00.914822 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:00.914798    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:00.918244 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:00.918220    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ede2050e-serving-cert\""
Sep 07 13:21:00.927564 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:00.927481    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"]
Sep 07 13:21:00.960293 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:00.960263    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-metrics-certs\") pod \"maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:00.960450 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:00.960306    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2421b4-c704-4085-b44b-f5d596be5a91-openshift-service-ca\") pod \"maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:00.960450 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:00.960335    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-maas-api-tls\") pod \"maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:00.960450 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:00.960420    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qng95\" (UniqueName: \"kubernetes.io/projected/1c2421b4-c704-4085-b44b-f5d596be5a91-kube-api-access-qng95\") pod \"maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:01.061658 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.061620    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-metrics-certs\") pod \"maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:01.061836 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.061673    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2421b4-c704-4085-b44b-f5d596be5a91-openshift-service-ca\") pod \"maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:01.061836 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.061706    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-maas-api-tls\") pod \"maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:01.061836 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.061735    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qng95\" (UniqueName: \"kubernetes.io/projected/1c2421b4-c704-4085-b44b-f5d596be5a91-kube-api-access-qng95\") pod \"maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:01.062009 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:21:01.061873    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-serving-cert: secret "maas-api-e2e-mt-ede2050e-serving-cert" not found
Sep 07 13:21:01.062009 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:21:01.061965    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-maas-api-tls podName:1c2421b4-c704-4085-b44b-f5d596be5a91 nodeName:}" failed. No retries permitted until 2026-09-07 13:21:01.561943644 +0000 UTC m=+1305.990241912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-maas-api-tls") pod "maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw" (UID: "1c2421b4-c704-4085-b44b-f5d596be5a91") : secret "maas-api-e2e-mt-ede2050e-serving-cert" not found
Sep 07 13:21:01.062489 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.062470    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2421b4-c704-4085-b44b-f5d596be5a91-openshift-service-ca\") pod \"maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:01.064137 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.064114    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-metrics-certs\") pod \"maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:01.071359 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.071335    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qng95\" (UniqueName: \"kubernetes.io/projected/1c2421b4-c704-4085-b44b-f5d596be5a91-kube-api-access-qng95\") pod \"maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:01.567554 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.567509    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-maas-api-tls\") pod \"maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:01.569921 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.569896    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-maas-api-tls\") pod \"maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:01.750826 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.750789    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"]
Sep 07 13:21:01.754570 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.754554    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"
Sep 07 13:21:01.757855 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.757835    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ede2050e-dockercfg-pm78b\""
Sep 07 13:21:01.757919 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.757877    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ede2050e\""
Sep 07 13:21:01.774426 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.774398    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"]
Sep 07 13:21:01.828857 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.828772    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:01.870795 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.870756    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r7w5\" (UniqueName: \"kubernetes.io/projected/9a253fda-2139-4fad-8704-a51fee5df9fe-kube-api-access-9r7w5\") pod \"payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv\" (UID: \"9a253fda-2139-4fad-8704-a51fee5df9fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"
Sep 07 13:21:01.870980 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.870920    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a253fda-2139-4fad-8704-a51fee5df9fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv\" (UID: \"9a253fda-2139-4fad-8704-a51fee5df9fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"
Sep 07 13:21:01.972346 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.972307    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a253fda-2139-4fad-8704-a51fee5df9fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv\" (UID: \"9a253fda-2139-4fad-8704-a51fee5df9fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"
Sep 07 13:21:01.972811 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.972428    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7w5\" (UniqueName: \"kubernetes.io/projected/9a253fda-2139-4fad-8704-a51fee5df9fe-kube-api-access-9r7w5\") pod \"payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv\" (UID: \"9a253fda-2139-4fad-8704-a51fee5df9fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"
Sep 07 13:21:01.973700 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.973672    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a253fda-2139-4fad-8704-a51fee5df9fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv\" (UID: \"9a253fda-2139-4fad-8704-a51fee5df9fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"
Sep 07 13:21:01.982607 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:01.982552    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7w5\" (UniqueName: \"kubernetes.io/projected/9a253fda-2139-4fad-8704-a51fee5df9fe-kube-api-access-9r7w5\") pod \"payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv\" (UID: \"9a253fda-2139-4fad-8704-a51fee5df9fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"
Sep 07 13:21:02.065247 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:02.065212    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"
Sep 07 13:21:02.125578 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:02.125331    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"]
Sep 07 13:21:02.125973 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:21:02.125938    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2421b4_c704_4085_b44b_f5d596be5a91.slice/crio-873a0087ce928cd53eee5078cac7efcb423a51aa61f737e386ee1bdbbb4dfd9d WatchSource:0}: Error finding container 873a0087ce928cd53eee5078cac7efcb423a51aa61f737e386ee1bdbbb4dfd9d: Status 404 returned error can't find the container with id 873a0087ce928cd53eee5078cac7efcb423a51aa61f737e386ee1bdbbb4dfd9d
Sep 07 13:21:02.361999 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:02.361976    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"]
Sep 07 13:21:02.363822 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:21:02.363795    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a253fda_2139_4fad_8704_a51fee5df9fe.slice/crio-0c0c5de372d1fe39dc6e8c0bef8ec18536be25b9a0dc655824329209af80e683 WatchSource:0}: Error finding container 0c0c5de372d1fe39dc6e8c0bef8ec18536be25b9a0dc655824329209af80e683: Status 404 returned error can't find the container with id 0c0c5de372d1fe39dc6e8c0bef8ec18536be25b9a0dc655824329209af80e683
Sep 07 13:21:02.846455 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:02.846415    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv" event={"ID":"9a253fda-2139-4fad-8704-a51fee5df9fe","Type":"ContainerStarted","Data":"b1a371ba0a6fc9502574bda9151b8e8a24becff53228388c0c79cda0b5063db6"}
Sep 07 13:21:02.846581 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:02.846484    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv" event={"ID":"9a253fda-2139-4fad-8704-a51fee5df9fe","Type":"ContainerStarted","Data":"0c0c5de372d1fe39dc6e8c0bef8ec18536be25b9a0dc655824329209af80e683"}
Sep 07 13:21:02.846668 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:02.846583    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"
Sep 07 13:21:02.848277 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:02.848251    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw" event={"ID":"1c2421b4-c704-4085-b44b-f5d596be5a91","Type":"ContainerStarted","Data":"f141301e50b185c5d19af3372e3be7ce1856c452c21bcec05aec9f62b5bf332f"}
Sep 07 13:21:02.848374 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:02.848298    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw" event={"ID":"1c2421b4-c704-4085-b44b-f5d596be5a91","Type":"ContainerStarted","Data":"873a0087ce928cd53eee5078cac7efcb423a51aa61f737e386ee1bdbbb4dfd9d"}
Sep 07 13:21:02.848374 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:02.848337    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:02.869091 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:02.869034    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv" podStartSLOduration=1.869021124 podStartE2EDuration="1.869021124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:21:02.86540426 +0000 UTC m=+1307.293702533" watchObservedRunningTime="2026-09-07 13:21:02.869021124 +0000 UTC m=+1307.297319406"
Sep 07 13:21:02.890283 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:02.890240    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw" podStartSLOduration=2.2507008600000002 podStartE2EDuration="2.890224296s" totalImagesPullingTime="639.523436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:21:02.887655675 +0000 UTC m=+1307.315953947" watchObservedRunningTime="2026-09-07 13:21:02.890224296 +0000 UTC m=+1307.318522627"
Sep 07 13:21:08.860548 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:08.860517    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:21:13.857261 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:21:13.857222    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"
Sep 07 13:22:25.102772 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.102737    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"]
Sep 07 13:22:25.103281 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.102934    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw" podUID="1c2421b4-c704-4085-b44b-f5d596be5a91" containerName="maas-api" containerID="cri-o://f141301e50b185c5d19af3372e3be7ce1856c452c21bcec05aec9f62b5bf332f" gracePeriod=30
Sep 07 13:22:25.198186 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.198154    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"]
Sep 07 13:22:25.198397 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.198368    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv" podUID="9a253fda-2139-4fad-8704-a51fee5df9fe" containerName="payload-pre-processing" containerID="cri-o://b1a371ba0a6fc9502574bda9151b8e8a24becff53228388c0c79cda0b5063db6" gracePeriod=30
Sep 07 13:22:25.346282 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.346244    2567 generic.go:356] "Generic (PLEG): container finished" podID="1c2421b4-c704-4085-b44b-f5d596be5a91" containerID="f141301e50b185c5d19af3372e3be7ce1856c452c21bcec05aec9f62b5bf332f" exitCode=0
Sep 07 13:22:25.346466 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.346324    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw" event={"ID":"1c2421b4-c704-4085-b44b-f5d596be5a91","Type":"ContainerDied","Data":"f141301e50b185c5d19af3372e3be7ce1856c452c21bcec05aec9f62b5bf332f"}
Sep 07 13:22:25.348372 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.348347    2567 generic.go:356] "Generic (PLEG): container finished" podID="9a253fda-2139-4fad-8704-a51fee5df9fe" containerID="b1a371ba0a6fc9502574bda9151b8e8a24becff53228388c0c79cda0b5063db6" exitCode=0
Sep 07 13:22:25.348495 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.348383    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv" event={"ID":"9a253fda-2139-4fad-8704-a51fee5df9fe","Type":"ContainerDied","Data":"b1a371ba0a6fc9502574bda9151b8e8a24becff53228388c0c79cda0b5063db6"}
Sep 07 13:22:25.387616 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.387292    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:22:25.484493 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.484468    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"
Sep 07 13:22:25.494718 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.494695    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2421b4-c704-4085-b44b-f5d596be5a91-openshift-service-ca\") pod \"1c2421b4-c704-4085-b44b-f5d596be5a91\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") "
Sep 07 13:22:25.494858 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.494748    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-maas-api-tls\") pod \"1c2421b4-c704-4085-b44b-f5d596be5a91\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") "
Sep 07 13:22:25.494858 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.494766    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-metrics-certs\") pod \"1c2421b4-c704-4085-b44b-f5d596be5a91\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") "
Sep 07 13:22:25.494858 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.494806    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qng95\" (UniqueName: \"kubernetes.io/projected/1c2421b4-c704-4085-b44b-f5d596be5a91-kube-api-access-qng95\") pod \"1c2421b4-c704-4085-b44b-f5d596be5a91\" (UID: \"1c2421b4-c704-4085-b44b-f5d596be5a91\") "
Sep 07 13:22:25.495132 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.495109    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2421b4-c704-4085-b44b-f5d596be5a91-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c2421b4-c704-4085-b44b-f5d596be5a91" (UID: "1c2421b4-c704-4085-b44b-f5d596be5a91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:22:25.496896 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.496868    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c2421b4-c704-4085-b44b-f5d596be5a91" (UID: "1c2421b4-c704-4085-b44b-f5d596be5a91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:22:25.497142 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.497115    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1c2421b4-c704-4085-b44b-f5d596be5a91" (UID: "1c2421b4-c704-4085-b44b-f5d596be5a91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:22:25.497243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.497159    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2421b4-c704-4085-b44b-f5d596be5a91-kube-api-access-qng95" (OuterVolumeSpecName: "kube-api-access-qng95") pod "1c2421b4-c704-4085-b44b-f5d596be5a91" (UID: "1c2421b4-c704-4085-b44b-f5d596be5a91"). InnerVolumeSpecName "kube-api-access-qng95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:22:25.595417 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.595377    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a253fda-2139-4fad-8704-a51fee5df9fe-plugins-config-volume\") pod \"9a253fda-2139-4fad-8704-a51fee5df9fe\" (UID: \"9a253fda-2139-4fad-8704-a51fee5df9fe\") "
Sep 07 13:22:25.595590 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.595529    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r7w5\" (UniqueName: \"kubernetes.io/projected/9a253fda-2139-4fad-8704-a51fee5df9fe-kube-api-access-9r7w5\") pod \"9a253fda-2139-4fad-8704-a51fee5df9fe\" (UID: \"9a253fda-2139-4fad-8704-a51fee5df9fe\") "
Sep 07 13:22:25.595721 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.595710    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:22:25.595763 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.595722    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2421b4-c704-4085-b44b-f5d596be5a91-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:22:25.595763 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.595732    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qng95\" (UniqueName: \"kubernetes.io/projected/1c2421b4-c704-4085-b44b-f5d596be5a91-kube-api-access-qng95\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:22:25.595763 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.595743    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2421b4-c704-4085-b44b-f5d596be5a91-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:22:25.595909 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.595755    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a253fda-2139-4fad-8704-a51fee5df9fe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a253fda-2139-4fad-8704-a51fee5df9fe" (UID: "9a253fda-2139-4fad-8704-a51fee5df9fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:22:25.597565 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.597546    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a253fda-2139-4fad-8704-a51fee5df9fe-kube-api-access-9r7w5" (OuterVolumeSpecName: "kube-api-access-9r7w5") pod "9a253fda-2139-4fad-8704-a51fee5df9fe" (UID: "9a253fda-2139-4fad-8704-a51fee5df9fe"). InnerVolumeSpecName "kube-api-access-9r7w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:22:25.697102 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.696992    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r7w5\" (UniqueName: \"kubernetes.io/projected/9a253fda-2139-4fad-8704-a51fee5df9fe-kube-api-access-9r7w5\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:22:25.697102 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:25.697021    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a253fda-2139-4fad-8704-a51fee5df9fe-plugins-config-volume\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:22:26.355947 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:26.355846    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw" event={"ID":"1c2421b4-c704-4085-b44b-f5d596be5a91","Type":"ContainerDied","Data":"873a0087ce928cd53eee5078cac7efcb423a51aa61f737e386ee1bdbbb4dfd9d"}
Sep 07 13:22:26.355947 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:26.355916    2567 scope.go:117] "RemoveContainer" containerID="f141301e50b185c5d19af3372e3be7ce1856c452c21bcec05aec9f62b5bf332f"
Sep 07 13:22:26.355947 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:26.355863    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"
Sep 07 13:22:26.357474 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:26.357455    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"
Sep 07 13:22:26.357543 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:26.357469    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv" event={"ID":"9a253fda-2139-4fad-8704-a51fee5df9fe","Type":"ContainerDied","Data":"0c0c5de372d1fe39dc6e8c0bef8ec18536be25b9a0dc655824329209af80e683"}
Sep 07 13:22:26.367100 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:26.367078    2567 scope.go:117] "RemoveContainer" containerID="b1a371ba0a6fc9502574bda9151b8e8a24becff53228388c0c79cda0b5063db6"
Sep 07 13:22:26.377537 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:26.377488    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"]
Sep 07 13:22:26.382477 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:26.382453    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ede2050e-69fb8b74dd-jtjjw"]
Sep 07 13:22:26.392922 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:26.392900    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"]
Sep 07 13:22:26.399379 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:26.399312    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ede2050e-748d58c55d-9kqfv"]
Sep 07 13:22:28.077233 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:28.077194    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2421b4-c704-4085-b44b-f5d596be5a91" path="/var/lib/kubelet/pods/1c2421b4-c704-4085-b44b-f5d596be5a91/volumes"
Sep 07 13:22:28.077906 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:28.077887    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a253fda-2139-4fad-8704-a51fee5df9fe" path="/var/lib/kubelet/pods/9a253fda-2139-4fad-8704-a51fee5df9fe/volumes"
Sep 07 13:22:33.858443 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:33.858405    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"]
Sep 07 13:22:48.877932 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:48.877883    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj" podUID="dbc38af4-1439-4d9d-87ee-3c6a983bdfa1" containerName="main" containerID="cri-o://68065d2e6d26e1ae1093b1e5683a3ca099babc7ea0037d82c4ebe26abf287aa1" gracePeriod=45
Sep 07 13:22:49.138968 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.138897    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:22:49.331649 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.331616    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-dshm\") pod \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") "
Sep 07 13:22:49.331649 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.331661    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-tls-certs\") pod \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") "
Sep 07 13:22:49.331913 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.331701    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-tmp-dir\") pod \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") "
Sep 07 13:22:49.331913 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.331718    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-model-cache\") pod \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") "
Sep 07 13:22:49.331913 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.331834    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsjcq\" (UniqueName: \"kubernetes.io/projected/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-kube-api-access-xsjcq\") pod \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") "
Sep 07 13:22:49.331913 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.331871    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-home\") pod \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\" (UID: \"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1\") "
Sep 07 13:22:49.332177 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.332048    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbc38af4-1439-4d9d-87ee-3c6a983bdfa1" (UID: "dbc38af4-1439-4d9d-87ee-3c6a983bdfa1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:22:49.332177 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.332092    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbc38af4-1439-4d9d-87ee-3c6a983bdfa1" (UID: "dbc38af4-1439-4d9d-87ee-3c6a983bdfa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:22:49.332309 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.332281    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-home" (OuterVolumeSpecName: "home") pod "dbc38af4-1439-4d9d-87ee-3c6a983bdfa1" (UID: "dbc38af4-1439-4d9d-87ee-3c6a983bdfa1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:22:49.332309 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.332301    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-tmp-dir\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:22:49.332397 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.332322    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-model-cache\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:22:49.334040 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.334013    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-kube-api-access-xsjcq" (OuterVolumeSpecName: "kube-api-access-xsjcq") pod "dbc38af4-1439-4d9d-87ee-3c6a983bdfa1" (UID: "dbc38af4-1439-4d9d-87ee-3c6a983bdfa1"). InnerVolumeSpecName "kube-api-access-xsjcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:22:49.334040 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.334032    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-dshm" (OuterVolumeSpecName: "dshm") pod "dbc38af4-1439-4d9d-87ee-3c6a983bdfa1" (UID: "dbc38af4-1439-4d9d-87ee-3c6a983bdfa1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:22:49.334233 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.334142    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbc38af4-1439-4d9d-87ee-3c6a983bdfa1" (UID: "dbc38af4-1439-4d9d-87ee-3c6a983bdfa1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:22:49.433205 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.433119    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-dshm\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:22:49.433205 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.433151    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-tls-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:22:49.433205 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.433163    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsjcq\" (UniqueName: \"kubernetes.io/projected/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-kube-api-access-xsjcq\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:22:49.433205 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.433173    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1-home\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:22:49.495502 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.495471    2567 generic.go:356] "Generic (PLEG): container finished" podID="dbc38af4-1439-4d9d-87ee-3c6a983bdfa1" containerID="68065d2e6d26e1ae1093b1e5683a3ca099babc7ea0037d82c4ebe26abf287aa1" exitCode=0
Sep 07 13:22:49.495673 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.495517    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"
Sep 07 13:22:49.495673 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.495549    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj" event={"ID":"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1","Type":"ContainerDied","Data":"68065d2e6d26e1ae1093b1e5683a3ca099babc7ea0037d82c4ebe26abf287aa1"}
Sep 07 13:22:49.495673 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.495596    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj" event={"ID":"dbc38af4-1439-4d9d-87ee-3c6a983bdfa1","Type":"ContainerDied","Data":"3bc2b01f8dd86e1465d5597e12dd9e1bfb816a4ffbd9d8f0578c1844ebdc6526"}
Sep 07 13:22:49.495673 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.495619    2567 scope.go:117] "RemoveContainer" containerID="68065d2e6d26e1ae1093b1e5683a3ca099babc7ea0037d82c4ebe26abf287aa1"
Sep 07 13:22:49.505508 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.505490    2567 scope.go:117] "RemoveContainer" containerID="68065d2e6d26e1ae1093b1e5683a3ca099babc7ea0037d82c4ebe26abf287aa1"
Sep 07 13:22:49.505774 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:22:49.505755    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68065d2e6d26e1ae1093b1e5683a3ca099babc7ea0037d82c4ebe26abf287aa1\": container with ID starting with 68065d2e6d26e1ae1093b1e5683a3ca099babc7ea0037d82c4ebe26abf287aa1 not found: ID does not exist" containerID="68065d2e6d26e1ae1093b1e5683a3ca099babc7ea0037d82c4ebe26abf287aa1"
Sep 07 13:22:49.505820 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.505782    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68065d2e6d26e1ae1093b1e5683a3ca099babc7ea0037d82c4ebe26abf287aa1"} err="failed to get container status \"68065d2e6d26e1ae1093b1e5683a3ca099babc7ea0037d82c4ebe26abf287aa1\": rpc error: code = NotFound desc = could not find container \"68065d2e6d26e1ae1093b1e5683a3ca099babc7ea0037d82c4ebe26abf287aa1\": container with ID starting with 68065d2e6d26e1ae1093b1e5683a3ca099babc7ea0037d82c4ebe26abf287aa1 not found: ID does not exist"
Sep 07 13:22:49.544473 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.544440    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"]
Sep 07 13:22:49.555344 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:49.555318    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/auth-test-model-c7c371-kserve-694c6d7775-5bhvj"]
Sep 07 13:22:50.077225 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:22:50.077185    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbc38af4-1439-4d9d-87ee-3c6a983bdfa1" path="/var/lib/kubelet/pods/dbc38af4-1439-4d9d-87ee-3c6a983bdfa1/volumes"
Sep 07 13:24:20.341430 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.341393    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"]
Sep 07 13:24:20.343753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.341781    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2421b4-c704-4085-b44b-f5d596be5a91" containerName="maas-api"
Sep 07 13:24:20.343753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.341793    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2421b4-c704-4085-b44b-f5d596be5a91" containerName="maas-api"
Sep 07 13:24:20.343753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.341810    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc38af4-1439-4d9d-87ee-3c6a983bdfa1" containerName="main"
Sep 07 13:24:20.343753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.341816    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc38af4-1439-4d9d-87ee-3c6a983bdfa1" containerName="main"
Sep 07 13:24:20.343753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.341825    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a253fda-2139-4fad-8704-a51fee5df9fe" containerName="payload-pre-processing"
Sep 07 13:24:20.343753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.341830    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a253fda-2139-4fad-8704-a51fee5df9fe" containerName="payload-pre-processing"
Sep 07 13:24:20.343753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.341885    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc38af4-1439-4d9d-87ee-3c6a983bdfa1" containerName="main"
Sep 07 13:24:20.343753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.341892    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2421b4-c704-4085-b44b-f5d596be5a91" containerName="maas-api"
Sep 07 13:24:20.343753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.341900    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a253fda-2139-4fad-8704-a51fee5df9fe" containerName="payload-pre-processing"
Sep 07 13:24:20.344807 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.344792    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.347709 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.347684    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c7c371\"/\"kube-root-ca.crt\""
Sep 07 13:24:20.347840 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.347715    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c7c371\"/\"default-dockercfg-gc9sj\""
Sep 07 13:24:20.347840 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.347696    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c7c371\"/\"openshift-service-ca.crt\""
Sep 07 13:24:20.348233 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.348218    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c7c371\"/\"rate-test-model-c7c371-kserve-self-signed-certs\""
Sep 07 13:24:20.359141 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.359118    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"]
Sep 07 13:24:20.410141 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.410105    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-model-cache\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.410300 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.410153    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-dshm\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.410300 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.410246    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-home\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.410300 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.410281    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-tmp-dir\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.410406 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.410344    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmc9t\" (UniqueName: \"kubernetes.io/projected/7152cd8f-abda-4ad1-be53-cb83de990de5-kube-api-access-kmc9t\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.410406 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.410378    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7152cd8f-abda-4ad1-be53-cb83de990de5-tls-certs\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.511797 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.511764    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-home\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.511797 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.511804    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-tmp-dir\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.512079 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.511936    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmc9t\" (UniqueName: \"kubernetes.io/projected/7152cd8f-abda-4ad1-be53-cb83de990de5-kube-api-access-kmc9t\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.512079 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.511988    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7152cd8f-abda-4ad1-be53-cb83de990de5-tls-certs\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.512079 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.512040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-model-cache\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.512259 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.512091    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-dshm\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.512315 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.512274    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-home\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.512371 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.512284    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-tmp-dir\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.512410 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.512380    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-model-cache\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.514337 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.514309    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-dshm\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.514560 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.514542    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7152cd8f-abda-4ad1-be53-cb83de990de5-tls-certs\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.523619 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.523596    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmc9t\" (UniqueName: \"kubernetes.io/projected/7152cd8f-abda-4ad1-be53-cb83de990de5-kube-api-access-kmc9t\") pod \"rate-test-model-c7c371-kserve-647b5d5ff-568vz\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.653885 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.653793    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:20.940002 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:20.939975    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"]
Sep 07 13:24:20.941797 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:24:20.941768    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7152cd8f_abda_4ad1_be53_cb83de990de5.slice/crio-ed17e5cac99e010d0926d4fc873d5a1637136a7c8477310409c874f3de594fd0 WatchSource:0}: Error finding container ed17e5cac99e010d0926d4fc873d5a1637136a7c8477310409c874f3de594fd0: Status 404 returned error can't find the container with id ed17e5cac99e010d0926d4fc873d5a1637136a7c8477310409c874f3de594fd0
Sep 07 13:24:21.032480 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:21.032429    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz" event={"ID":"7152cd8f-abda-4ad1-be53-cb83de990de5","Type":"ContainerStarted","Data":"f8bdecb487c6bfaea988da325082965267824ec5a6ea920a80b90cbacbbfc90a"}
Sep 07 13:24:21.032725 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:21.032488    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz" event={"ID":"7152cd8f-abda-4ad1-be53-cb83de990de5","Type":"ContainerStarted","Data":"ed17e5cac99e010d0926d4fc873d5a1637136a7c8477310409c874f3de594fd0"}
Sep 07 13:24:21.051158 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:21.051109    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz" podStartSLOduration=1.051093417 podStartE2EDuration="1.051093417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:24:21.048932019 +0000 UTC m=+1505.477230318" watchObservedRunningTime="2026-09-07 13:24:21.051093417 +0000 UTC m=+1505.479391690"
Sep 07 13:24:28.114977 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.114939    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"]
Sep 07 13:24:28.119236 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.119212    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:28.122294 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.122176    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-00fbc4-serving-cert\""
Sep 07 13:24:28.132868 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.132840    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"]
Sep 07 13:24:28.185376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.185343    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-maas-api-tls\") pod \"maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:28.185535 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.185395    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4xd5\" (UniqueName: \"kubernetes.io/projected/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-kube-api-access-v4xd5\") pod \"maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:28.185581 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.185520    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-openshift-service-ca\") pod \"maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:28.185625 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.185587    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-metrics-certs\") pod \"maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:28.286645 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.286606    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-openshift-service-ca\") pod \"maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:28.286840 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.286663    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-metrics-certs\") pod \"maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:28.286840 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.286747    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-maas-api-tls\") pod \"maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:28.286840 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.286800    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4xd5\" (UniqueName: \"kubernetes.io/projected/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-kube-api-access-v4xd5\") pod \"maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:28.287083 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:24:28.286891    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-serving-cert: secret "maas-api-e2e-api-a-00fbc4-serving-cert" not found
Sep 07 13:24:28.287083 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:24:28.286968    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-maas-api-tls podName:4ee15dbd-daa7-49d0-972a-fd05cd18ffdc nodeName:}" failed. No retries permitted until 2026-09-07 13:24:28.786947212 +0000 UTC m=+1513.215245480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-maas-api-tls") pod "maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt" (UID: "4ee15dbd-daa7-49d0-972a-fd05cd18ffdc") : secret "maas-api-e2e-api-a-00fbc4-serving-cert" not found
Sep 07 13:24:28.287424 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.287401    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-openshift-service-ca\") pod \"maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:28.289340 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.289316    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-metrics-certs\") pod \"maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:28.301565 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.301537    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4xd5\" (UniqueName: \"kubernetes.io/projected/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-kube-api-access-v4xd5\") pod \"maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:28.792680 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.792642    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-maas-api-tls\") pod \"maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:28.795049 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:28.795023    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-maas-api-tls\") pod \"maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:29.033330 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.033292    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:29.211463 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.211431    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"]
Sep 07 13:24:29.215207 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.215185    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"
Sep 07 13:24:29.217730 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.217703    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-00fbc4-dockercfg-vzk6d\""
Sep 07 13:24:29.217844 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.217703    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-00fbc4\""
Sep 07 13:24:29.231371 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.231349    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"]
Sep 07 13:24:29.299459 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.299385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjsrq\" (UniqueName: \"kubernetes.io/projected/2bcd69da-cdbc-4ac1-8961-cad592c3c973-kube-api-access-rjsrq\") pod \"payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49\" (UID: \"2bcd69da-cdbc-4ac1-8961-cad592c3c973\") " pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"
Sep 07 13:24:29.299459 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.299444    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bcd69da-cdbc-4ac1-8961-cad592c3c973-plugins-config-volume\") pod \"payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49\" (UID: \"2bcd69da-cdbc-4ac1-8961-cad592c3c973\") " pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"
Sep 07 13:24:29.313093 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.313043    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"]
Sep 07 13:24:29.313780 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:24:29.313746    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee15dbd_daa7_49d0_972a_fd05cd18ffdc.slice/crio-9175cfff90a877bf07fe7edba52ee7d8d0be993479d619e005aed89975b414cb WatchSource:0}: Error finding container 9175cfff90a877bf07fe7edba52ee7d8d0be993479d619e005aed89975b414cb: Status 404 returned error can't find the container with id 9175cfff90a877bf07fe7edba52ee7d8d0be993479d619e005aed89975b414cb
Sep 07 13:24:29.315398 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.315379    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 13:24:29.400621 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.400593    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjsrq\" (UniqueName: \"kubernetes.io/projected/2bcd69da-cdbc-4ac1-8961-cad592c3c973-kube-api-access-rjsrq\") pod \"payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49\" (UID: \"2bcd69da-cdbc-4ac1-8961-cad592c3c973\") " pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"
Sep 07 13:24:29.400797 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.400632    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bcd69da-cdbc-4ac1-8961-cad592c3c973-plugins-config-volume\") pod \"payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49\" (UID: \"2bcd69da-cdbc-4ac1-8961-cad592c3c973\") " pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"
Sep 07 13:24:29.401400 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.401374    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bcd69da-cdbc-4ac1-8961-cad592c3c973-plugins-config-volume\") pod \"payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49\" (UID: \"2bcd69da-cdbc-4ac1-8961-cad592c3c973\") " pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"
Sep 07 13:24:29.409302 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.409281    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjsrq\" (UniqueName: \"kubernetes.io/projected/2bcd69da-cdbc-4ac1-8961-cad592c3c973-kube-api-access-rjsrq\") pod \"payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49\" (UID: \"2bcd69da-cdbc-4ac1-8961-cad592c3c973\") " pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"
Sep 07 13:24:29.526653 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.526621    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"
Sep 07 13:24:29.808386 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:29.808337    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"]
Sep 07 13:24:29.809745 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:24:29.809714    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcd69da_cdbc_4ac1_8961_cad592c3c973.slice/crio-2712d1d33267b6ce9eed5eb0ba1a0f504562add55d5f5b3b956b91c0a2c20ca4 WatchSource:0}: Error finding container 2712d1d33267b6ce9eed5eb0ba1a0f504562add55d5f5b3b956b91c0a2c20ca4: Status 404 returned error can't find the container with id 2712d1d33267b6ce9eed5eb0ba1a0f504562add55d5f5b3b956b91c0a2c20ca4
Sep 07 13:24:30.087562 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:30.087473    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt" event={"ID":"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc","Type":"ContainerStarted","Data":"304dd1f8fe9f41ae2d9f0a276cc2fa90b3e691300dc1ae3f384304287e9118b0"}
Sep 07 13:24:30.087832 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:30.087805    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:30.087955 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:30.087841    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt" event={"ID":"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc","Type":"ContainerStarted","Data":"9175cfff90a877bf07fe7edba52ee7d8d0be993479d619e005aed89975b414cb"}
Sep 07 13:24:30.089247 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:30.089224    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49" event={"ID":"2bcd69da-cdbc-4ac1-8961-cad592c3c973","Type":"ContainerStarted","Data":"99097806be6322af8375a2d888759d07c660c8aeded5d981f16e4fc6b99dc16d"}
Sep 07 13:24:30.089335 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:30.089272    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49" event={"ID":"2bcd69da-cdbc-4ac1-8961-cad592c3c973","Type":"ContainerStarted","Data":"2712d1d33267b6ce9eed5eb0ba1a0f504562add55d5f5b3b956b91c0a2c20ca4"}
Sep 07 13:24:30.089335 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:30.089311    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"
Sep 07 13:24:30.108802 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:30.108748    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt" podStartSLOduration=1.548214898 podStartE2EDuration="2.108727708s" totalImagesPullingTime="560.51281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:24:30.106620885 +0000 UTC m=+1514.534919209" watchObservedRunningTime="2026-09-07 13:24:30.108727708 +0000 UTC m=+1514.537025983"
Sep 07 13:24:30.127541 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:30.127489    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49" podStartSLOduration=1.127472652 podStartE2EDuration="1.127472652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:24:30.126409233 +0000 UTC m=+1514.554707533" watchObservedRunningTime="2026-09-07 13:24:30.127472652 +0000 UTC m=+1514.555770925"
Sep 07 13:24:30.654772 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:30.654739    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:30.655168 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:30.654792    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:30.667348 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:30.667325    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:31.109345 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:31.109317    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:24:36.103655 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:36.103622    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:24:41.100130 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:24:41.100015    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"
Sep 07 13:25:31.029345 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:31.029313    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"]
Sep 07 13:25:37.639906 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:37.639871    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"]
Sep 07 13:25:37.643356 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:37.643324    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"
Sep 07 13:25:37.645748 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:37.645721    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-219f63\""
Sep 07 13:25:37.661828 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:37.661801    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"]
Sep 07 13:25:37.684398 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:37.684364    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14f9c48-5510-4850-88c2-a47612dcb838-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-219f63-658d774675-qvffs\" (UID: \"c14f9c48-5510-4850-88c2-a47612dcb838\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"
Sep 07 13:25:37.684586 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:37.684458    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxqqq\" (UniqueName: \"kubernetes.io/projected/c14f9c48-5510-4850-88c2-a47612dcb838-kube-api-access-cxqqq\") pod \"payload-processing-e2e-ipp-a-219f63-658d774675-qvffs\" (UID: \"c14f9c48-5510-4850-88c2-a47612dcb838\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"
Sep 07 13:25:37.795185 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:37.795137    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxqqq\" (UniqueName: \"kubernetes.io/projected/c14f9c48-5510-4850-88c2-a47612dcb838-kube-api-access-cxqqq\") pod \"payload-processing-e2e-ipp-a-219f63-658d774675-qvffs\" (UID: \"c14f9c48-5510-4850-88c2-a47612dcb838\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"
Sep 07 13:25:37.795390 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:37.795318    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14f9c48-5510-4850-88c2-a47612dcb838-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-219f63-658d774675-qvffs\" (UID: \"c14f9c48-5510-4850-88c2-a47612dcb838\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"
Sep 07 13:25:37.796301 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:37.796269    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14f9c48-5510-4850-88c2-a47612dcb838-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-219f63-658d774675-qvffs\" (UID: \"c14f9c48-5510-4850-88c2-a47612dcb838\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"
Sep 07 13:25:37.808746 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:37.808717    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxqqq\" (UniqueName: \"kubernetes.io/projected/c14f9c48-5510-4850-88c2-a47612dcb838-kube-api-access-cxqqq\") pod \"payload-processing-e2e-ipp-a-219f63-658d774675-qvffs\" (UID: \"c14f9c48-5510-4850-88c2-a47612dcb838\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"
Sep 07 13:25:37.954097 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:37.953982    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"
Sep 07 13:25:38.239990 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:38.239908    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"]
Sep 07 13:25:38.484019 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:38.483971    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs" event={"ID":"c14f9c48-5510-4850-88c2-a47612dcb838","Type":"ContainerStarted","Data":"15f157041b61dccf2f0828b87d061744806649db87f169137eb0395736a43d60"}
Sep 07 13:25:38.484205 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:38.484047    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"
Sep 07 13:25:38.484205 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:38.484101    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs" event={"ID":"c14f9c48-5510-4850-88c2-a47612dcb838","Type":"ContainerStarted","Data":"637910bec927a96f95510bda0b48367e80820fbaf979d9842ced5995e259895c"}
Sep 07 13:25:38.513673 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:38.513575    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs" podStartSLOduration=1.513559799 podStartE2EDuration="1.513559799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:25:38.510279015 +0000 UTC m=+1582.938577311" watchObservedRunningTime="2026-09-07 13:25:38.513559799 +0000 UTC m=+1582.941858072"
Sep 07 13:25:40.243923 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.243875    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"]
Sep 07 13:25:40.244412 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.244164    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt" podUID="4ee15dbd-daa7-49d0-972a-fd05cd18ffdc" containerName="maas-api" containerID="cri-o://304dd1f8fe9f41ae2d9f0a276cc2fa90b3e691300dc1ae3f384304287e9118b0" gracePeriod=30
Sep 07 13:25:40.349091 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.349045    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"]
Sep 07 13:25:40.349355 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.349324    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49" podUID="2bcd69da-cdbc-4ac1-8961-cad592c3c973" containerName="payload-processing" containerID="cri-o://99097806be6322af8375a2d888759d07c660c8aeded5d981f16e4fc6b99dc16d" gracePeriod=30
Sep 07 13:25:40.500531 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.500327    2567 generic.go:356] "Generic (PLEG): container finished" podID="4ee15dbd-daa7-49d0-972a-fd05cd18ffdc" containerID="304dd1f8fe9f41ae2d9f0a276cc2fa90b3e691300dc1ae3f384304287e9118b0" exitCode=0
Sep 07 13:25:40.500531 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.500395    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt" event={"ID":"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc","Type":"ContainerDied","Data":"304dd1f8fe9f41ae2d9f0a276cc2fa90b3e691300dc1ae3f384304287e9118b0"}
Sep 07 13:25:40.500531 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.500484    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt" event={"ID":"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc","Type":"ContainerDied","Data":"9175cfff90a877bf07fe7edba52ee7d8d0be993479d619e005aed89975b414cb"}
Sep 07 13:25:40.500531 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.500522    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9175cfff90a877bf07fe7edba52ee7d8d0be993479d619e005aed89975b414cb"
Sep 07 13:25:40.502434 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.502408    2567 generic.go:356] "Generic (PLEG): container finished" podID="2bcd69da-cdbc-4ac1-8961-cad592c3c973" containerID="99097806be6322af8375a2d888759d07c660c8aeded5d981f16e4fc6b99dc16d" exitCode=0
Sep 07 13:25:40.502564 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.502454    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49" event={"ID":"2bcd69da-cdbc-4ac1-8961-cad592c3c973","Type":"ContainerDied","Data":"99097806be6322af8375a2d888759d07c660c8aeded5d981f16e4fc6b99dc16d"}
Sep 07 13:25:40.509096 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.509073    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:25:40.620005 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.619982    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"
Sep 07 13:25:40.624015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.623994    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4xd5\" (UniqueName: \"kubernetes.io/projected/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-kube-api-access-v4xd5\") pod \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") "
Sep 07 13:25:40.624129 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.624033    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-openshift-service-ca\") pod \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") "
Sep 07 13:25:40.624129 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.624096    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-metrics-certs\") pod \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") "
Sep 07 13:25:40.624213 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.624150    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-maas-api-tls\") pod \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\" (UID: \"4ee15dbd-daa7-49d0-972a-fd05cd18ffdc\") "
Sep 07 13:25:40.624493 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.624446    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ee15dbd-daa7-49d0-972a-fd05cd18ffdc" (UID: "4ee15dbd-daa7-49d0-972a-fd05cd18ffdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:25:40.626139 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.626114    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ee15dbd-daa7-49d0-972a-fd05cd18ffdc" (UID: "4ee15dbd-daa7-49d0-972a-fd05cd18ffdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:25:40.626245 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.626134    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4ee15dbd-daa7-49d0-972a-fd05cd18ffdc" (UID: "4ee15dbd-daa7-49d0-972a-fd05cd18ffdc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:25:40.626245 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.626142    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-kube-api-access-v4xd5" (OuterVolumeSpecName: "kube-api-access-v4xd5") pod "4ee15dbd-daa7-49d0-972a-fd05cd18ffdc" (UID: "4ee15dbd-daa7-49d0-972a-fd05cd18ffdc"). InnerVolumeSpecName "kube-api-access-v4xd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:25:40.725289 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.725258    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjsrq\" (UniqueName: \"kubernetes.io/projected/2bcd69da-cdbc-4ac1-8961-cad592c3c973-kube-api-access-rjsrq\") pod \"2bcd69da-cdbc-4ac1-8961-cad592c3c973\" (UID: \"2bcd69da-cdbc-4ac1-8961-cad592c3c973\") "
Sep 07 13:25:40.725465 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.725321    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bcd69da-cdbc-4ac1-8961-cad592c3c973-plugins-config-volume\") pod \"2bcd69da-cdbc-4ac1-8961-cad592c3c973\" (UID: \"2bcd69da-cdbc-4ac1-8961-cad592c3c973\") "
Sep 07 13:25:40.725633 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.725607    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcd69da-cdbc-4ac1-8961-cad592c3c973-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2bcd69da-cdbc-4ac1-8961-cad592c3c973" (UID: "2bcd69da-cdbc-4ac1-8961-cad592c3c973"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:25:40.725695 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.725677    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4xd5\" (UniqueName: \"kubernetes.io/projected/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-kube-api-access-v4xd5\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:25:40.725734 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.725701    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:25:40.725734 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.725716    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:25:40.725734 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.725727    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:25:40.727415 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.727397    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcd69da-cdbc-4ac1-8961-cad592c3c973-kube-api-access-rjsrq" (OuterVolumeSpecName: "kube-api-access-rjsrq") pod "2bcd69da-cdbc-4ac1-8961-cad592c3c973" (UID: "2bcd69da-cdbc-4ac1-8961-cad592c3c973"). InnerVolumeSpecName "kube-api-access-rjsrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:25:40.826941 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.826906    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjsrq\" (UniqueName: \"kubernetes.io/projected/2bcd69da-cdbc-4ac1-8961-cad592c3c973-kube-api-access-rjsrq\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:25:40.826941 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:40.826941    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bcd69da-cdbc-4ac1-8961-cad592c3c973-plugins-config-volume\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:25:41.509645 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:41.509560    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"
Sep 07 13:25:41.509645 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:41.509596    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49" event={"ID":"2bcd69da-cdbc-4ac1-8961-cad592c3c973","Type":"ContainerDied","Data":"2712d1d33267b6ce9eed5eb0ba1a0f504562add55d5f5b3b956b91c0a2c20ca4"}
Sep 07 13:25:41.510182 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:41.509667    2567 scope.go:117] "RemoveContainer" containerID="99097806be6322af8375a2d888759d07c660c8aeded5d981f16e4fc6b99dc16d"
Sep 07 13:25:41.510182 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:41.509838    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"
Sep 07 13:25:41.538889 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:41.538860    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"]
Sep 07 13:25:41.541739 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:41.541716    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-00fbc4-8457b74845-n2j49"]
Sep 07 13:25:41.558490 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:41.558471    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"]
Sep 07 13:25:41.564666 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:41.564645    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00fbc4-98f6b659b-4pmlt"]
Sep 07 13:25:42.075502 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:42.075465    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bcd69da-cdbc-4ac1-8961-cad592c3c973" path="/var/lib/kubelet/pods/2bcd69da-cdbc-4ac1-8961-cad592c3c973/volumes"
Sep 07 13:25:42.075893 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:42.075871    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ee15dbd-daa7-49d0-972a-fd05cd18ffdc" path="/var/lib/kubelet/pods/4ee15dbd-daa7-49d0-972a-fd05cd18ffdc/volumes"
Sep 07 13:25:46.048687 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.048629    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz" podUID="7152cd8f-abda-4ad1-be53-cb83de990de5" containerName="main" containerID="cri-o://f8bdecb487c6bfaea988da325082965267824ec5a6ea920a80b90cbacbbfc90a" gracePeriod=45
Sep 07 13:25:46.295470 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.295444    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:25:46.379167 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.379070    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7152cd8f-abda-4ad1-be53-cb83de990de5-tls-certs\") pod \"7152cd8f-abda-4ad1-be53-cb83de990de5\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") "
Sep 07 13:25:46.379167 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.379129    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-home\") pod \"7152cd8f-abda-4ad1-be53-cb83de990de5\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") "
Sep 07 13:25:46.379398 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.379176    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-model-cache\") pod \"7152cd8f-abda-4ad1-be53-cb83de990de5\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") "
Sep 07 13:25:46.379398 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.379207    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-dshm\") pod \"7152cd8f-abda-4ad1-be53-cb83de990de5\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") "
Sep 07 13:25:46.379398 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.379233    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmc9t\" (UniqueName: \"kubernetes.io/projected/7152cd8f-abda-4ad1-be53-cb83de990de5-kube-api-access-kmc9t\") pod \"7152cd8f-abda-4ad1-be53-cb83de990de5\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") "
Sep 07 13:25:46.379398 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.379311    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-tmp-dir\") pod \"7152cd8f-abda-4ad1-be53-cb83de990de5\" (UID: \"7152cd8f-abda-4ad1-be53-cb83de990de5\") "
Sep 07 13:25:46.379603 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.379462    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-home" (OuterVolumeSpecName: "home") pod "7152cd8f-abda-4ad1-be53-cb83de990de5" (UID: "7152cd8f-abda-4ad1-be53-cb83de990de5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:25:46.379603 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.379470    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-model-cache" (OuterVolumeSpecName: "model-cache") pod "7152cd8f-abda-4ad1-be53-cb83de990de5" (UID: "7152cd8f-abda-4ad1-be53-cb83de990de5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:25:46.379768 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.379746    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-home\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:25:46.379853 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.379774    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-model-cache\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:25:46.379853 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.379771    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7152cd8f-abda-4ad1-be53-cb83de990de5" (UID: "7152cd8f-abda-4ad1-be53-cb83de990de5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:25:46.381520 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.381498    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7152cd8f-abda-4ad1-be53-cb83de990de5-kube-api-access-kmc9t" (OuterVolumeSpecName: "kube-api-access-kmc9t") pod "7152cd8f-abda-4ad1-be53-cb83de990de5" (UID: "7152cd8f-abda-4ad1-be53-cb83de990de5"). InnerVolumeSpecName "kube-api-access-kmc9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:25:46.381752 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.381722    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7152cd8f-abda-4ad1-be53-cb83de990de5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7152cd8f-abda-4ad1-be53-cb83de990de5" (UID: "7152cd8f-abda-4ad1-be53-cb83de990de5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:25:46.381752 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.381750    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-dshm" (OuterVolumeSpecName: "dshm") pod "7152cd8f-abda-4ad1-be53-cb83de990de5" (UID: "7152cd8f-abda-4ad1-be53-cb83de990de5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:25:46.480468 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.480427    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-dshm\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:25:46.480468 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.480461    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmc9t\" (UniqueName: \"kubernetes.io/projected/7152cd8f-abda-4ad1-be53-cb83de990de5-kube-api-access-kmc9t\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:25:46.480715 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.480477    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7152cd8f-abda-4ad1-be53-cb83de990de5-tmp-dir\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:25:46.480715 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.480490    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7152cd8f-abda-4ad1-be53-cb83de990de5-tls-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:25:46.544997 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.544924    2567 generic.go:356] "Generic (PLEG): container finished" podID="7152cd8f-abda-4ad1-be53-cb83de990de5" containerID="f8bdecb487c6bfaea988da325082965267824ec5a6ea920a80b90cbacbbfc90a" exitCode=0
Sep 07 13:25:46.544997 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.544983    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz" event={"ID":"7152cd8f-abda-4ad1-be53-cb83de990de5","Type":"ContainerDied","Data":"f8bdecb487c6bfaea988da325082965267824ec5a6ea920a80b90cbacbbfc90a"}
Sep 07 13:25:46.545282 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.545008    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"
Sep 07 13:25:46.545282 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.545079    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz" event={"ID":"7152cd8f-abda-4ad1-be53-cb83de990de5","Type":"ContainerDied","Data":"ed17e5cac99e010d0926d4fc873d5a1637136a7c8477310409c874f3de594fd0"}
Sep 07 13:25:46.545282 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.545112    2567 scope.go:117] "RemoveContainer" containerID="f8bdecb487c6bfaea988da325082965267824ec5a6ea920a80b90cbacbbfc90a"
Sep 07 13:25:46.556488 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.556467    2567 scope.go:117] "RemoveContainer" containerID="f8bdecb487c6bfaea988da325082965267824ec5a6ea920a80b90cbacbbfc90a"
Sep 07 13:25:46.556783 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:25:46.556757    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8bdecb487c6bfaea988da325082965267824ec5a6ea920a80b90cbacbbfc90a\": container with ID starting with f8bdecb487c6bfaea988da325082965267824ec5a6ea920a80b90cbacbbfc90a not found: ID does not exist" containerID="f8bdecb487c6bfaea988da325082965267824ec5a6ea920a80b90cbacbbfc90a"
Sep 07 13:25:46.556886 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.556797    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8bdecb487c6bfaea988da325082965267824ec5a6ea920a80b90cbacbbfc90a"} err="failed to get container status \"f8bdecb487c6bfaea988da325082965267824ec5a6ea920a80b90cbacbbfc90a\": rpc error: code = NotFound desc = could not find container \"f8bdecb487c6bfaea988da325082965267824ec5a6ea920a80b90cbacbbfc90a\": container with ID starting with f8bdecb487c6bfaea988da325082965267824ec5a6ea920a80b90cbacbbfc90a not found: ID does not exist"
Sep 07 13:25:46.570740 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.570708    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"]
Sep 07 13:25:46.584961 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:46.584924    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/rate-test-model-c7c371-kserve-647b5d5ff-568vz"]
Sep 07 13:25:48.075789 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:48.075745    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7152cd8f-abda-4ad1-be53-cb83de990de5" path="/var/lib/kubelet/pods/7152cd8f-abda-4ad1-be53-cb83de990de5/volumes"
Sep 07 13:25:49.492576 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:25:49.492540    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"
Sep 07 13:26:01.198410 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.198374    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"]
Sep 07 13:26:01.198982 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.198961    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bcd69da-cdbc-4ac1-8961-cad592c3c973" containerName="payload-processing"
Sep 07 13:26:01.199090 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.198987    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bcd69da-cdbc-4ac1-8961-cad592c3c973" containerName="payload-processing"
Sep 07 13:26:01.199090 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.199020    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7152cd8f-abda-4ad1-be53-cb83de990de5" containerName="main"
Sep 07 13:26:01.199090 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.199042    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7152cd8f-abda-4ad1-be53-cb83de990de5" containerName="main"
Sep 07 13:26:01.199090 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.199086    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee15dbd-daa7-49d0-972a-fd05cd18ffdc" containerName="maas-api"
Sep 07 13:26:01.199390 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.199103    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee15dbd-daa7-49d0-972a-fd05cd18ffdc" containerName="maas-api"
Sep 07 13:26:01.199390 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.199214    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ee15dbd-daa7-49d0-972a-fd05cd18ffdc" containerName="maas-api"
Sep 07 13:26:01.199390 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.199229    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7152cd8f-abda-4ad1-be53-cb83de990de5" containerName="main"
Sep 07 13:26:01.199390 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.199241    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bcd69da-cdbc-4ac1-8961-cad592c3c973" containerName="payload-processing"
Sep 07 13:26:01.202375 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.202350    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.204937 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.204916    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8dc72a-openshift-default-dockercfg-zklwb\""
Sep 07 13:26:01.217353 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.217319    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-credential-socket\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.217488 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.217396    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-podinfo\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.217488 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.217428    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-data\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.217488 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.217455    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-workload-certs\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.217741 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.217547    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-workload-socket\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.217741 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.217584    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-envoy\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.217741 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.217622    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-token\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.217741 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.217706    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpj5\" (UniqueName: \"kubernetes.io/projected/87ef1aaf-726b-44ab-b453-1a861b06dbe7-kube-api-access-hwpj5\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.217964 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.217750    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istiod-ca-cert\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.221459 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.221430    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"]
Sep 07 13:26:01.319327 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.319262    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-workload-socket\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.319509 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.319362    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-envoy\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.319509 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.319413    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-token\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.319509 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.319481    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpj5\" (UniqueName: \"kubernetes.io/projected/87ef1aaf-726b-44ab-b453-1a861b06dbe7-kube-api-access-hwpj5\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.319699 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.319528    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istiod-ca-cert\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.319699 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.319575    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-credential-socket\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.319699 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.319634    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-podinfo\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.319699 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.319660    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-data\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.319905 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.319696    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-workload-certs\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.320857 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.320084    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-workload-certs\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.320857 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.320132    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-credential-socket\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.320857 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.320382    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-data\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.320857 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.320671    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-workload-socket\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.320857 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.320700    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istiod-ca-cert\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.322869 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.322848    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-podinfo\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.322993 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.322928    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-envoy\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.328735 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.328709    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-token\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.329732 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.329692    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpj5\" (UniqueName: \"kubernetes.io/projected/87ef1aaf-726b-44ab-b453-1a861b06dbe7-kube-api-access-hwpj5\") pod \"e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") " pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.513585 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.513493    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:01.796077 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.796030    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"]
Sep 07 13:26:01.797653 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:26:01.797613    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ef1aaf_726b_44ab_b453_1a861b06dbe7.slice/crio-c1f6b56ba4b77392f04eee2cd1202a7b2d8d467086af3de913069ae76ef4eb74 WatchSource:0}: Error finding container c1f6b56ba4b77392f04eee2cd1202a7b2d8d467086af3de913069ae76ef4eb74: Status 404 returned error can't find the container with id c1f6b56ba4b77392f04eee2cd1202a7b2d8d467086af3de913069ae76ef4eb74
Sep 07 13:26:01.800039 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.800007    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:26:01.800157 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.800086    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:26:01.800157 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:01.800117    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:26:02.641715 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:02.641674    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst" event={"ID":"87ef1aaf-726b-44ab-b453-1a861b06dbe7","Type":"ContainerStarted","Data":"5f2f48c32ada135934e9598d86de4a9e2d0b816e7eb6cc2366ea0e3e7ebb689c"}
Sep 07 13:26:02.642215 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:02.641740    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst" event={"ID":"87ef1aaf-726b-44ab-b453-1a861b06dbe7","Type":"ContainerStarted","Data":"c1f6b56ba4b77392f04eee2cd1202a7b2d8d467086af3de913069ae76ef4eb74"}
Sep 07 13:26:02.672472 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:02.672412    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst" podStartSLOduration=1.6723915809999998 podStartE2EDuration="1.672391581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:26:02.67165506 +0000 UTC m=+1607.099953367" watchObservedRunningTime="2026-09-07 13:26:02.672391581 +0000 UTC m=+1607.100689855"
Sep 07 13:26:03.514299 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:03.514263    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:03.519697 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:03.519673    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:03.650830 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:03.650799    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:03.654167 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:03.654141    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:07.782983 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:07.782945    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"]
Sep 07 13:26:07.786811 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:07.786787    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"
Sep 07 13:26:07.790000 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:07.789977    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8dc72a\""
Sep 07 13:26:07.790194 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:07.790173    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8dc72a-dockercfg-ftmnf\""
Sep 07 13:26:07.801416 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:07.801392    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"]
Sep 07 13:26:07.884119 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:07.884076    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnjfc\" (UniqueName: \"kubernetes.io/projected/6554a9a0-9500-439d-a3bc-52e5d45a9ba8-kube-api-access-qnjfc\") pod \"payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546\" (UID: \"6554a9a0-9500-439d-a3bc-52e5d45a9ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"
Sep 07 13:26:07.884302 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:07.884148    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6554a9a0-9500-439d-a3bc-52e5d45a9ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546\" (UID: \"6554a9a0-9500-439d-a3bc-52e5d45a9ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"
Sep 07 13:26:07.984993 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:07.984954    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6554a9a0-9500-439d-a3bc-52e5d45a9ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546\" (UID: \"6554a9a0-9500-439d-a3bc-52e5d45a9ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"
Sep 07 13:26:07.985199 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:07.985134    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnjfc\" (UniqueName: \"kubernetes.io/projected/6554a9a0-9500-439d-a3bc-52e5d45a9ba8-kube-api-access-qnjfc\") pod \"payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546\" (UID: \"6554a9a0-9500-439d-a3bc-52e5d45a9ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"
Sep 07 13:26:07.985781 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:07.985755    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6554a9a0-9500-439d-a3bc-52e5d45a9ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546\" (UID: \"6554a9a0-9500-439d-a3bc-52e5d45a9ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"
Sep 07 13:26:07.994078 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:07.994037    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnjfc\" (UniqueName: \"kubernetes.io/projected/6554a9a0-9500-439d-a3bc-52e5d45a9ba8-kube-api-access-qnjfc\") pod \"payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546\" (UID: \"6554a9a0-9500-439d-a3bc-52e5d45a9ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"
Sep 07 13:26:08.097725 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:08.097637    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"
Sep 07 13:26:08.376275 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:08.376248    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"]
Sep 07 13:26:08.379540 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:26:08.379514    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6554a9a0_9500_439d_a3bc_52e5d45a9ba8.slice/crio-e0a7273b6db55ae25b6141ca1936d83c99e55411a67be9f8d708c96b063c3c7c WatchSource:0}: Error finding container e0a7273b6db55ae25b6141ca1936d83c99e55411a67be9f8d708c96b063c3c7c: Status 404 returned error can't find the container with id e0a7273b6db55ae25b6141ca1936d83c99e55411a67be9f8d708c96b063c3c7c
Sep 07 13:26:08.686982 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:08.686893    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546" event={"ID":"6554a9a0-9500-439d-a3bc-52e5d45a9ba8","Type":"ContainerStarted","Data":"169be037f112acc3f15f9aa5d97109c80fd50ce4f9e5a10fa15bc232df54528b"}
Sep 07 13:26:08.686982 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:08.686955    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546" event={"ID":"6554a9a0-9500-439d-a3bc-52e5d45a9ba8","Type":"ContainerStarted","Data":"e0a7273b6db55ae25b6141ca1936d83c99e55411a67be9f8d708c96b063c3c7c"}
Sep 07 13:26:08.687221 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:08.687126    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"
Sep 07 13:26:08.712079 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:08.712002    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546" podStartSLOduration=1.711982571 podStartE2EDuration="1.711982571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:26:08.710003627 +0000 UTC m=+1613.138301901" watchObservedRunningTime="2026-09-07 13:26:08.711982571 +0000 UTC m=+1613.140280845"
Sep 07 13:26:19.697279 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:19.697244    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"
Sep 07 13:26:22.329953 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.329912    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"]
Sep 07 13:26:22.330429 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.330169    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g" podUID="3f644a7a-56bd-4026-8df6-c4b86ee680ff" containerName="maas-api" containerID="cri-o://24e6b6d747e7ecd76d4fb2de60dd01402c22e811d8356db84f82551e37c90ee8" gracePeriod=30
Sep 07 13:26:22.407572 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.407536    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"]
Sep 07 13:26:22.407782 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.407760    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9" podUID="70883334-79e1-41c2-9668-c9dfddf251c1" containerName="payload-pre-processing" containerID="cri-o://e5dfe890b7653508987d2abc52967d9270497ae162a3c60dedd370d9c9f1b5d7" gracePeriod=30
Sep 07 13:26:22.629397 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.629310    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:26:22.739464 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.739423    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scnhz\" (UniqueName: \"kubernetes.io/projected/3f644a7a-56bd-4026-8df6-c4b86ee680ff-kube-api-access-scnhz\") pod \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") "
Sep 07 13:26:22.739881 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.739860    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-metrics-certs\") pod \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") "
Sep 07 13:26:22.740011 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.739910    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-maas-api-tls\") pod \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") "
Sep 07 13:26:22.740011 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.739935    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f644a7a-56bd-4026-8df6-c4b86ee680ff-openshift-service-ca\") pod \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\" (UID: \"3f644a7a-56bd-4026-8df6-c4b86ee680ff\") "
Sep 07 13:26:22.740648 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.740614    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f644a7a-56bd-4026-8df6-c4b86ee680ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f644a7a-56bd-4026-8df6-c4b86ee680ff" (UID: "3f644a7a-56bd-4026-8df6-c4b86ee680ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:26:22.740814 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.740793    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"
Sep 07 13:26:22.742106 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.742048    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f644a7a-56bd-4026-8df6-c4b86ee680ff-kube-api-access-scnhz" (OuterVolumeSpecName: "kube-api-access-scnhz") pod "3f644a7a-56bd-4026-8df6-c4b86ee680ff" (UID: "3f644a7a-56bd-4026-8df6-c4b86ee680ff"). InnerVolumeSpecName "kube-api-access-scnhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:26:22.742333 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.742311    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3f644a7a-56bd-4026-8df6-c4b86ee680ff" (UID: "3f644a7a-56bd-4026-8df6-c4b86ee680ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:26:22.742451 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.742429    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f644a7a-56bd-4026-8df6-c4b86ee680ff" (UID: "3f644a7a-56bd-4026-8df6-c4b86ee680ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:26:22.785584 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.785547    2567 generic.go:356] "Generic (PLEG): container finished" podID="3f644a7a-56bd-4026-8df6-c4b86ee680ff" containerID="24e6b6d747e7ecd76d4fb2de60dd01402c22e811d8356db84f82551e37c90ee8" exitCode=0
Sep 07 13:26:22.785786 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.785603    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"
Sep 07 13:26:22.785786 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.785635    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g" event={"ID":"3f644a7a-56bd-4026-8df6-c4b86ee680ff","Type":"ContainerDied","Data":"24e6b6d747e7ecd76d4fb2de60dd01402c22e811d8356db84f82551e37c90ee8"}
Sep 07 13:26:22.785786 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.785695    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g" event={"ID":"3f644a7a-56bd-4026-8df6-c4b86ee680ff","Type":"ContainerDied","Data":"ce7ad34a09e0c5a770bdb7204108c350f26bd6e865c542736236637f55dfc0d6"}
Sep 07 13:26:22.785786 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.785712    2567 scope.go:117] "RemoveContainer" containerID="24e6b6d747e7ecd76d4fb2de60dd01402c22e811d8356db84f82551e37c90ee8"
Sep 07 13:26:22.787584 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.787558    2567 generic.go:356] "Generic (PLEG): container finished" podID="70883334-79e1-41c2-9668-c9dfddf251c1" containerID="e5dfe890b7653508987d2abc52967d9270497ae162a3c60dedd370d9c9f1b5d7" exitCode=0
Sep 07 13:26:22.787711 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.787603    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"
Sep 07 13:26:22.787711 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.787649    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9" event={"ID":"70883334-79e1-41c2-9668-c9dfddf251c1","Type":"ContainerDied","Data":"e5dfe890b7653508987d2abc52967d9270497ae162a3c60dedd370d9c9f1b5d7"}
Sep 07 13:26:22.787711 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.787689    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9" event={"ID":"70883334-79e1-41c2-9668-c9dfddf251c1","Type":"ContainerDied","Data":"0c4fc33266476dbc37447f3f69c71f02666826b10f3f07cf9d89519a8ab1a409"}
Sep 07 13:26:22.798027 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.798006    2567 scope.go:117] "RemoveContainer" containerID="24e6b6d747e7ecd76d4fb2de60dd01402c22e811d8356db84f82551e37c90ee8"
Sep 07 13:26:22.798340 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:26:22.798319    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e6b6d747e7ecd76d4fb2de60dd01402c22e811d8356db84f82551e37c90ee8\": container with ID starting with 24e6b6d747e7ecd76d4fb2de60dd01402c22e811d8356db84f82551e37c90ee8 not found: ID does not exist" containerID="24e6b6d747e7ecd76d4fb2de60dd01402c22e811d8356db84f82551e37c90ee8"
Sep 07 13:26:22.798416 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.798348    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e6b6d747e7ecd76d4fb2de60dd01402c22e811d8356db84f82551e37c90ee8"} err="failed to get container status \"24e6b6d747e7ecd76d4fb2de60dd01402c22e811d8356db84f82551e37c90ee8\": rpc error: code = NotFound desc = could not find container \"24e6b6d747e7ecd76d4fb2de60dd01402c22e811d8356db84f82551e37c90ee8\": container with ID starting with 24e6b6d747e7ecd76d4fb2de60dd01402c22e811d8356db84f82551e37c90ee8 not found: ID does not exist"
Sep 07 13:26:22.798416 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.798365    2567 scope.go:117] "RemoveContainer" containerID="e5dfe890b7653508987d2abc52967d9270497ae162a3c60dedd370d9c9f1b5d7"
Sep 07 13:26:22.808874 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.808825    2567 scope.go:117] "RemoveContainer" containerID="e5dfe890b7653508987d2abc52967d9270497ae162a3c60dedd370d9c9f1b5d7"
Sep 07 13:26:22.809378 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:26:22.809354    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5dfe890b7653508987d2abc52967d9270497ae162a3c60dedd370d9c9f1b5d7\": container with ID starting with e5dfe890b7653508987d2abc52967d9270497ae162a3c60dedd370d9c9f1b5d7 not found: ID does not exist" containerID="e5dfe890b7653508987d2abc52967d9270497ae162a3c60dedd370d9c9f1b5d7"
Sep 07 13:26:22.809537 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.809511    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5dfe890b7653508987d2abc52967d9270497ae162a3c60dedd370d9c9f1b5d7"} err="failed to get container status \"e5dfe890b7653508987d2abc52967d9270497ae162a3c60dedd370d9c9f1b5d7\": rpc error: code = NotFound desc = could not find container \"e5dfe890b7653508987d2abc52967d9270497ae162a3c60dedd370d9c9f1b5d7\": container with ID starting with e5dfe890b7653508987d2abc52967d9270497ae162a3c60dedd370d9c9f1b5d7 not found: ID does not exist"
Sep 07 13:26:22.813554 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.813530    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"]
Sep 07 13:26:22.819633 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.819609    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d5da3f-678cbf64f5-9k79g"]
Sep 07 13:26:22.841289 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.841256    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70883334-79e1-41c2-9668-c9dfddf251c1-plugins-config-volume\") pod \"70883334-79e1-41c2-9668-c9dfddf251c1\" (UID: \"70883334-79e1-41c2-9668-c9dfddf251c1\") "
Sep 07 13:26:22.841402 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.841332    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtklx\" (UniqueName: \"kubernetes.io/projected/70883334-79e1-41c2-9668-c9dfddf251c1-kube-api-access-xtklx\") pod \"70883334-79e1-41c2-9668-c9dfddf251c1\" (UID: \"70883334-79e1-41c2-9668-c9dfddf251c1\") "
Sep 07 13:26:22.841661 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.841636    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70883334-79e1-41c2-9668-c9dfddf251c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70883334-79e1-41c2-9668-c9dfddf251c1" (UID: "70883334-79e1-41c2-9668-c9dfddf251c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:26:22.841719 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.841705    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:26:22.841758 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.841730    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f644a7a-56bd-4026-8df6-c4b86ee680ff-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:26:22.841758 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.841744    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f644a7a-56bd-4026-8df6-c4b86ee680ff-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:26:22.841839 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.841757    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scnhz\" (UniqueName: \"kubernetes.io/projected/3f644a7a-56bd-4026-8df6-c4b86ee680ff-kube-api-access-scnhz\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:26:22.843570 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.843546    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70883334-79e1-41c2-9668-c9dfddf251c1-kube-api-access-xtklx" (OuterVolumeSpecName: "kube-api-access-xtklx") pod "70883334-79e1-41c2-9668-c9dfddf251c1" (UID: "70883334-79e1-41c2-9668-c9dfddf251c1"). InnerVolumeSpecName "kube-api-access-xtklx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:26:22.942383 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.942293    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtklx\" (UniqueName: \"kubernetes.io/projected/70883334-79e1-41c2-9668-c9dfddf251c1-kube-api-access-xtklx\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:26:22.942383 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:22.942329    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70883334-79e1-41c2-9668-c9dfddf251c1-plugins-config-volume\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:26:23.112032 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:23.112000    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"]
Sep 07 13:26:23.116224 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:23.116196    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d5da3f-546f76f466-psmn9"]
Sep 07 13:26:24.083398 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:24.081001    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f644a7a-56bd-4026-8df6-c4b86ee680ff" path="/var/lib/kubelet/pods/3f644a7a-56bd-4026-8df6-c4b86ee680ff/volumes"
Sep 07 13:26:24.083398 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:24.081721    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70883334-79e1-41c2-9668-c9dfddf251c1" path="/var/lib/kubelet/pods/70883334-79e1-41c2-9668-c9dfddf251c1/volumes"
Sep 07 13:26:24.806187 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:24.806155    2567 generic.go:356] "Generic (PLEG): container finished" podID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerID="5f2f48c32ada135934e9598d86de4a9e2d0b816e7eb6cc2366ea0e3e7ebb689c" exitCode=0
Sep 07 13:26:24.806343 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:24.806187    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst" event={"ID":"87ef1aaf-726b-44ab-b453-1a861b06dbe7","Type":"ContainerDied","Data":"5f2f48c32ada135934e9598d86de4a9e2d0b816e7eb6cc2366ea0e3e7ebb689c"}
Sep 07 13:26:24.806627 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:24.806609    2567 scope.go:117] "RemoveContainer" containerID="5f2f48c32ada135934e9598d86de4a9e2d0b816e7eb6cc2366ea0e3e7ebb689c"
Sep 07 13:26:24.807891 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:24.807864    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:26:24.807966 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:24.807932    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:26:24.808007 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:24.807965    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 13:26:25.813608 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:25.813576    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst" event={"ID":"87ef1aaf-726b-44ab-b453-1a861b06dbe7","Type":"ContainerStarted","Data":"232f5fd896adca332db0e004dc30b9c60b1c07b20fcb81d69d020a47ce08d2f9"}
Sep 07 13:26:26.513772 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:26.513737    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:26.514016 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:26.513993    2567 patch_prober.go:28] interesting pod/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 07 13:26:26.514105 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:26.514049    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst" podUID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 07 13:26:27.514394 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:27.514350    2567 patch_prober.go:28] interesting pod/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 07 13:26:27.514794 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:27.514419    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst" podUID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 07 13:26:28.513867 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:28.513829    2567 patch_prober.go:28] interesting pod/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 07 13:26:28.514327 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:28.513884    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst" podUID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 07 13:26:29.514789 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:29.514746    2567 patch_prober.go:28] interesting pod/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 07 13:26:29.515220 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:29.514800    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst" podUID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 07 13:26:30.514192 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:30.514154    2567 patch_prober.go:28] interesting pod/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 07 13:26:30.514409 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:30.514219    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst" podUID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 07 13:26:31.517840 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:31.517810    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:31.518280 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:31.518107    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:31.518830 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:31.518811    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:26:34.994197 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:34.994163    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lkvfr_0f8cf259-4a5d-4e00-9f43-7bee665cbd7e/payload-processing/0.log"
Sep 07 13:26:41.672790 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:41.672756    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"]
Sep 07 13:26:41.673211 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:41.673003    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b" podUID="f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1" containerName="maas-api" containerID="cri-o://0799ebdab5f6a296102c5f164d4b322076796b37868ff4d5b70745fc7fde1e95" gracePeriod=30
Sep 07 13:26:41.919180 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:41.919148    2567 generic.go:356] "Generic (PLEG): container finished" podID="f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1" containerID="0799ebdab5f6a296102c5f164d4b322076796b37868ff4d5b70745fc7fde1e95" exitCode=0
Sep 07 13:26:41.919309 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:41.919227    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b" event={"ID":"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1","Type":"ContainerDied","Data":"0799ebdab5f6a296102c5f164d4b322076796b37868ff4d5b70745fc7fde1e95"}
Sep 07 13:26:41.919309 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:41.919278    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b" event={"ID":"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1","Type":"ContainerDied","Data":"c7020925927d6222089e06dbb5d1ed746c43dacdac335298acbe95ce9a1de43d"}
Sep 07 13:26:41.919309 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:41.919294    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7020925927d6222089e06dbb5d1ed746c43dacdac335298acbe95ce9a1de43d"
Sep 07 13:26:41.931833 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:41.931812    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:26:42.039633 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.039600    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-openshift-service-ca\") pod \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") "
Sep 07 13:26:42.039819 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.039683    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw2h8\" (UniqueName: \"kubernetes.io/projected/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-kube-api-access-tw2h8\") pod \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") "
Sep 07 13:26:42.039819 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.039714    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-metrics-certs\") pod \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") "
Sep 07 13:26:42.039819 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.039742    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-maas-api-tls\") pod \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\" (UID: \"f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1\") "
Sep 07 13:26:42.040001 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.039954    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1" (UID: "f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:26:42.041841 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.041808    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1" (UID: "f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:26:42.041952 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.041848    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1" (UID: "f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:26:42.041952 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.041884    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-kube-api-access-tw2h8" (OuterVolumeSpecName: "kube-api-access-tw2h8") pod "f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1" (UID: "f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1"). InnerVolumeSpecName "kube-api-access-tw2h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:26:42.141356 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.141324    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:26:42.141356 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.141352    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw2h8\" (UniqueName: \"kubernetes.io/projected/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-kube-api-access-tw2h8\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:26:42.141356 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.141362    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:26:42.141581 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.141370    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:26:42.924638 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.924609    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"
Sep 07 13:26:42.948658 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.948621    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"]
Sep 07 13:26:42.960172 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:42.960144    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8e7ddc-58c996f644-pvr9b"]
Sep 07 13:26:44.075263 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:26:44.075229    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1" path="/var/lib/kubelet/pods/f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1/volumes"
Sep 07 13:27:10.860797 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:10.860755    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-219f63-658d774675-qvffs_c14f9c48-5510-4850-88c2-a47612dcb838/payload-processing/0.log"
Sep 07 13:27:10.981710 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:10.981667    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lkvfr_0f8cf259-4a5d-4e00-9f43-7bee665cbd7e/payload-processing/0.log"
Sep 07 13:27:16.321444 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:16.321410    2567 scope.go:117] "RemoveContainer" containerID="0799ebdab5f6a296102c5f164d4b322076796b37868ff4d5b70745fc7fde1e95"
Sep 07 13:27:16.984195 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:16.983993    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"]
Sep 07 13:27:16.984657 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:16.984634    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1" containerName="maas-api"
Sep 07 13:27:16.984835 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:16.984822    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1" containerName="maas-api"
Sep 07 13:27:16.984945 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:16.984935    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f644a7a-56bd-4026-8df6-c4b86ee680ff" containerName="maas-api"
Sep 07 13:27:16.985027 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:16.985019    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f644a7a-56bd-4026-8df6-c4b86ee680ff" containerName="maas-api"
Sep 07 13:27:16.985142 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:16.985131    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70883334-79e1-41c2-9668-c9dfddf251c1" containerName="payload-pre-processing"
Sep 07 13:27:16.985232 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:16.985223    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="70883334-79e1-41c2-9668-c9dfddf251c1" containerName="payload-pre-processing"
Sep 07 13:27:16.985414 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:16.985403    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8962cd5-0271-4a1d-b92e-1c1c43e1e9e1" containerName="maas-api"
Sep 07 13:27:16.985496 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:16.985487    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="70883334-79e1-41c2-9668-c9dfddf251c1" containerName="payload-pre-processing"
Sep 07 13:27:16.985573 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:16.985564    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f644a7a-56bd-4026-8df6-c4b86ee680ff" containerName="maas-api"
Sep 07 13:27:16.990870 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:16.990847    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"
Sep 07 13:27:16.994626 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:16.994508    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-d14aa2\""
Sep 07 13:27:17.002920 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:17.002895    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-d14aa2-dockercfg-qc7zv\""
Sep 07 13:27:17.010167 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:17.009626    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"]
Sep 07 13:27:17.083949 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:17.083910    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ad2f71-e0bc-468a-80e6-e127e6c9296d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2\" (UID: \"e2ad2f71-e0bc-468a-80e6-e127e6c9296d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"
Sep 07 13:27:17.084242 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:17.084216    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4z5j\" (UniqueName: \"kubernetes.io/projected/e2ad2f71-e0bc-468a-80e6-e127e6c9296d-kube-api-access-x4z5j\") pod \"payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2\" (UID: \"e2ad2f71-e0bc-468a-80e6-e127e6c9296d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"
Sep 07 13:27:17.184874 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:17.184835    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ad2f71-e0bc-468a-80e6-e127e6c9296d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2\" (UID: \"e2ad2f71-e0bc-468a-80e6-e127e6c9296d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"
Sep 07 13:27:17.185086 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:17.185015    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4z5j\" (UniqueName: \"kubernetes.io/projected/e2ad2f71-e0bc-468a-80e6-e127e6c9296d-kube-api-access-x4z5j\") pod \"payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2\" (UID: \"e2ad2f71-e0bc-468a-80e6-e127e6c9296d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"
Sep 07 13:27:17.185563 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:17.185540    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ad2f71-e0bc-468a-80e6-e127e6c9296d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2\" (UID: \"e2ad2f71-e0bc-468a-80e6-e127e6c9296d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"
Sep 07 13:27:17.210563 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:17.210534    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4z5j\" (UniqueName: \"kubernetes.io/projected/e2ad2f71-e0bc-468a-80e6-e127e6c9296d-kube-api-access-x4z5j\") pod \"payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2\" (UID: \"e2ad2f71-e0bc-468a-80e6-e127e6c9296d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"
Sep 07 13:27:17.314357 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:17.314325    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"
Sep 07 13:27:17.606108 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:17.606008    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"]
Sep 07 13:27:17.609135 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:27:17.609104    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ad2f71_e0bc_468a_80e6_e127e6c9296d.slice/crio-6e76a6fbecb43436c6cd805f6c50a0e1e762cb947109107bfca434a4141a7651 WatchSource:0}: Error finding container 6e76a6fbecb43436c6cd805f6c50a0e1e762cb947109107bfca434a4141a7651: Status 404 returned error can't find the container with id 6e76a6fbecb43436c6cd805f6c50a0e1e762cb947109107bfca434a4141a7651
Sep 07 13:27:18.142834 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:18.142790    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2" event={"ID":"e2ad2f71-e0bc-468a-80e6-e127e6c9296d","Type":"ContainerStarted","Data":"28f813aa72ff169b26c32439c0add0e4a0b1053be3c6501e81f1253e1e287f3f"}
Sep 07 13:27:18.143046 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:18.142967    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"
Sep 07 13:27:18.143046 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:18.143017    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2" event={"ID":"e2ad2f71-e0bc-468a-80e6-e127e6c9296d","Type":"ContainerStarted","Data":"6e76a6fbecb43436c6cd805f6c50a0e1e762cb947109107bfca434a4141a7651"}
Sep 07 13:27:18.181965 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:18.181902    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2" podStartSLOduration=2.18188269 podStartE2EDuration="2.18188269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:27:18.180551099 +0000 UTC m=+1682.608849393" watchObservedRunningTime="2026-09-07 13:27:18.18188269 +0000 UTC m=+1682.610180963"
Sep 07 13:27:29.150820 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:29.150786    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"
Sep 07 13:27:46.960701 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:46.960605    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"]
Sep 07 13:27:46.961288 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:46.960872    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2" podUID="e2ad2f71-e0bc-468a-80e6-e127e6c9296d" containerName="payload-processing" containerID="cri-o://28f813aa72ff169b26c32439c0add0e4a0b1053be3c6501e81f1253e1e287f3f" gracePeriod=30
Sep 07 13:27:47.240084 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.239467    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"
Sep 07 13:27:47.316156 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.316122    2567 generic.go:356] "Generic (PLEG): container finished" podID="e2ad2f71-e0bc-468a-80e6-e127e6c9296d" containerID="28f813aa72ff169b26c32439c0add0e4a0b1053be3c6501e81f1253e1e287f3f" exitCode=0
Sep 07 13:27:47.316361 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.316167    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"
Sep 07 13:27:47.316361 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.316210    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2" event={"ID":"e2ad2f71-e0bc-468a-80e6-e127e6c9296d","Type":"ContainerDied","Data":"28f813aa72ff169b26c32439c0add0e4a0b1053be3c6501e81f1253e1e287f3f"}
Sep 07 13:27:47.316361 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.316270    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2" event={"ID":"e2ad2f71-e0bc-468a-80e6-e127e6c9296d","Type":"ContainerDied","Data":"6e76a6fbecb43436c6cd805f6c50a0e1e762cb947109107bfca434a4141a7651"}
Sep 07 13:27:47.316361 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.316300    2567 scope.go:117] "RemoveContainer" containerID="28f813aa72ff169b26c32439c0add0e4a0b1053be3c6501e81f1253e1e287f3f"
Sep 07 13:27:47.326598 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.326564    2567 scope.go:117] "RemoveContainer" containerID="28f813aa72ff169b26c32439c0add0e4a0b1053be3c6501e81f1253e1e287f3f"
Sep 07 13:27:47.327235 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:27:47.327163    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f813aa72ff169b26c32439c0add0e4a0b1053be3c6501e81f1253e1e287f3f\": container with ID starting with 28f813aa72ff169b26c32439c0add0e4a0b1053be3c6501e81f1253e1e287f3f not found: ID does not exist" containerID="28f813aa72ff169b26c32439c0add0e4a0b1053be3c6501e81f1253e1e287f3f"
Sep 07 13:27:47.327235 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.327201    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f813aa72ff169b26c32439c0add0e4a0b1053be3c6501e81f1253e1e287f3f"} err="failed to get container status \"28f813aa72ff169b26c32439c0add0e4a0b1053be3c6501e81f1253e1e287f3f\": rpc error: code = NotFound desc = could not find container \"28f813aa72ff169b26c32439c0add0e4a0b1053be3c6501e81f1253e1e287f3f\": container with ID starting with 28f813aa72ff169b26c32439c0add0e4a0b1053be3c6501e81f1253e1e287f3f not found: ID does not exist"
Sep 07 13:27:47.372870 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.372844    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ad2f71-e0bc-468a-80e6-e127e6c9296d-plugins-config-volume\") pod \"e2ad2f71-e0bc-468a-80e6-e127e6c9296d\" (UID: \"e2ad2f71-e0bc-468a-80e6-e127e6c9296d\") "
Sep 07 13:27:47.373021 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.372876    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4z5j\" (UniqueName: \"kubernetes.io/projected/e2ad2f71-e0bc-468a-80e6-e127e6c9296d-kube-api-access-x4z5j\") pod \"e2ad2f71-e0bc-468a-80e6-e127e6c9296d\" (UID: \"e2ad2f71-e0bc-468a-80e6-e127e6c9296d\") "
Sep 07 13:27:47.373252 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.373222    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ad2f71-e0bc-468a-80e6-e127e6c9296d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2ad2f71-e0bc-468a-80e6-e127e6c9296d" (UID: "e2ad2f71-e0bc-468a-80e6-e127e6c9296d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:27:47.375160 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.375132    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ad2f71-e0bc-468a-80e6-e127e6c9296d-kube-api-access-x4z5j" (OuterVolumeSpecName: "kube-api-access-x4z5j") pod "e2ad2f71-e0bc-468a-80e6-e127e6c9296d" (UID: "e2ad2f71-e0bc-468a-80e6-e127e6c9296d"). InnerVolumeSpecName "kube-api-access-x4z5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:27:47.474681 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.474596    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ad2f71-e0bc-468a-80e6-e127e6c9296d-plugins-config-volume\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:27:47.474681 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.474628    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4z5j\" (UniqueName: \"kubernetes.io/projected/e2ad2f71-e0bc-468a-80e6-e127e6c9296d-kube-api-access-x4z5j\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:27:47.644236 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.644207    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"]
Sep 07 13:27:47.650667 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:47.650641    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d14aa2-795bd867cc-st4c2"]
Sep 07 13:27:48.075192 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:27:48.075154    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ad2f71-e0bc-468a-80e6-e127e6c9296d" path="/var/lib/kubelet/pods/e2ad2f71-e0bc-468a-80e6-e127e6c9296d/volumes"
Sep 07 13:28:13.936615 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:13.936579    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"]
Sep 07 13:28:13.937114 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:13.936791    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs" podUID="c14f9c48-5510-4850-88c2-a47612dcb838" containerName="payload-processing" containerID="cri-o://15f157041b61dccf2f0828b87d061744806649db87f169137eb0395736a43d60" gracePeriod=30
Sep 07 13:28:14.213127 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.213101    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"
Sep 07 13:28:14.316837 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.316803    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14f9c48-5510-4850-88c2-a47612dcb838-plugins-config-volume\") pod \"c14f9c48-5510-4850-88c2-a47612dcb838\" (UID: \"c14f9c48-5510-4850-88c2-a47612dcb838\") "
Sep 07 13:28:14.316990 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.316890    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxqqq\" (UniqueName: \"kubernetes.io/projected/c14f9c48-5510-4850-88c2-a47612dcb838-kube-api-access-cxqqq\") pod \"c14f9c48-5510-4850-88c2-a47612dcb838\" (UID: \"c14f9c48-5510-4850-88c2-a47612dcb838\") "
Sep 07 13:28:14.317179 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.317158    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14f9c48-5510-4850-88c2-a47612dcb838-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c14f9c48-5510-4850-88c2-a47612dcb838" (UID: "c14f9c48-5510-4850-88c2-a47612dcb838"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:28:14.318971 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.318946    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14f9c48-5510-4850-88c2-a47612dcb838-kube-api-access-cxqqq" (OuterVolumeSpecName: "kube-api-access-cxqqq") pod "c14f9c48-5510-4850-88c2-a47612dcb838" (UID: "c14f9c48-5510-4850-88c2-a47612dcb838"). InnerVolumeSpecName "kube-api-access-cxqqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:28:14.417949 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.417915    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14f9c48-5510-4850-88c2-a47612dcb838-plugins-config-volume\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:28:14.417949 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.417942    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxqqq\" (UniqueName: \"kubernetes.io/projected/c14f9c48-5510-4850-88c2-a47612dcb838-kube-api-access-cxqqq\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:28:14.480667 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.480579    2567 generic.go:356] "Generic (PLEG): container finished" podID="c14f9c48-5510-4850-88c2-a47612dcb838" containerID="15f157041b61dccf2f0828b87d061744806649db87f169137eb0395736a43d60" exitCode=0
Sep 07 13:28:14.480667 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.480632    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"
Sep 07 13:28:14.480873 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.480664    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs" event={"ID":"c14f9c48-5510-4850-88c2-a47612dcb838","Type":"ContainerDied","Data":"15f157041b61dccf2f0828b87d061744806649db87f169137eb0395736a43d60"}
Sep 07 13:28:14.480873 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.480735    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs" event={"ID":"c14f9c48-5510-4850-88c2-a47612dcb838","Type":"ContainerDied","Data":"637910bec927a96f95510bda0b48367e80820fbaf979d9842ced5995e259895c"}
Sep 07 13:28:14.480873 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.480756    2567 scope.go:117] "RemoveContainer" containerID="15f157041b61dccf2f0828b87d061744806649db87f169137eb0395736a43d60"
Sep 07 13:28:14.490075 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.490042    2567 scope.go:117] "RemoveContainer" containerID="15f157041b61dccf2f0828b87d061744806649db87f169137eb0395736a43d60"
Sep 07 13:28:14.490373 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:28:14.490355    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f157041b61dccf2f0828b87d061744806649db87f169137eb0395736a43d60\": container with ID starting with 15f157041b61dccf2f0828b87d061744806649db87f169137eb0395736a43d60 not found: ID does not exist" containerID="15f157041b61dccf2f0828b87d061744806649db87f169137eb0395736a43d60"
Sep 07 13:28:14.490414 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.490382    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f157041b61dccf2f0828b87d061744806649db87f169137eb0395736a43d60"} err="failed to get container status \"15f157041b61dccf2f0828b87d061744806649db87f169137eb0395736a43d60\": rpc error: code = NotFound desc = could not find container \"15f157041b61dccf2f0828b87d061744806649db87f169137eb0395736a43d60\": container with ID starting with 15f157041b61dccf2f0828b87d061744806649db87f169137eb0395736a43d60 not found: ID does not exist"
Sep 07 13:28:14.503480 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.503456    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"]
Sep 07 13:28:14.508811 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:14.508792    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-219f63-658d774675-qvffs"]
Sep 07 13:28:16.077290 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:28:16.077255    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c14f9c48-5510-4850-88c2-a47612dcb838" path="/var/lib/kubelet/pods/c14f9c48-5510-4850-88c2-a47612dcb838/volumes"
Sep 07 13:29:14.620259 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:14.620219    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"]
Sep 07 13:29:14.620711 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:14.620473    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546" podUID="6554a9a0-9500-439d-a3bc-52e5d45a9ba8" containerName="payload-pre-processing" containerID="cri-o://169be037f112acc3f15f9aa5d97109c80fd50ce4f9e5a10fa15bc232df54528b" gracePeriod=30
Sep 07 13:29:14.839732 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:14.839698    2567 generic.go:356] "Generic (PLEG): container finished" podID="6554a9a0-9500-439d-a3bc-52e5d45a9ba8" containerID="169be037f112acc3f15f9aa5d97109c80fd50ce4f9e5a10fa15bc232df54528b" exitCode=0
Sep 07 13:29:14.840101 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:14.839766    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546" event={"ID":"6554a9a0-9500-439d-a3bc-52e5d45a9ba8","Type":"ContainerDied","Data":"169be037f112acc3f15f9aa5d97109c80fd50ce4f9e5a10fa15bc232df54528b"}
Sep 07 13:29:14.901326 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:14.901299    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"
Sep 07 13:29:15.095538 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:15.095497    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnjfc\" (UniqueName: \"kubernetes.io/projected/6554a9a0-9500-439d-a3bc-52e5d45a9ba8-kube-api-access-qnjfc\") pod \"6554a9a0-9500-439d-a3bc-52e5d45a9ba8\" (UID: \"6554a9a0-9500-439d-a3bc-52e5d45a9ba8\") "
Sep 07 13:29:15.095719 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:15.095659    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6554a9a0-9500-439d-a3bc-52e5d45a9ba8-plugins-config-volume\") pod \"6554a9a0-9500-439d-a3bc-52e5d45a9ba8\" (UID: \"6554a9a0-9500-439d-a3bc-52e5d45a9ba8\") "
Sep 07 13:29:15.096021 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:15.095998    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6554a9a0-9500-439d-a3bc-52e5d45a9ba8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6554a9a0-9500-439d-a3bc-52e5d45a9ba8" (UID: "6554a9a0-9500-439d-a3bc-52e5d45a9ba8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:29:15.097591 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:15.097570    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6554a9a0-9500-439d-a3bc-52e5d45a9ba8-kube-api-access-qnjfc" (OuterVolumeSpecName: "kube-api-access-qnjfc") pod "6554a9a0-9500-439d-a3bc-52e5d45a9ba8" (UID: "6554a9a0-9500-439d-a3bc-52e5d45a9ba8"). InnerVolumeSpecName "kube-api-access-qnjfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:29:15.196420 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:15.196380    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6554a9a0-9500-439d-a3bc-52e5d45a9ba8-plugins-config-volume\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:15.196420 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:15.196417    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnjfc\" (UniqueName: \"kubernetes.io/projected/6554a9a0-9500-439d-a3bc-52e5d45a9ba8-kube-api-access-qnjfc\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:15.847308 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:15.847274    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"
Sep 07 13:29:15.847308 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:15.847290    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546" event={"ID":"6554a9a0-9500-439d-a3bc-52e5d45a9ba8","Type":"ContainerDied","Data":"e0a7273b6db55ae25b6141ca1936d83c99e55411a67be9f8d708c96b063c3c7c"}
Sep 07 13:29:15.847825 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:15.847362    2567 scope.go:117] "RemoveContainer" containerID="169be037f112acc3f15f9aa5d97109c80fd50ce4f9e5a10fa15bc232df54528b"
Sep 07 13:29:15.872611 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:15.872578    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"]
Sep 07 13:29:15.881947 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:15.881918    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8dc72a-9c5485fbb-b8546"]
Sep 07 13:29:16.075626 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:16.075591    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6554a9a0-9500-439d-a3bc-52e5d45a9ba8" path="/var/lib/kubelet/pods/6554a9a0-9500-439d-a3bc-52e5d45a9ba8/volumes"
Sep 07 13:29:21.763641 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:21.763607    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"]
Sep 07 13:29:21.764037 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:21.763836    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst" podUID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerName="istio-proxy" containerID="cri-o://232f5fd896adca332db0e004dc30b9c60b1c07b20fcb81d69d020a47ce08d2f9" gracePeriod=30
Sep 07 13:29:26.914894 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:26.914860    2567 generic.go:356] "Generic (PLEG): container finished" podID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerID="232f5fd896adca332db0e004dc30b9c60b1c07b20fcb81d69d020a47ce08d2f9" exitCode=0
Sep 07 13:29:26.915339 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:26.914937    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst" event={"ID":"87ef1aaf-726b-44ab-b453-1a861b06dbe7","Type":"ContainerDied","Data":"232f5fd896adca332db0e004dc30b9c60b1c07b20fcb81d69d020a47ce08d2f9"}
Sep 07 13:29:26.915339 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:26.915018    2567 scope.go:117] "RemoveContainer" containerID="5f2f48c32ada135934e9598d86de4a9e2d0b816e7eb6cc2366ea0e3e7ebb689c"
Sep 07 13:29:27.017730 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.017700    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:29:27.119272 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119172    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-podinfo\") pod \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") "
Sep 07 13:29:27.119272 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119248    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-envoy\") pod \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") "
Sep 07 13:29:27.119272 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119268    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-credential-socket\") pod \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") "
Sep 07 13:29:27.119617 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119305    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-token\") pod \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") "
Sep 07 13:29:27.119617 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119330    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istiod-ca-cert\") pod \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") "
Sep 07 13:29:27.119617 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119367    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwpj5\" (UniqueName: \"kubernetes.io/projected/87ef1aaf-726b-44ab-b453-1a861b06dbe7-kube-api-access-hwpj5\") pod \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") "
Sep 07 13:29:27.119617 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119393    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-workload-certs\") pod \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") "
Sep 07 13:29:27.119617 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119444    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-data\") pod \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") "
Sep 07 13:29:27.119617 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119472    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-workload-socket\") pod \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\" (UID: \"87ef1aaf-726b-44ab-b453-1a861b06dbe7\") "
Sep 07 13:29:27.119956 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119610    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "87ef1aaf-726b-44ab-b453-1a861b06dbe7" (UID: "87ef1aaf-726b-44ab-b453-1a861b06dbe7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:29:27.119956 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119769    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "87ef1aaf-726b-44ab-b453-1a861b06dbe7" (UID: "87ef1aaf-726b-44ab-b453-1a861b06dbe7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:29:27.119956 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119822    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-credential-socket\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:27.119956 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119842    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "87ef1aaf-726b-44ab-b453-1a861b06dbe7" (UID: "87ef1aaf-726b-44ab-b453-1a861b06dbe7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:29:27.119956 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119871    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "87ef1aaf-726b-44ab-b453-1a861b06dbe7" (UID: "87ef1aaf-726b-44ab-b453-1a861b06dbe7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:29:27.119956 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.119949    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-data" (OuterVolumeSpecName: "istio-data") pod "87ef1aaf-726b-44ab-b453-1a861b06dbe7" (UID: "87ef1aaf-726b-44ab-b453-1a861b06dbe7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:29:27.121605 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.121581    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "87ef1aaf-726b-44ab-b453-1a861b06dbe7" (UID: "87ef1aaf-726b-44ab-b453-1a861b06dbe7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:29:27.121732 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.121642    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ef1aaf-726b-44ab-b453-1a861b06dbe7-kube-api-access-hwpj5" (OuterVolumeSpecName: "kube-api-access-hwpj5") pod "87ef1aaf-726b-44ab-b453-1a861b06dbe7" (UID: "87ef1aaf-726b-44ab-b453-1a861b06dbe7"). InnerVolumeSpecName "kube-api-access-hwpj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:29:27.121732 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.121677    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "87ef1aaf-726b-44ab-b453-1a861b06dbe7" (UID: "87ef1aaf-726b-44ab-b453-1a861b06dbe7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:29:27.121732 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.121680    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-token" (OuterVolumeSpecName: "istio-token") pod "87ef1aaf-726b-44ab-b453-1a861b06dbe7" (UID: "87ef1aaf-726b-44ab-b453-1a861b06dbe7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:29:27.220620 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.220586    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-token\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:27.220620 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.220615    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istiod-ca-cert\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:27.220620 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.220625    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwpj5\" (UniqueName: \"kubernetes.io/projected/87ef1aaf-726b-44ab-b453-1a861b06dbe7-kube-api-access-hwpj5\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:27.220880 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.220635    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-workload-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:27.220880 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.220644    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-data\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:27.220880 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.220652    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-workload-socket\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:27.220880 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.220659    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-podinfo\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:27.220880 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.220667    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87ef1aaf-726b-44ab-b453-1a861b06dbe7-istio-envoy\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:27.922702 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.922661    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"
Sep 07 13:29:27.923167 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.922660    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst" event={"ID":"87ef1aaf-726b-44ab-b453-1a861b06dbe7","Type":"ContainerDied","Data":"c1f6b56ba4b77392f04eee2cd1202a7b2d8d467086af3de913069ae76ef4eb74"}
Sep 07 13:29:27.923167 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.922820    2567 scope.go:117] "RemoveContainer" containerID="232f5fd896adca332db0e004dc30b9c60b1c07b20fcb81d69d020a47ce08d2f9"
Sep 07 13:29:27.945260 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.945235    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"]
Sep 07 13:29:27.952211 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:27.952188    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8dc72a-openshift-default-5cd7b44f4-jmbst"]
Sep 07 13:29:28.077510 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:28.077470    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" path="/var/lib/kubelet/pods/87ef1aaf-726b-44ab-b453-1a861b06dbe7/volumes"
Sep 07 13:29:32.920231 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:32.920194    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"]
Sep 07 13:29:32.920632 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:32.920415    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr" podUID="e5e7fa90-f906-4ee5-af46-2b942481bd11" containerName="maas-api" containerID="cri-o://68b9c339ecc9f1fd18b1b5f5a53ee614ea51c6dda6fed0a8020c1ec60e3d2786" gracePeriod=30
Sep 07 13:29:33.046070 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.046020    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr" podUID="e5e7fa90-f906-4ee5-af46-2b942481bd11" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.61:8443/health\": dial tcp 10.132.0.61:8443: connect: connection refused"
Sep 07 13:29:33.175535 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.175471    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:29:33.279075 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.279019    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g4jz\" (UniqueName: \"kubernetes.io/projected/e5e7fa90-f906-4ee5-af46-2b942481bd11-kube-api-access-9g4jz\") pod \"e5e7fa90-f906-4ee5-af46-2b942481bd11\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") "
Sep 07 13:29:33.279377 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.279093    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-metrics-certs\") pod \"e5e7fa90-f906-4ee5-af46-2b942481bd11\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") "
Sep 07 13:29:33.279377 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.279134    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-maas-api-tls\") pod \"e5e7fa90-f906-4ee5-af46-2b942481bd11\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") "
Sep 07 13:29:33.279377 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.279174    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e7fa90-f906-4ee5-af46-2b942481bd11-openshift-service-ca\") pod \"e5e7fa90-f906-4ee5-af46-2b942481bd11\" (UID: \"e5e7fa90-f906-4ee5-af46-2b942481bd11\") "
Sep 07 13:29:33.279690 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.279651    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e7fa90-f906-4ee5-af46-2b942481bd11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5e7fa90-f906-4ee5-af46-2b942481bd11" (UID: "e5e7fa90-f906-4ee5-af46-2b942481bd11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:29:33.281496 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.281467    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5e7fa90-f906-4ee5-af46-2b942481bd11" (UID: "e5e7fa90-f906-4ee5-af46-2b942481bd11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:29:33.281733 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.281704    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e5e7fa90-f906-4ee5-af46-2b942481bd11" (UID: "e5e7fa90-f906-4ee5-af46-2b942481bd11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:29:33.281733 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.281709    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e7fa90-f906-4ee5-af46-2b942481bd11-kube-api-access-9g4jz" (OuterVolumeSpecName: "kube-api-access-9g4jz") pod "e5e7fa90-f906-4ee5-af46-2b942481bd11" (UID: "e5e7fa90-f906-4ee5-af46-2b942481bd11"). InnerVolumeSpecName "kube-api-access-9g4jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:29:33.380115 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.380038    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g4jz\" (UniqueName: \"kubernetes.io/projected/e5e7fa90-f906-4ee5-af46-2b942481bd11-kube-api-access-9g4jz\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:33.380115 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.380101    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:33.380115 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.380117    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5e7fa90-f906-4ee5-af46-2b942481bd11-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:33.380359 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.380129    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e7fa90-f906-4ee5-af46-2b942481bd11-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:33.961120 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.961075    2567 generic.go:356] "Generic (PLEG): container finished" podID="e5e7fa90-f906-4ee5-af46-2b942481bd11" containerID="68b9c339ecc9f1fd18b1b5f5a53ee614ea51c6dda6fed0a8020c1ec60e3d2786" exitCode=0
Sep 07 13:29:33.961498 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.961155    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"
Sep 07 13:29:33.961498 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.961149    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr" event={"ID":"e5e7fa90-f906-4ee5-af46-2b942481bd11","Type":"ContainerDied","Data":"68b9c339ecc9f1fd18b1b5f5a53ee614ea51c6dda6fed0a8020c1ec60e3d2786"}
Sep 07 13:29:33.961498 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.961301    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr" event={"ID":"e5e7fa90-f906-4ee5-af46-2b942481bd11","Type":"ContainerDied","Data":"b2b5f7ab37035ad17773c3a84fbb664635c3cb1b1169c2df8e2588e645588d30"}
Sep 07 13:29:33.961498 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.961323    2567 scope.go:117] "RemoveContainer" containerID="68b9c339ecc9f1fd18b1b5f5a53ee614ea51c6dda6fed0a8020c1ec60e3d2786"
Sep 07 13:29:33.970992 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.970972    2567 scope.go:117] "RemoveContainer" containerID="68b9c339ecc9f1fd18b1b5f5a53ee614ea51c6dda6fed0a8020c1ec60e3d2786"
Sep 07 13:29:33.971317 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:29:33.971296    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b9c339ecc9f1fd18b1b5f5a53ee614ea51c6dda6fed0a8020c1ec60e3d2786\": container with ID starting with 68b9c339ecc9f1fd18b1b5f5a53ee614ea51c6dda6fed0a8020c1ec60e3d2786 not found: ID does not exist" containerID="68b9c339ecc9f1fd18b1b5f5a53ee614ea51c6dda6fed0a8020c1ec60e3d2786"
Sep 07 13:29:33.971374 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.971327    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b9c339ecc9f1fd18b1b5f5a53ee614ea51c6dda6fed0a8020c1ec60e3d2786"} err="failed to get container status \"68b9c339ecc9f1fd18b1b5f5a53ee614ea51c6dda6fed0a8020c1ec60e3d2786\": rpc error: code = NotFound desc = could not find container \"68b9c339ecc9f1fd18b1b5f5a53ee614ea51c6dda6fed0a8020c1ec60e3d2786\": container with ID starting with 68b9c339ecc9f1fd18b1b5f5a53ee614ea51c6dda6fed0a8020c1ec60e3d2786 not found: ID does not exist"
Sep 07 13:29:33.990575 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.990545    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"]
Sep 07 13:29:33.992775 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:33.992754    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5910bb-84786878fd-9nzkr"]
Sep 07 13:29:34.075783 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:34.075743    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e7fa90-f906-4ee5-af46-2b942481bd11" path="/var/lib/kubelet/pods/e5e7fa90-f906-4ee5-af46-2b942481bd11/volumes"
Sep 07 13:29:40.875360 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:40.875319    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"]
Sep 07 13:29:40.875797 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:40.875581    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy" containerID="cri-o://29680f742e73ba27d8a70a01228ba672e149ee260985f3c948689d4cec066e13" gracePeriod=30
Sep 07 13:29:41.729437 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:41.729397    2567 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 13:29:41.729590 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:41.729472    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 13:29:46.035236 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.035194    2567 generic.go:356] "Generic (PLEG): container finished" podID="863eb763-4af8-41b7-b579-2652c3351065" containerID="29680f742e73ba27d8a70a01228ba672e149ee260985f3c948689d4cec066e13" exitCode=0
Sep 07 13:29:46.035676 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.035263    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" event={"ID":"863eb763-4af8-41b7-b579-2652c3351065","Type":"ContainerDied","Data":"29680f742e73ba27d8a70a01228ba672e149ee260985f3c948689d4cec066e13"}
Sep 07 13:29:46.035676 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.035376    2567 scope.go:117] "RemoveContainer" containerID="69afb87447f753de05c38b47d4b219fae103047c099386fbb574f23b2b794701"
Sep 07 13:29:46.134509 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.134485    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:29:46.206104 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.205987    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-credential-socket\") pod \"863eb763-4af8-41b7-b579-2652c3351065\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") "
Sep 07 13:29:46.206104 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206037    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863eb763-4af8-41b7-b579-2652c3351065-istio-token\") pod \"863eb763-4af8-41b7-b579-2652c3351065\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") "
Sep 07 13:29:46.206104 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206098    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-workload-socket\") pod \"863eb763-4af8-41b7-b579-2652c3351065\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") "
Sep 07 13:29:46.206381 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206125    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-istio-envoy\") pod \"863eb763-4af8-41b7-b579-2652c3351065\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") "
Sep 07 13:29:46.206381 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206319    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-istio-data\") pod \"863eb763-4af8-41b7-b579-2652c3351065\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") "
Sep 07 13:29:46.206381 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206369    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr5cm\" (UniqueName: \"kubernetes.io/projected/863eb763-4af8-41b7-b579-2652c3351065-kube-api-access-mr5cm\") pod \"863eb763-4af8-41b7-b579-2652c3351065\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") "
Sep 07 13:29:46.206381 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206374    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "863eb763-4af8-41b7-b579-2652c3351065" (UID: "863eb763-4af8-41b7-b579-2652c3351065"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:29:46.207015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206394    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-workload-certs\") pod \"863eb763-4af8-41b7-b579-2652c3351065\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") "
Sep 07 13:29:46.207015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206465    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863eb763-4af8-41b7-b579-2652c3351065-istio-podinfo\") pod \"863eb763-4af8-41b7-b579-2652c3351065\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") "
Sep 07 13:29:46.207015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206503    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "863eb763-4af8-41b7-b579-2652c3351065" (UID: "863eb763-4af8-41b7-b579-2652c3351065"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:29:46.207015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206532    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863eb763-4af8-41b7-b579-2652c3351065-istiod-ca-cert\") pod \"863eb763-4af8-41b7-b579-2652c3351065\" (UID: \"863eb763-4af8-41b7-b579-2652c3351065\") "
Sep 07 13:29:46.207015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206609    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-istio-data" (OuterVolumeSpecName: "istio-data") pod "863eb763-4af8-41b7-b579-2652c3351065" (UID: "863eb763-4af8-41b7-b579-2652c3351065"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:29:46.207015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206696    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "863eb763-4af8-41b7-b579-2652c3351065" (UID: "863eb763-4af8-41b7-b579-2652c3351065"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:29:46.207015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206877    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-workload-socket\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:46.207015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206912    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-istio-data\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:46.207015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206926    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-workload-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:46.207015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.206939    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-credential-socket\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:46.207534 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.207427    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863eb763-4af8-41b7-b579-2652c3351065-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "863eb763-4af8-41b7-b579-2652c3351065" (UID: "863eb763-4af8-41b7-b579-2652c3351065"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:29:46.208610 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.208587    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863eb763-4af8-41b7-b579-2652c3351065-kube-api-access-mr5cm" (OuterVolumeSpecName: "kube-api-access-mr5cm") pod "863eb763-4af8-41b7-b579-2652c3351065" (UID: "863eb763-4af8-41b7-b579-2652c3351065"). InnerVolumeSpecName "kube-api-access-mr5cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:29:46.208705 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.208625    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/863eb763-4af8-41b7-b579-2652c3351065-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "863eb763-4af8-41b7-b579-2652c3351065" (UID: "863eb763-4af8-41b7-b579-2652c3351065"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:29:46.208873 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.208849    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "863eb763-4af8-41b7-b579-2652c3351065" (UID: "863eb763-4af8-41b7-b579-2652c3351065"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:29:46.208913 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.208899    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863eb763-4af8-41b7-b579-2652c3351065-istio-token" (OuterVolumeSpecName: "istio-token") pod "863eb763-4af8-41b7-b579-2652c3351065" (UID: "863eb763-4af8-41b7-b579-2652c3351065"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:29:46.307974 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.307936    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863eb763-4af8-41b7-b579-2652c3351065-istiod-ca-cert\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:46.307974 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.307968    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863eb763-4af8-41b7-b579-2652c3351065-istio-token\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:46.307974 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.307976    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863eb763-4af8-41b7-b579-2652c3351065-istio-envoy\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:46.308242 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.307985    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr5cm\" (UniqueName: \"kubernetes.io/projected/863eb763-4af8-41b7-b579-2652c3351065-kube-api-access-mr5cm\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:46.308242 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:46.307995    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863eb763-4af8-41b7-b579-2652c3351065-istio-podinfo\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:47.043015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:47.042980    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs" event={"ID":"863eb763-4af8-41b7-b579-2652c3351065","Type":"ContainerDied","Data":"bb1422e656828fbf0e2ccac3878563ad0d080b05330cd1532a14dd625d99a561"}
Sep 07 13:29:47.043476 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:47.043053    2567 scope.go:117] "RemoveContainer" containerID="29680f742e73ba27d8a70a01228ba672e149ee260985f3c948689d4cec066e13"
Sep 07 13:29:47.043476 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:47.042995    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"
Sep 07 13:29:47.077407 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:47.077372    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"]
Sep 07 13:29:47.087502 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:47.087472    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5910bb-openshift-default-8454df58d6-ml5bs"]
Sep 07 13:29:48.075009 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:48.074976    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="863eb763-4af8-41b7-b579-2652c3351065" path="/var/lib/kubelet/pods/863eb763-4af8-41b7-b579-2652c3351065/volumes"
Sep 07 13:29:51.947190 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:51.947154    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"]
Sep 07 13:29:51.947687 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:51.947470    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44" podUID="41f02309-4419-4035-999e-4eff749cc77b" containerName="maas-api" containerID="cri-o://824786c296df8184d29a3283adbc4b29ae69dd244a1b20343de7857067902f8d" gracePeriod=30
Sep 07 13:29:52.080671 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.080635    2567 generic.go:356] "Generic (PLEG): container finished" podID="41f02309-4419-4035-999e-4eff749cc77b" containerID="824786c296df8184d29a3283adbc4b29ae69dd244a1b20343de7857067902f8d" exitCode=0
Sep 07 13:29:52.080874 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.080750    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44" event={"ID":"41f02309-4419-4035-999e-4eff749cc77b","Type":"ContainerDied","Data":"824786c296df8184d29a3283adbc4b29ae69dd244a1b20343de7857067902f8d"}
Sep 07 13:29:52.193575 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.193547    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:29:52.365366 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.365322    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk2q8\" (UniqueName: \"kubernetes.io/projected/41f02309-4419-4035-999e-4eff749cc77b-kube-api-access-tk2q8\") pod \"41f02309-4419-4035-999e-4eff749cc77b\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") "
Sep 07 13:29:52.365572 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.365386    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-maas-api-tls\") pod \"41f02309-4419-4035-999e-4eff749cc77b\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") "
Sep 07 13:29:52.365572 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.365448    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-metrics-certs\") pod \"41f02309-4419-4035-999e-4eff749cc77b\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") "
Sep 07 13:29:52.365572 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.365512    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f02309-4419-4035-999e-4eff749cc77b-openshift-service-ca\") pod \"41f02309-4419-4035-999e-4eff749cc77b\" (UID: \"41f02309-4419-4035-999e-4eff749cc77b\") "
Sep 07 13:29:52.365957 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.365923    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f02309-4419-4035-999e-4eff749cc77b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "41f02309-4419-4035-999e-4eff749cc77b" (UID: "41f02309-4419-4035-999e-4eff749cc77b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:29:52.367525 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.367502    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "41f02309-4419-4035-999e-4eff749cc77b" (UID: "41f02309-4419-4035-999e-4eff749cc77b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:29:52.367649 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.367555    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f02309-4419-4035-999e-4eff749cc77b-kube-api-access-tk2q8" (OuterVolumeSpecName: "kube-api-access-tk2q8") pod "41f02309-4419-4035-999e-4eff749cc77b" (UID: "41f02309-4419-4035-999e-4eff749cc77b"). InnerVolumeSpecName "kube-api-access-tk2q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:29:52.367649 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.367585    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "41f02309-4419-4035-999e-4eff749cc77b" (UID: "41f02309-4419-4035-999e-4eff749cc77b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:29:52.466482 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.466446    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk2q8\" (UniqueName: \"kubernetes.io/projected/41f02309-4419-4035-999e-4eff749cc77b-kube-api-access-tk2q8\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:52.466482 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.466479    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:52.466482 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.466489    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41f02309-4419-4035-999e-4eff749cc77b-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:52.466729 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:52.466497    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f02309-4419-4035-999e-4eff749cc77b-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:29:53.088716 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:53.088683    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44" event={"ID":"41f02309-4419-4035-999e-4eff749cc77b","Type":"ContainerDied","Data":"be7d83e8bc0a23654dbc1c63886ba42880be9b510f02286fdd9d73bdd89ad19c"}
Sep 07 13:29:53.089228 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:53.088722    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"
Sep 07 13:29:53.089228 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:53.088746    2567 scope.go:117] "RemoveContainer" containerID="824786c296df8184d29a3283adbc4b29ae69dd244a1b20343de7857067902f8d"
Sep 07 13:29:53.114080 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:53.114022    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"]
Sep 07 13:29:53.116792 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:53.116765    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c7c371-68ff675bd4-7rf44"]
Sep 07 13:29:54.075568 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:29:54.075533    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41f02309-4419-4035-999e-4eff749cc77b" path="/var/lib/kubelet/pods/41f02309-4419-4035-999e-4eff749cc77b/volumes"
Sep 07 13:30:02.251175 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:02.251136    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66fb99b89f-gtwxv"]
Sep 07 13:30:02.251719 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:02.251342    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-66fb99b89f-gtwxv" podUID="d4ab6f84-8ff8-4270-8b8b-868329a20cba" containerName="manager" containerID="cri-o://27ae9f62791ade57413e213f0844dadc68060f19eb9b4c500077761363efd201" gracePeriod=30
Sep 07 13:30:02.504975 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:02.504909    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:30:02.553497 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:02.553457    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4ab6f84-8ff8-4270-8b8b-868329a20cba-webhook-cert\") pod \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\" (UID: \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\") "
Sep 07 13:30:02.553664 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:02.553527    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7dwb\" (UniqueName: \"kubernetes.io/projected/d4ab6f84-8ff8-4270-8b8b-868329a20cba-kube-api-access-b7dwb\") pod \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\" (UID: \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\") "
Sep 07 13:30:02.553664 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:02.553551    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4ab6f84-8ff8-4270-8b8b-868329a20cba-metrics-certs\") pod \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\" (UID: \"d4ab6f84-8ff8-4270-8b8b-868329a20cba\") "
Sep 07 13:30:02.555745 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:02.555707    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ab6f84-8ff8-4270-8b8b-868329a20cba-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d4ab6f84-8ff8-4270-8b8b-868329a20cba" (UID: "d4ab6f84-8ff8-4270-8b8b-868329a20cba"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:30:02.555910 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:02.555864    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ab6f84-8ff8-4270-8b8b-868329a20cba-kube-api-access-b7dwb" (OuterVolumeSpecName: "kube-api-access-b7dwb") pod "d4ab6f84-8ff8-4270-8b8b-868329a20cba" (UID: "d4ab6f84-8ff8-4270-8b8b-868329a20cba"). InnerVolumeSpecName "kube-api-access-b7dwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:30:02.555996 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:02.555973    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ab6f84-8ff8-4270-8b8b-868329a20cba-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d4ab6f84-8ff8-4270-8b8b-868329a20cba" (UID: "d4ab6f84-8ff8-4270-8b8b-868329a20cba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:30:02.654375 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:02.654333    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4ab6f84-8ff8-4270-8b8b-868329a20cba-webhook-cert\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:30:02.654375 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:02.654365    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7dwb\" (UniqueName: \"kubernetes.io/projected/d4ab6f84-8ff8-4270-8b8b-868329a20cba-kube-api-access-b7dwb\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:30:02.654375 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:02.654376    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4ab6f84-8ff8-4270-8b8b-868329a20cba-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:30:03.146800 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:03.146765    2567 generic.go:356] "Generic (PLEG): container finished" podID="d4ab6f84-8ff8-4270-8b8b-868329a20cba" containerID="27ae9f62791ade57413e213f0844dadc68060f19eb9b4c500077761363efd201" exitCode=0
Sep 07 13:30:03.147015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:03.146828    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fb99b89f-gtwxv"
Sep 07 13:30:03.147015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:03.146849    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fb99b89f-gtwxv" event={"ID":"d4ab6f84-8ff8-4270-8b8b-868329a20cba","Type":"ContainerDied","Data":"27ae9f62791ade57413e213f0844dadc68060f19eb9b4c500077761363efd201"}
Sep 07 13:30:03.147015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:03.146913    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fb99b89f-gtwxv" event={"ID":"d4ab6f84-8ff8-4270-8b8b-868329a20cba","Type":"ContainerDied","Data":"931f09e6c25ec8fcbc592bec162b6ebe303092326fa8d5104b9376d63dfe9164"}
Sep 07 13:30:03.147015 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:03.146928    2567 scope.go:117] "RemoveContainer" containerID="27ae9f62791ade57413e213f0844dadc68060f19eb9b4c500077761363efd201"
Sep 07 13:30:03.157163 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:03.157136    2567 scope.go:117] "RemoveContainer" containerID="27ae9f62791ade57413e213f0844dadc68060f19eb9b4c500077761363efd201"
Sep 07 13:30:03.157469 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:30:03.157448    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ae9f62791ade57413e213f0844dadc68060f19eb9b4c500077761363efd201\": container with ID starting with 27ae9f62791ade57413e213f0844dadc68060f19eb9b4c500077761363efd201 not found: ID does not exist" containerID="27ae9f62791ade57413e213f0844dadc68060f19eb9b4c500077761363efd201"
Sep 07 13:30:03.157546 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:03.157481    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ae9f62791ade57413e213f0844dadc68060f19eb9b4c500077761363efd201"} err="failed to get container status \"27ae9f62791ade57413e213f0844dadc68060f19eb9b4c500077761363efd201\": rpc error: code = NotFound desc = could not find container \"27ae9f62791ade57413e213f0844dadc68060f19eb9b4c500077761363efd201\": container with ID starting with 27ae9f62791ade57413e213f0844dadc68060f19eb9b4c500077761363efd201 not found: ID does not exist"
Sep 07 13:30:03.169363 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:03.169334    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66fb99b89f-gtwxv"]
Sep 07 13:30:03.174945 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:03.174922    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66fb99b89f-gtwxv"]
Sep 07 13:30:04.077451 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:30:04.077412    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4ab6f84-8ff8-4270-8b8b-868329a20cba" path="/var/lib/kubelet/pods/d4ab6f84-8ff8-4270-8b8b-868329a20cba/volumes"
Sep 07 13:31:03.985891 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.985849    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66fb99b89f-h7vxz"]
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986209    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ab6f84-8ff8-4270-8b8b-868329a20cba" containerName="manager"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986224    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ab6f84-8ff8-4270-8b8b-868329a20cba" containerName="manager"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986235    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerName="istio-proxy"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986241    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerName="istio-proxy"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986251    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986258    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986272    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6554a9a0-9500-439d-a3bc-52e5d45a9ba8" containerName="payload-pre-processing"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986277    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6554a9a0-9500-439d-a3bc-52e5d45a9ba8" containerName="payload-pre-processing"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986286    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ad2f71-e0bc-468a-80e6-e127e6c9296d" containerName="payload-processing"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986291    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ad2f71-e0bc-468a-80e6-e127e6c9296d" containerName="payload-processing"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986299    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f02309-4419-4035-999e-4eff749cc77b" containerName="maas-api"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986305    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f02309-4419-4035-999e-4eff749cc77b" containerName="maas-api"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986317    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e7fa90-f906-4ee5-af46-2b942481bd11" containerName="maas-api"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986322    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e7fa90-f906-4ee5-af46-2b942481bd11" containerName="maas-api"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986330    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c14f9c48-5510-4850-88c2-a47612dcb838" containerName="payload-processing"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986335    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c14f9c48-5510-4850-88c2-a47612dcb838" containerName="payload-processing"
Sep 07 13:31:03.986376 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986385    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerName="istio-proxy"
Sep 07 13:31:03.986925 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986392    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6554a9a0-9500-439d-a3bc-52e5d45a9ba8" containerName="payload-pre-processing"
Sep 07 13:31:03.986925 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986399    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ab6f84-8ff8-4270-8b8b-868329a20cba" containerName="manager"
Sep 07 13:31:03.986925 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986405    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e7fa90-f906-4ee5-af46-2b942481bd11" containerName="maas-api"
Sep 07 13:31:03.986925 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986411    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy"
Sep 07 13:31:03.986925 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986416    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c14f9c48-5510-4850-88c2-a47612dcb838" containerName="payload-processing"
Sep 07 13:31:03.986925 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986421    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy"
Sep 07 13:31:03.986925 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986428    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f02309-4419-4035-999e-4eff749cc77b" containerName="maas-api"
Sep 07 13:31:03.986925 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.986432    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ad2f71-e0bc-468a-80e6-e127e6c9296d" containerName="payload-processing"
Sep 07 13:31:03.989524 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.989506    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:03.992311 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.992274    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 07 13:31:03.992501 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.992364    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-97pz6\""
Sep 07 13:31:03.992501 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.992371    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 07 13:31:03.992715 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:03.992698    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 07 13:31:04.001021 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:04.001000    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66fb99b89f-h7vxz"]
Sep 07 13:31:04.090618 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:04.090554    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12448933-ce99-4503-87cb-da8f6214cd62-metrics-certs\") pod \"maas-controller-66fb99b89f-h7vxz\" (UID: \"12448933-ce99-4503-87cb-da8f6214cd62\") " pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:04.090618 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:04.090599    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12448933-ce99-4503-87cb-da8f6214cd62-webhook-cert\") pod \"maas-controller-66fb99b89f-h7vxz\" (UID: \"12448933-ce99-4503-87cb-da8f6214cd62\") " pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:04.090968 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:04.090746    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2ntx\" (UniqueName: \"kubernetes.io/projected/12448933-ce99-4503-87cb-da8f6214cd62-kube-api-access-h2ntx\") pod \"maas-controller-66fb99b89f-h7vxz\" (UID: \"12448933-ce99-4503-87cb-da8f6214cd62\") " pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:04.192137 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:04.192096    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12448933-ce99-4503-87cb-da8f6214cd62-metrics-certs\") pod \"maas-controller-66fb99b89f-h7vxz\" (UID: \"12448933-ce99-4503-87cb-da8f6214cd62\") " pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:04.192357 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:04.192218    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12448933-ce99-4503-87cb-da8f6214cd62-webhook-cert\") pod \"maas-controller-66fb99b89f-h7vxz\" (UID: \"12448933-ce99-4503-87cb-da8f6214cd62\") " pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:04.192357 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:04.192291    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2ntx\" (UniqueName: \"kubernetes.io/projected/12448933-ce99-4503-87cb-da8f6214cd62-kube-api-access-h2ntx\") pod \"maas-controller-66fb99b89f-h7vxz\" (UID: \"12448933-ce99-4503-87cb-da8f6214cd62\") " pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:04.194754 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:04.194730    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12448933-ce99-4503-87cb-da8f6214cd62-metrics-certs\") pod \"maas-controller-66fb99b89f-h7vxz\" (UID: \"12448933-ce99-4503-87cb-da8f6214cd62\") " pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:04.194869 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:04.194763    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12448933-ce99-4503-87cb-da8f6214cd62-webhook-cert\") pod \"maas-controller-66fb99b89f-h7vxz\" (UID: \"12448933-ce99-4503-87cb-da8f6214cd62\") " pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:04.202129 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:04.202107    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2ntx\" (UniqueName: \"kubernetes.io/projected/12448933-ce99-4503-87cb-da8f6214cd62-kube-api-access-h2ntx\") pod \"maas-controller-66fb99b89f-h7vxz\" (UID: \"12448933-ce99-4503-87cb-da8f6214cd62\") " pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:04.300280 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:04.300241    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:04.575421 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:04.575394    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66fb99b89f-h7vxz"]
Sep 07 13:31:04.576828 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:31:04.576802    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12448933_ce99_4503_87cb_da8f6214cd62.slice/crio-3c270751ac250bdcf0787d12e28b13ef01451eadb31bf3882d6f3f6368bee517 WatchSource:0}: Error finding container 3c270751ac250bdcf0787d12e28b13ef01451eadb31bf3882d6f3f6368bee517: Status 404 returned error can't find the container with id 3c270751ac250bdcf0787d12e28b13ef01451eadb31bf3882d6f3f6368bee517
Sep 07 13:31:04.578252 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:04.578233    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 13:31:05.500600 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:05.500491    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fb99b89f-h7vxz" event={"ID":"12448933-ce99-4503-87cb-da8f6214cd62","Type":"ContainerStarted","Data":"1957b7c7cc355d160587a751667b432a5fb68d804014a46872cab9cd8d160e7f"}
Sep 07 13:31:05.500600 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:05.500589    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fb99b89f-h7vxz" event={"ID":"12448933-ce99-4503-87cb-da8f6214cd62","Type":"ContainerStarted","Data":"3c270751ac250bdcf0787d12e28b13ef01451eadb31bf3882d6f3f6368bee517"}
Sep 07 13:31:05.501086 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:05.500618    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:05.522567 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:05.522511    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66fb99b89f-h7vxz" podStartSLOduration=2.012767931 podStartE2EDuration="2.522496463s" totalImagesPullingTime="509.728532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:31:05.519675034 +0000 UTC m=+1909.947973306" watchObservedRunningTime="2026-09-07 13:31:05.522496463 +0000 UTC m=+1909.950794737"
Sep 07 13:31:16.413530 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:16.413496    2567 scope.go:117] "RemoveContainer" containerID="304dd1f8fe9f41ae2d9f0a276cc2fa90b3e691300dc1ae3f384304287e9118b0"
Sep 07 13:31:16.508138 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:16.508102    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:25.992590 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:25.992546    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66fb99b89f-h7vxz"]
Sep 07 13:31:25.994935 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:25.992793    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-66fb99b89f-h7vxz" podUID="12448933-ce99-4503-87cb-da8f6214cd62" containerName="manager" containerID="cri-o://1957b7c7cc355d160587a751667b432a5fb68d804014a46872cab9cd8d160e7f" gracePeriod=30
Sep 07 13:31:26.244110 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.244034    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:26.399858 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.399816    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12448933-ce99-4503-87cb-da8f6214cd62-webhook-cert\") pod \"12448933-ce99-4503-87cb-da8f6214cd62\" (UID: \"12448933-ce99-4503-87cb-da8f6214cd62\") "
Sep 07 13:31:26.400089 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.399882    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2ntx\" (UniqueName: \"kubernetes.io/projected/12448933-ce99-4503-87cb-da8f6214cd62-kube-api-access-h2ntx\") pod \"12448933-ce99-4503-87cb-da8f6214cd62\" (UID: \"12448933-ce99-4503-87cb-da8f6214cd62\") "
Sep 07 13:31:26.400089 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.399904    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12448933-ce99-4503-87cb-da8f6214cd62-metrics-certs\") pod \"12448933-ce99-4503-87cb-da8f6214cd62\" (UID: \"12448933-ce99-4503-87cb-da8f6214cd62\") "
Sep 07 13:31:26.402656 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.402621    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12448933-ce99-4503-87cb-da8f6214cd62-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "12448933-ce99-4503-87cb-da8f6214cd62" (UID: "12448933-ce99-4503-87cb-da8f6214cd62"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:31:26.402787 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.402744    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12448933-ce99-4503-87cb-da8f6214cd62-kube-api-access-h2ntx" (OuterVolumeSpecName: "kube-api-access-h2ntx") pod "12448933-ce99-4503-87cb-da8f6214cd62" (UID: "12448933-ce99-4503-87cb-da8f6214cd62"). InnerVolumeSpecName "kube-api-access-h2ntx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:31:26.402787 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.402745    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12448933-ce99-4503-87cb-da8f6214cd62-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "12448933-ce99-4503-87cb-da8f6214cd62" (UID: "12448933-ce99-4503-87cb-da8f6214cd62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:31:26.501427 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.501316    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2ntx\" (UniqueName: \"kubernetes.io/projected/12448933-ce99-4503-87cb-da8f6214cd62-kube-api-access-h2ntx\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:31:26.501427 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.501363    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12448933-ce99-4503-87cb-da8f6214cd62-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:31:26.501427 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.501380    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12448933-ce99-4503-87cb-da8f6214cd62-webhook-cert\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:31:26.619714 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.619667    2567 generic.go:356] "Generic (PLEG): container finished" podID="12448933-ce99-4503-87cb-da8f6214cd62" containerID="1957b7c7cc355d160587a751667b432a5fb68d804014a46872cab9cd8d160e7f" exitCode=0
Sep 07 13:31:26.619923 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.619728    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fb99b89f-h7vxz"
Sep 07 13:31:26.619923 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.619755    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fb99b89f-h7vxz" event={"ID":"12448933-ce99-4503-87cb-da8f6214cd62","Type":"ContainerDied","Data":"1957b7c7cc355d160587a751667b432a5fb68d804014a46872cab9cd8d160e7f"}
Sep 07 13:31:26.619923 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.619821    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fb99b89f-h7vxz" event={"ID":"12448933-ce99-4503-87cb-da8f6214cd62","Type":"ContainerDied","Data":"3c270751ac250bdcf0787d12e28b13ef01451eadb31bf3882d6f3f6368bee517"}
Sep 07 13:31:26.619923 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.619851    2567 scope.go:117] "RemoveContainer" containerID="1957b7c7cc355d160587a751667b432a5fb68d804014a46872cab9cd8d160e7f"
Sep 07 13:31:26.630155 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.630129    2567 scope.go:117] "RemoveContainer" containerID="1957b7c7cc355d160587a751667b432a5fb68d804014a46872cab9cd8d160e7f"
Sep 07 13:31:26.630578 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:31:26.630500    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1957b7c7cc355d160587a751667b432a5fb68d804014a46872cab9cd8d160e7f\": container with ID starting with 1957b7c7cc355d160587a751667b432a5fb68d804014a46872cab9cd8d160e7f not found: ID does not exist" containerID="1957b7c7cc355d160587a751667b432a5fb68d804014a46872cab9cd8d160e7f"
Sep 07 13:31:26.630578 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.630540    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1957b7c7cc355d160587a751667b432a5fb68d804014a46872cab9cd8d160e7f"} err="failed to get container status \"1957b7c7cc355d160587a751667b432a5fb68d804014a46872cab9cd8d160e7f\": rpc error: code = NotFound desc = could not find container \"1957b7c7cc355d160587a751667b432a5fb68d804014a46872cab9cd8d160e7f\": container with ID starting with 1957b7c7cc355d160587a751667b432a5fb68d804014a46872cab9cd8d160e7f not found: ID does not exist"
Sep 07 13:31:26.653962 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.653934    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66fb99b89f-h7vxz"]
Sep 07 13:31:26.659565 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:26.659542    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66fb99b89f-h7vxz"]
Sep 07 13:31:27.491337 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.491301    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66fb99b89f-lx24m"]
Sep 07 13:31:27.491753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.491657    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerName="istio-proxy"
Sep 07 13:31:27.491753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.491669    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerName="istio-proxy"
Sep 07 13:31:27.491753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.491678    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12448933-ce99-4503-87cb-da8f6214cd62" containerName="manager"
Sep 07 13:31:27.491753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.491684    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="12448933-ce99-4503-87cb-da8f6214cd62" containerName="manager"
Sep 07 13:31:27.491753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.491695    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy"
Sep 07 13:31:27.491753 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.491701    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="863eb763-4af8-41b7-b579-2652c3351065" containerName="istio-proxy"
Sep 07 13:31:27.491944 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.491763    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ef1aaf-726b-44ab-b453-1a861b06dbe7" containerName="istio-proxy"
Sep 07 13:31:27.491944 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.491771    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="12448933-ce99-4503-87cb-da8f6214cd62" containerName="manager"
Sep 07 13:31:27.495946 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.495928    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fb99b89f-lx24m"
Sep 07 13:31:27.499025 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.499004    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-97pz6\""
Sep 07 13:31:27.499539 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.499516    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 07 13:31:27.499763 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.499534    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 07 13:31:27.499856 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.499568    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 07 13:31:27.515229 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.515204    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66fb99b89f-lx24m"]
Sep 07 13:31:27.610413 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.610364    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47463315-4cc1-477b-bb2c-cab8b7e4f8b3-webhook-cert\") pod \"maas-controller-66fb99b89f-lx24m\" (UID: \"47463315-4cc1-477b-bb2c-cab8b7e4f8b3\") " pod="opendatahub/maas-controller-66fb99b89f-lx24m"
Sep 07 13:31:27.610617 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.610451    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47463315-4cc1-477b-bb2c-cab8b7e4f8b3-metrics-certs\") pod \"maas-controller-66fb99b89f-lx24m\" (UID: \"47463315-4cc1-477b-bb2c-cab8b7e4f8b3\") " pod="opendatahub/maas-controller-66fb99b89f-lx24m"
Sep 07 13:31:27.610617 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.610509    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ww96\" (UniqueName: \"kubernetes.io/projected/47463315-4cc1-477b-bb2c-cab8b7e4f8b3-kube-api-access-6ww96\") pod \"maas-controller-66fb99b89f-lx24m\" (UID: \"47463315-4cc1-477b-bb2c-cab8b7e4f8b3\") " pod="opendatahub/maas-controller-66fb99b89f-lx24m"
Sep 07 13:31:27.711548 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.711505    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47463315-4cc1-477b-bb2c-cab8b7e4f8b3-metrics-certs\") pod \"maas-controller-66fb99b89f-lx24m\" (UID: \"47463315-4cc1-477b-bb2c-cab8b7e4f8b3\") " pod="opendatahub/maas-controller-66fb99b89f-lx24m"
Sep 07 13:31:27.711740 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.711583    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ww96\" (UniqueName: \"kubernetes.io/projected/47463315-4cc1-477b-bb2c-cab8b7e4f8b3-kube-api-access-6ww96\") pod \"maas-controller-66fb99b89f-lx24m\" (UID: \"47463315-4cc1-477b-bb2c-cab8b7e4f8b3\") " pod="opendatahub/maas-controller-66fb99b89f-lx24m"
Sep 07 13:31:27.711740 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.711620    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47463315-4cc1-477b-bb2c-cab8b7e4f8b3-webhook-cert\") pod \"maas-controller-66fb99b89f-lx24m\" (UID: \"47463315-4cc1-477b-bb2c-cab8b7e4f8b3\") " pod="opendatahub/maas-controller-66fb99b89f-lx24m"
Sep 07 13:31:27.714106 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.714048    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47463315-4cc1-477b-bb2c-cab8b7e4f8b3-metrics-certs\") pod \"maas-controller-66fb99b89f-lx24m\" (UID: \"47463315-4cc1-477b-bb2c-cab8b7e4f8b3\") " pod="opendatahub/maas-controller-66fb99b89f-lx24m"
Sep 07 13:31:27.714222 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.714110    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47463315-4cc1-477b-bb2c-cab8b7e4f8b3-webhook-cert\") pod \"maas-controller-66fb99b89f-lx24m\" (UID: \"47463315-4cc1-477b-bb2c-cab8b7e4f8b3\") " pod="opendatahub/maas-controller-66fb99b89f-lx24m"
Sep 07 13:31:27.720843 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.720813    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ww96\" (UniqueName: \"kubernetes.io/projected/47463315-4cc1-477b-bb2c-cab8b7e4f8b3-kube-api-access-6ww96\") pod \"maas-controller-66fb99b89f-lx24m\" (UID: \"47463315-4cc1-477b-bb2c-cab8b7e4f8b3\") " pod="opendatahub/maas-controller-66fb99b89f-lx24m"
Sep 07 13:31:27.807868 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:27.807826    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fb99b89f-lx24m"
Sep 07 13:31:28.076037 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:28.075954    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12448933-ce99-4503-87cb-da8f6214cd62" path="/var/lib/kubelet/pods/12448933-ce99-4503-87cb-da8f6214cd62/volumes"
Sep 07 13:31:28.085426 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:28.085400    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66fb99b89f-lx24m"]
Sep 07 13:31:28.088183 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:31:28.088146    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47463315_4cc1_477b_bb2c_cab8b7e4f8b3.slice/crio-1cb675a86ed812e5cf9f4568cf2446e4d37830b4b2a0e10dc68c60d521a0e40c WatchSource:0}: Error finding container 1cb675a86ed812e5cf9f4568cf2446e4d37830b4b2a0e10dc68c60d521a0e40c: Status 404 returned error can't find the container with id 1cb675a86ed812e5cf9f4568cf2446e4d37830b4b2a0e10dc68c60d521a0e40c
Sep 07 13:31:28.631614 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:28.631581    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fb99b89f-lx24m" event={"ID":"47463315-4cc1-477b-bb2c-cab8b7e4f8b3","Type":"ContainerStarted","Data":"1cb675a86ed812e5cf9f4568cf2446e4d37830b4b2a0e10dc68c60d521a0e40c"}
Sep 07 13:31:29.640516 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:29.640474    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fb99b89f-lx24m" event={"ID":"47463315-4cc1-477b-bb2c-cab8b7e4f8b3","Type":"ContainerStarted","Data":"8ca72580b5216e55a38fabb10ac0a982155882a204497c798a1b305da92acf97"}
Sep 07 13:31:29.641103 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:29.640664    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66fb99b89f-lx24m"
Sep 07 13:31:29.666098 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:29.666028    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66fb99b89f-lx24m" podStartSLOduration=1.951404967 podStartE2EDuration="2.666012793s" totalImagesPullingTime="714.607826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:31:29.664364352 +0000 UTC m=+1934.092662619" watchObservedRunningTime="2026-09-07 13:31:29.666012793 +0000 UTC m=+1934.094311067"
Sep 07 13:31:40.647579 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:31:40.647543    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66fb99b89f-lx24m"
Sep 07 13:32:42.166393 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:42.166353    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"]
Sep 07 13:32:42.166846 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:42.166560    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d" podUID="6d2c3c54-003d-4d49-8044-f27430dbdcad" containerName="manager" containerID="cri-o://0962d8897b8927d287040042fb11c1c713caa5668180b5cacbca70768bb38875" gracePeriod=10
Sep 07 13:32:42.413149 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:42.413126    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"
Sep 07 13:32:42.475462 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:42.475373    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d2c3c54-003d-4d49-8044-f27430dbdcad-extensions-socket-volume\") pod \"6d2c3c54-003d-4d49-8044-f27430dbdcad\" (UID: \"6d2c3c54-003d-4d49-8044-f27430dbdcad\") "
Sep 07 13:32:42.475462 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:42.475434    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj9jb\" (UniqueName: \"kubernetes.io/projected/6d2c3c54-003d-4d49-8044-f27430dbdcad-kube-api-access-vj9jb\") pod \"6d2c3c54-003d-4d49-8044-f27430dbdcad\" (UID: \"6d2c3c54-003d-4d49-8044-f27430dbdcad\") "
Sep 07 13:32:42.475733 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:42.475702    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2c3c54-003d-4d49-8044-f27430dbdcad-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6d2c3c54-003d-4d49-8044-f27430dbdcad" (UID: "6d2c3c54-003d-4d49-8044-f27430dbdcad"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:32:42.477592 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:42.477549    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2c3c54-003d-4d49-8044-f27430dbdcad-kube-api-access-vj9jb" (OuterVolumeSpecName: "kube-api-access-vj9jb") pod "6d2c3c54-003d-4d49-8044-f27430dbdcad" (UID: "6d2c3c54-003d-4d49-8044-f27430dbdcad"). InnerVolumeSpecName "kube-api-access-vj9jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:32:42.576154 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:42.576123    2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d2c3c54-003d-4d49-8044-f27430dbdcad-extensions-socket-volume\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:32:42.576154 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:42.576149    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj9jb\" (UniqueName: \"kubernetes.io/projected/6d2c3c54-003d-4d49-8044-f27430dbdcad-kube-api-access-vj9jb\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:32:43.036754 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:43.036718    2567 generic.go:356] "Generic (PLEG): container finished" podID="6d2c3c54-003d-4d49-8044-f27430dbdcad" containerID="0962d8897b8927d287040042fb11c1c713caa5668180b5cacbca70768bb38875" exitCode=0
Sep 07 13:32:43.036930 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:43.036763    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"
Sep 07 13:32:43.036930 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:43.036803    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d" event={"ID":"6d2c3c54-003d-4d49-8044-f27430dbdcad","Type":"ContainerDied","Data":"0962d8897b8927d287040042fb11c1c713caa5668180b5cacbca70768bb38875"}
Sep 07 13:32:43.036930 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:43.036856    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d" event={"ID":"6d2c3c54-003d-4d49-8044-f27430dbdcad","Type":"ContainerDied","Data":"e57f913a409dc3e0daf2de82c0e7368d82d2f6fe3ac24faad3f00874a1761619"}
Sep 07 13:32:43.036930 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:43.036879    2567 scope.go:117] "RemoveContainer" containerID="0962d8897b8927d287040042fb11c1c713caa5668180b5cacbca70768bb38875"
Sep 07 13:32:43.046841 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:43.046825    2567 scope.go:117] "RemoveContainer" containerID="0962d8897b8927d287040042fb11c1c713caa5668180b5cacbca70768bb38875"
Sep 07 13:32:43.047139 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:32:43.047122    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0962d8897b8927d287040042fb11c1c713caa5668180b5cacbca70768bb38875\": container with ID starting with 0962d8897b8927d287040042fb11c1c713caa5668180b5cacbca70768bb38875 not found: ID does not exist" containerID="0962d8897b8927d287040042fb11c1c713caa5668180b5cacbca70768bb38875"
Sep 07 13:32:43.047194 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:43.047150    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0962d8897b8927d287040042fb11c1c713caa5668180b5cacbca70768bb38875"} err="failed to get container status \"0962d8897b8927d287040042fb11c1c713caa5668180b5cacbca70768bb38875\": rpc error: code = NotFound desc = could not find container \"0962d8897b8927d287040042fb11c1c713caa5668180b5cacbca70768bb38875\": container with ID starting with 0962d8897b8927d287040042fb11c1c713caa5668180b5cacbca70768bb38875 not found: ID does not exist"
Sep 07 13:32:43.064859 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:43.064834    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"]
Sep 07 13:32:43.067772 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:43.067750    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-pkx4d"]
Sep 07 13:32:44.074493 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:32:44.074455    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d2c3c54-003d-4d49-8044-f27430dbdcad" path="/var/lib/kubelet/pods/6d2c3c54-003d-4d49-8044-f27430dbdcad/volumes"
Sep 07 13:33:42.881864 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:33:42.881784    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj"]
Sep 07 13:33:42.882355 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:33:42.882175    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2c3c54-003d-4d49-8044-f27430dbdcad" containerName="manager"
Sep 07 13:33:42.882355 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:33:42.882188    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2c3c54-003d-4d49-8044-f27430dbdcad" containerName="manager"
Sep 07 13:33:42.882355 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:33:42.882242    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d2c3c54-003d-4d49-8044-f27430dbdcad" containerName="manager"
Sep 07 13:33:42.885264 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:33:42.885248    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj"
Sep 07 13:33:42.887369 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:33:42.887350    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tgv8t\""
Sep 07 13:33:43.008445 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:33:43.008404    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e32de8c3-fdf6-41d5-bc11-1d1dc8461047-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-rr6rj\" (UID: \"e32de8c3-fdf6-41d5-bc11-1d1dc8461047\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj"
Sep 07 13:33:43.008445 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:33:43.008453    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7g2f\" (UniqueName: \"kubernetes.io/projected/e32de8c3-fdf6-41d5-bc11-1d1dc8461047-kube-api-access-m7g2f\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-rr6rj\" (UID: \"e32de8c3-fdf6-41d5-bc11-1d1dc8461047\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj"
Sep 07 13:33:43.109420 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:33:43.109379    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e32de8c3-fdf6-41d5-bc11-1d1dc8461047-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-rr6rj\" (UID: \"e32de8c3-fdf6-41d5-bc11-1d1dc8461047\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj"
Sep 07 13:33:43.109607 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:33:43.109428    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7g2f\" (UniqueName: \"kubernetes.io/projected/e32de8c3-fdf6-41d5-bc11-1d1dc8461047-kube-api-access-m7g2f\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-rr6rj\" (UID: \"e32de8c3-fdf6-41d5-bc11-1d1dc8461047\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj"
Sep 07 13:33:43.109766 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:33:43.109746    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e32de8c3-fdf6-41d5-bc11-1d1dc8461047-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-rr6rj\" (UID: \"e32de8c3-fdf6-41d5-bc11-1d1dc8461047\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj"
Sep 07 13:33:43.120007 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:33:43.119981    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7g2f\" (UniqueName: \"kubernetes.io/projected/e32de8c3-fdf6-41d5-bc11-1d1dc8461047-kube-api-access-m7g2f\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-rr6rj\" (UID: \"e32de8c3-fdf6-41d5-bc11-1d1dc8461047\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj"
Sep 07 13:33:43.195179 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:33:43.195087    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj"
Sep 07 13:34:13.322497 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:34:13.322464    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj"]
Sep 07 13:34:13.591714 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:34:13.591681    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj"]
Sep 07 13:34:13.592036 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:34:13.592017    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32de8c3_fdf6_41d5_bc11_1d1dc8461047.slice/crio-4d5dbe4de6fa75528459b8294131b3bc41a86131502f43bfacb66727a289e319 WatchSource:0}: Error finding container 4d5dbe4de6fa75528459b8294131b3bc41a86131502f43bfacb66727a289e319: Status 404 returned error can't find the container with id 4d5dbe4de6fa75528459b8294131b3bc41a86131502f43bfacb66727a289e319
Sep 07 13:34:14.511243 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:34:14.511203    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj" event={"ID":"e32de8c3-fdf6-41d5-bc11-1d1dc8461047","Type":"ContainerStarted","Data":"c510765308b20cd99bfbb7b676f350d0be5299a8d74ec037483dcf3d76cfb6ef"}
Sep 07 13:34:14.511717 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:34:14.511258    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj" event={"ID":"e32de8c3-fdf6-41d5-bc11-1d1dc8461047","Type":"ContainerStarted","Data":"4d5dbe4de6fa75528459b8294131b3bc41a86131502f43bfacb66727a289e319"}
Sep 07 13:34:14.511717 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:34:14.511337    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj"
Sep 07 13:34:14.548490 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:34:14.548426    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj" podStartSLOduration=32.54841035 podStartE2EDuration="32.54841035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:34:14.545725701 +0000 UTC m=+2098.974023973" watchObservedRunningTime="2026-09-07 13:34:14.54841035 +0000 UTC m=+2098.976708623"
Sep 07 13:34:25.521131 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:34:25.521092    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-rr6rj"
Sep 07 13:35:49.788518 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.788486    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"]
Sep 07 13:35:49.791898 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.791877    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:49.810451 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.810427    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"]
Sep 07 13:35:49.891495 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.891470    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5cf1503-4e25-461c-9835-55f5c93f296b-maas-api-tls\") pod \"maas-api-758db4577b-qfv8t\" (UID: \"d5cf1503-4e25-461c-9835-55f5c93f296b\") " pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:49.891624 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.891506    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92hlh\" (UniqueName: \"kubernetes.io/projected/d5cf1503-4e25-461c-9835-55f5c93f296b-kube-api-access-92hlh\") pod \"maas-api-758db4577b-qfv8t\" (UID: \"d5cf1503-4e25-461c-9835-55f5c93f296b\") " pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:49.891624 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.891538    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5cf1503-4e25-461c-9835-55f5c93f296b-metrics-certs\") pod \"maas-api-758db4577b-qfv8t\" (UID: \"d5cf1503-4e25-461c-9835-55f5c93f296b\") " pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:49.891727 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.891646    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5cf1503-4e25-461c-9835-55f5c93f296b-openshift-service-ca\") pod \"maas-api-758db4577b-qfv8t\" (UID: \"d5cf1503-4e25-461c-9835-55f5c93f296b\") " pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:49.992130 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.992101    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92hlh\" (UniqueName: \"kubernetes.io/projected/d5cf1503-4e25-461c-9835-55f5c93f296b-kube-api-access-92hlh\") pod \"maas-api-758db4577b-qfv8t\" (UID: \"d5cf1503-4e25-461c-9835-55f5c93f296b\") " pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:49.992250 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.992149    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5cf1503-4e25-461c-9835-55f5c93f296b-metrics-certs\") pod \"maas-api-758db4577b-qfv8t\" (UID: \"d5cf1503-4e25-461c-9835-55f5c93f296b\") " pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:49.992250 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.992205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5cf1503-4e25-461c-9835-55f5c93f296b-openshift-service-ca\") pod \"maas-api-758db4577b-qfv8t\" (UID: \"d5cf1503-4e25-461c-9835-55f5c93f296b\") " pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:49.992250 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.992224    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5cf1503-4e25-461c-9835-55f5c93f296b-maas-api-tls\") pod \"maas-api-758db4577b-qfv8t\" (UID: \"d5cf1503-4e25-461c-9835-55f5c93f296b\") " pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:49.992832 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.992806    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5cf1503-4e25-461c-9835-55f5c93f296b-openshift-service-ca\") pod \"maas-api-758db4577b-qfv8t\" (UID: \"d5cf1503-4e25-461c-9835-55f5c93f296b\") " pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:49.994516 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.994487    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5cf1503-4e25-461c-9835-55f5c93f296b-metrics-certs\") pod \"maas-api-758db4577b-qfv8t\" (UID: \"d5cf1503-4e25-461c-9835-55f5c93f296b\") " pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:49.994516 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:49.994503    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5cf1503-4e25-461c-9835-55f5c93f296b-maas-api-tls\") pod \"maas-api-758db4577b-qfv8t\" (UID: \"d5cf1503-4e25-461c-9835-55f5c93f296b\") " pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:50.002534 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:50.002513    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92hlh\" (UniqueName: \"kubernetes.io/projected/d5cf1503-4e25-461c-9835-55f5c93f296b-kube-api-access-92hlh\") pod \"maas-api-758db4577b-qfv8t\" (UID: \"d5cf1503-4e25-461c-9835-55f5c93f296b\") " pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:50.102630 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:50.102561    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:50.384422 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:50.384338    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"]
Sep 07 13:35:50.390273 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:35:50.390195    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5cf1503_4e25_461c_9835_55f5c93f296b.slice/crio-e8ddb4c11ef6c5764d4f5de300d4033efb20875b2a47274e2b6a6453fe18713b WatchSource:0}: Error finding container e8ddb4c11ef6c5764d4f5de300d4033efb20875b2a47274e2b6a6453fe18713b: Status 404 returned error can't find the container with id e8ddb4c11ef6c5764d4f5de300d4033efb20875b2a47274e2b6a6453fe18713b
Sep 07 13:35:51.016875 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:51.016850    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t" event={"ID":"d5cf1503-4e25-461c-9835-55f5c93f296b","Type":"ContainerStarted","Data":"e8ddb4c11ef6c5764d4f5de300d4033efb20875b2a47274e2b6a6453fe18713b"}
Sep 07 13:35:52.024339 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:52.024304    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t" event={"ID":"d5cf1503-4e25-461c-9835-55f5c93f296b","Type":"ContainerStarted","Data":"78254370aa6a35d86342bbdda367d1258585d48fe85df2c3ac2088587817ad15"}
Sep 07 13:35:52.024705 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:52.024416    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:52.046076 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:52.046009    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t" podStartSLOduration=2.486044654 podStartE2EDuration="3.045992895s" totalImagesPullingTime="559.948241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:35:52.044641026 +0000 UTC m=+2196.472939322" watchObservedRunningTime="2026-09-07 13:35:52.045992895 +0000 UTC m=+2196.474291180"
Sep 07 13:35:58.035221 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:58.035190    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-758db4577b-qfv8t"
Sep 07 13:35:58.107511 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:58.107477    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"]
Sep 07 13:35:58.107707 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:58.107662    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr" podUID="3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98" containerName="maas-api" containerID="cri-o://65c7053dae7abd4d98db91e3794eff3a52074590e5bcfe374d45bd98c747c761" gracePeriod=30
Sep 07 13:35:59.371012 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:59.370988    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:35:59.567980 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:59.567955    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-maas-api-tls\") pod \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") "
Sep 07 13:35:59.568157 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:59.568012    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn4xg\" (UniqueName: \"kubernetes.io/projected/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-kube-api-access-tn4xg\") pod \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") "
Sep 07 13:35:59.568157 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:59.568046    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-metrics-certs\") pod \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") "
Sep 07 13:35:59.568258 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:59.568182    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-openshift-service-ca\") pod \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\" (UID: \"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98\") "
Sep 07 13:35:59.568509 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:59.568479    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98" (UID: "3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:35:59.570155 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:59.570116    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-kube-api-access-tn4xg" (OuterVolumeSpecName: "kube-api-access-tn4xg") pod "3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98" (UID: "3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98"). InnerVolumeSpecName "kube-api-access-tn4xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:35:59.570579 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:59.570549    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98" (UID: "3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:35:59.570657 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:59.570551    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98" (UID: "3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:35:59.668995 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:59.668968    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-metrics-certs\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:35:59.669103 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:59.669003    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-openshift-service-ca\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:35:59.669103 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:59.669019    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-maas-api-tls\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:35:59.669103 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:35:59.669033    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4xg\" (UniqueName: \"kubernetes.io/projected/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98-kube-api-access-tn4xg\") on node \"ip-10-0-128-114.ec2.internal\" DevicePath \"\""
Sep 07 13:36:00.070756 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:00.070725    2567 generic.go:356] "Generic (PLEG): container finished" podID="3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98" containerID="65c7053dae7abd4d98db91e3794eff3a52074590e5bcfe374d45bd98c747c761" exitCode=0
Sep 07 13:36:00.070913 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:00.070861    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"
Sep 07 13:36:00.075532 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:00.075503    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr" event={"ID":"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98","Type":"ContainerDied","Data":"65c7053dae7abd4d98db91e3794eff3a52074590e5bcfe374d45bd98c747c761"}
Sep 07 13:36:00.075673 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:00.075547    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr" event={"ID":"3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98","Type":"ContainerDied","Data":"3c00534177303a17a9edf33b31afc3edcb68def9d7d0b9b8a4e2c1b8fac7c235"}
Sep 07 13:36:00.075673 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:00.075576    2567 scope.go:117] "RemoveContainer" containerID="65c7053dae7abd4d98db91e3794eff3a52074590e5bcfe374d45bd98c747c761"
Sep 07 13:36:00.085864 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:00.085854    2567 scope.go:117] "RemoveContainer" containerID="65c7053dae7abd4d98db91e3794eff3a52074590e5bcfe374d45bd98c747c761"
Sep 07 13:36:00.086153 ip-10-0-128-114 kubenswrapper[2567]: E0907 13:36:00.086124    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c7053dae7abd4d98db91e3794eff3a52074590e5bcfe374d45bd98c747c761\": container with ID starting with 65c7053dae7abd4d98db91e3794eff3a52074590e5bcfe374d45bd98c747c761 not found: ID does not exist" containerID="65c7053dae7abd4d98db91e3794eff3a52074590e5bcfe374d45bd98c747c761"
Sep 07 13:36:00.086246 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:00.086161    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c7053dae7abd4d98db91e3794eff3a52074590e5bcfe374d45bd98c747c761"} err="failed to get container status \"65c7053dae7abd4d98db91e3794eff3a52074590e5bcfe374d45bd98c747c761\": rpc error: code = NotFound desc = could not find container \"65c7053dae7abd4d98db91e3794eff3a52074590e5bcfe374d45bd98c747c761\": container with ID starting with 65c7053dae7abd4d98db91e3794eff3a52074590e5bcfe374d45bd98c747c761 not found: ID does not exist"
Sep 07 13:36:00.096876 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:00.096854    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"]
Sep 07 13:36:00.102726 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:00.102703    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-849b74d9cb-2s4lr"]
Sep 07 13:36:02.075201 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:02.075164    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98" path="/var/lib/kubelet/pods/3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98/volumes"
Sep 07 13:36:13.302736 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:13.302680    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-fkppx" podUID="fa91b78f-8f6e-41e5-83df-322e9825cec5" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 13:36:13.303181 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:13.302792    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fkppx"
Sep 07 13:36:13.303463 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:13.303433    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-fkppx" podUID="fa91b78f-8f6e-41e5-83df-322e9825cec5" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 13:36:28.755737 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:28.755688    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66fb99b89f-lx24m_47463315-4cc1-477b-bb2c-cab8b7e4f8b3/manager/0.log"
Sep 07 13:36:29.411776 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:29.411743    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5545ff49d6-pkmvq_fc7d25ca-6842-4db6-a4c4-ebc46f8d3023/manager/0.log"
Sep 07 13:36:29.425007 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:29.424982    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5545ff49d6-pkmvq_fc7d25ca-6842-4db6-a4c4-ebc46f8d3023/copy-manifests/0.log"
Sep 07 13:36:29.892088 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:29.892048    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-758db4577b-qfv8t_d5cf1503-4e25-461c-9835-55f5c93f296b/maas-api/0.log"
Sep 07 13:36:30.251733 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:30.251670    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-67b6d_f751fe86-77a6-4fce-ab07-1372f66c6593/postgres/0.log"
Sep 07 13:36:31.282883 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:31.282842    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-6wh7q_81c4e661-8058-4bc7-bcfd-8939b6c4f93f/manager/0.log"
Sep 07 13:36:31.533681 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:31.533592    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-rr6rj_e32de8c3-fdf6-41d5-bc11-1d1dc8461047/manager/0.log"
Sep 07 13:36:32.125263 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:32.125234    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5_d17e721b-4419-4dd2-a78e-339f4b969e8d/istio-proxy/0.log"
Sep 07 13:36:32.250796 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:32.250768    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dhxxb_e19341b0-ac2f-46bb-a710-7e86f8bad345/discovery/0.log"
Sep 07 13:36:32.371735 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:32.371713    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546b89bbb7-99rvx_2ea763de-018b-432b-8322-0345cec5ec7e/kube-auth-proxy/0.log"
Sep 07 13:36:32.633852 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:32.633824    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vglg_03a5b4a5-e033-43a7-bf17-a35a40587f00/istio-proxy/1.log"
Sep 07 13:36:32.899815 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:32.899732    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lkvfr_0f8cf259-4a5d-4e00-9f43-7bee665cbd7e/payload-processing/0.log"
Sep 07 13:36:33.365747 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:33.365721    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-twnlk_25d1f400-6375-4c37-b4fa-629a376674a7/main/0.log"
Sep 07 13:36:33.856250 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:33.856216    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fnwmx4_f14eb704-fc90-4a37-8835-069ea0fe87bf/main/0.log"
Sep 07 13:36:43.304407 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:43.304188    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fkppx"
Sep 07 13:36:43.566375 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:43.566291    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-njk59_d1dbf089-2e69-485b-9e8b-eb30c69e15aa/global-pull-secret-syncer/0.log"
Sep 07 13:36:43.768955 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:43.768917    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fkppx_fa91b78f-8f6e-41e5-83df-322e9825cec5/konnectivity-agent/0.log"
Sep 07 13:36:43.855086 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:43.855008    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-114.ec2.internal_68fb4e41d01603c60ad3ed3f0a6da717/haproxy/0.log"
Sep 07 13:36:48.963990 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:48.963958    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-6wh7q_81c4e661-8058-4bc7-bcfd-8939b6c4f93f/manager/0.log"
Sep 07 13:36:49.077417 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:49.077389    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-rr6rj_e32de8c3-fdf6-41d5-bc11-1d1dc8461047/manager/0.log"
Sep 07 13:36:51.387242 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:51.387217    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lc2x_e473f2b3-d603-4b42-b40d-3616dbcbb50b/node-exporter/0.log"
Sep 07 13:36:51.413365 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:51.413343    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lc2x_e473f2b3-d603-4b42-b40d-3616dbcbb50b/kube-rbac-proxy/0.log"
Sep 07 13:36:51.445519 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:51.445500    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lc2x_e473f2b3-d603-4b42-b40d-3616dbcbb50b/init-textfile/0.log"
Sep 07 13:36:53.766137 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.766093    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"]
Sep 07 13:36:53.766535 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.766479    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98" containerName="maas-api"
Sep 07 13:36:53.766535 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.766494    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98" containerName="maas-api"
Sep 07 13:36:53.766618 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.766564    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b40a9f5-ebe9-4883-8a6e-1cf9c1420d98" containerName="maas-api"
Sep 07 13:36:53.769593 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.769573    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.775328 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.775288    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-scn9z\"/\"kube-root-ca.crt\""
Sep 07 13:36:53.775328 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.775309    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-scn9z\"/\"openshift-service-ca.crt\""
Sep 07 13:36:53.775643 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.775626    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-scn9z\"/\"default-dockercfg-wv848\""
Sep 07 13:36:53.780731 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.780703    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"]
Sep 07 13:36:53.885475 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.885441    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5p2\" (UniqueName: \"kubernetes.io/projected/31fa9f9f-3bc9-4df1-a520-31e668e20600-kube-api-access-xq5p2\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.885618 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.885499    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31fa9f9f-3bc9-4df1-a520-31e668e20600-podres\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.885618 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.885539    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31fa9f9f-3bc9-4df1-a520-31e668e20600-lib-modules\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.885618 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.885573    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31fa9f9f-3bc9-4df1-a520-31e668e20600-proc\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.885785 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.885693    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31fa9f9f-3bc9-4df1-a520-31e668e20600-sys\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.986286 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.986253    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31fa9f9f-3bc9-4df1-a520-31e668e20600-lib-modules\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.986421 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.986292    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31fa9f9f-3bc9-4df1-a520-31e668e20600-proc\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.986421 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.986319    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31fa9f9f-3bc9-4df1-a520-31e668e20600-sys\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.986421 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.986357    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5p2\" (UniqueName: \"kubernetes.io/projected/31fa9f9f-3bc9-4df1-a520-31e668e20600-kube-api-access-xq5p2\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.986421 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.986392    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31fa9f9f-3bc9-4df1-a520-31e668e20600-podres\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.986583 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.986420    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31fa9f9f-3bc9-4df1-a520-31e668e20600-sys\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.986583 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.986388    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31fa9f9f-3bc9-4df1-a520-31e668e20600-proc\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.986583 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.986454    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31fa9f9f-3bc9-4df1-a520-31e668e20600-lib-modules\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.986583 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.986513    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31fa9f9f-3bc9-4df1-a520-31e668e20600-podres\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:53.995634 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:53.995598    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5p2\" (UniqueName: \"kubernetes.io/projected/31fa9f9f-3bc9-4df1-a520-31e668e20600-kube-api-access-xq5p2\") pod \"perf-node-gather-daemonset-xg2ws\" (UID: \"31fa9f9f-3bc9-4df1-a520-31e668e20600\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:54.081025 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:54.081003    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:54.358626 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:54.358600    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"]
Sep 07 13:36:54.360021 ip-10-0-128-114 kubenswrapper[2567]: W0907 13:36:54.359994    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod31fa9f9f_3bc9_4df1_a520_31e668e20600.slice/crio-9272056415fa3dc961355c1e47f0b49bec98475ac63dd67c167bd95b970756c4 WatchSource:0}: Error finding container 9272056415fa3dc961355c1e47f0b49bec98475ac63dd67c167bd95b970756c4: Status 404 returned error can't find the container with id 9272056415fa3dc961355c1e47f0b49bec98475ac63dd67c167bd95b970756c4
Sep 07 13:36:54.366989 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:54.366966    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws" event={"ID":"31fa9f9f-3bc9-4df1-a520-31e668e20600","Type":"ContainerStarted","Data":"9272056415fa3dc961355c1e47f0b49bec98475ac63dd67c167bd95b970756c4"}
Sep 07 13:36:55.374866 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:55.374826    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws" event={"ID":"31fa9f9f-3bc9-4df1-a520-31e668e20600","Type":"ContainerStarted","Data":"0445ec72c693abaf2ae9aa83862b36de48a5df560570b5576d39da0768aac7d2"}
Sep 07 13:36:55.375279 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:55.374981    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:36:55.396597 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:55.396540    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws" podStartSLOduration=2.396524796 podStartE2EDuration="2.396524796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 13:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:36:55.395006799 +0000 UTC m=+2259.823305071" watchObservedRunningTime="2026-09-07 13:36:55.396524796 +0000 UTC m=+2259.824823067"
Sep 07 13:36:56.681458 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:56.681423    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g54bn_6c700ee4-1744-4956-9728-eb2911022015/dns/0.log"
Sep 07 13:36:56.709968 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:56.709933    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g54bn_6c700ee4-1744-4956-9728-eb2911022015/kube-rbac-proxy/0.log"
Sep 07 13:36:56.897109 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:56.897079    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sqnx4_8f1b14f3-6875-4d07-859d-ada2d3bb1f0f/dns-node-resolver/0.log"
Sep 07 13:36:57.582937 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:57.582909    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kdkj7_ea2d8b03-e721-4102-846e-ee7d41d13e96/node-ca/0.log"
Sep 07 13:36:58.630203 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:58.630165    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5_d17e721b-4419-4dd2-a78e-339f4b969e8d/istio-proxy/0.log"
Sep 07 13:36:59.172133 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:59.172105    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dhxxb_e19341b0-ac2f-46bb-a710-7e86f8bad345/discovery/0.log"
Sep 07 13:36:59.230555 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:59.230527    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546b89bbb7-99rvx_2ea763de-018b-432b-8322-0345cec5ec7e/kube-auth-proxy/0.log"
Sep 07 13:36:59.295630 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:59.295603    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vglg_03a5b4a5-e033-43a7-bf17-a35a40587f00/istio-proxy/0.log"
Sep 07 13:36:59.312345 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:59.312323    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vglg_03a5b4a5-e033-43a7-bf17-a35a40587f00/istio-proxy/1.log"
Sep 07 13:36:59.415585 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:36:59.415563    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lkvfr_0f8cf259-4a5d-4e00-9f43-7bee665cbd7e/payload-processing/0.log"
Sep 07 13:37:00.432653 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:00.432624    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xvkgl_4d36321e-8fe0-4c81-a143-d99bf792bd99/serve-healthcheck-canary/0.log"
Sep 07 13:37:00.986918 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:00.986885    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzp8t_2469a9d8-5736-4c2b-a707-8128164910e8/kube-rbac-proxy/0.log"
Sep 07 13:37:01.036194 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:01.036164    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzp8t_2469a9d8-5736-4c2b-a707-8128164910e8/exporter/0.log"
Sep 07 13:37:01.068707 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:01.068676    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzp8t_2469a9d8-5736-4c2b-a707-8128164910e8/extractor/0.log"
Sep 07 13:37:01.392053 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:01.392031    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-xg2ws"
Sep 07 13:37:04.134529 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:04.134499    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5_d17e721b-4419-4dd2-a78e-339f4b969e8d/istio-proxy/0.log"
Sep 07 13:37:04.233292 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:04.233269    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66fb99b89f-lx24m_47463315-4cc1-477b-bb2c-cab8b7e4f8b3/manager/0.log"
Sep 07 13:37:04.588990 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:04.588966    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dhxxb_e19341b0-ac2f-46bb-a710-7e86f8bad345/discovery/0.log"
Sep 07 13:37:04.615814 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:04.615789    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546b89bbb7-99rvx_2ea763de-018b-432b-8322-0345cec5ec7e/kube-auth-proxy/0.log"
Sep 07 13:37:04.664331 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:04.664311    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vglg_03a5b4a5-e033-43a7-bf17-a35a40587f00/istio-proxy/0.log"
Sep 07 13:37:04.680553 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:04.680525    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vglg_03a5b4a5-e033-43a7-bf17-a35a40587f00/istio-proxy/1.log"
Sep 07 13:37:04.790004 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:04.789981    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lkvfr_0f8cf259-4a5d-4e00-9f43-7bee665cbd7e/payload-processing/0.log"
Sep 07 13:37:04.831602 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:04.831580    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5545ff49d6-pkmvq_fc7d25ca-6842-4db6-a4c4-ebc46f8d3023/manager/0.log"
Sep 07 13:37:04.860888 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:04.860834    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5545ff49d6-pkmvq_fc7d25ca-6842-4db6-a4c4-ebc46f8d3023/copy-manifests/0.log"
Sep 07 13:37:06.305589 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:06.305558    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-f4wpb_f006b9b6-5821-48d1-82f4-aa06177b80eb/openshift-lws-operator/0.log"
Sep 07 13:37:06.376125 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:06.376042    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-q87x5_d17e721b-4419-4dd2-a78e-339f4b969e8d/istio-proxy/0.log"
Sep 07 13:37:06.938517 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:06.938490    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dhxxb_e19341b0-ac2f-46bb-a710-7e86f8bad345/discovery/0.log"
Sep 07 13:37:06.987406 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:06.987386    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546b89bbb7-99rvx_2ea763de-018b-432b-8322-0345cec5ec7e/kube-auth-proxy/0.log"
Sep 07 13:37:07.039987 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:07.039962    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vglg_03a5b4a5-e033-43a7-bf17-a35a40587f00/istio-proxy/0.log"
Sep 07 13:37:07.056927 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:07.056909    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6vglg_03a5b4a5-e033-43a7-bf17-a35a40587f00/istio-proxy/1.log"
Sep 07 13:37:07.138836 ip-10-0-128-114 kubenswrapper[2567]: I0907 13:37:07.138811    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lkvfr_0f8cf259-4a5d-4e00-9f43-7bee665cbd7e/payload-processing/0.log"