Sep 09 03:56:12.504276 ip-10-0-142-198 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 03:56:12.504286 ip-10-0-142-198 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 03:56:12.504296 ip-10-0-142-198 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 03:56:12.504484 ip-10-0-142-198 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 03:56:22.673265 ip-10-0-142-198 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 03:56:22.673282 ip-10-0-142-198 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d8979cadc67044728cf09fd2044834ac --
Sep 09 03:57:58.945094 ip-10-0-142-198 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 03:57:59.364599 ip-10-0-142-198 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:57:59.364599 ip-10-0-142-198 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 03:57:59.364599 ip-10-0-142-198 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:57:59.364599 ip-10-0-142-198 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 03:57:59.364599 ip-10-0-142-198 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 03:57:59.368174 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.368087    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 03:57:59.372291 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372276    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:57:59.372291 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372292    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372296    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372299    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372302    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372304    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372307    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372310    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372313    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372315    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372319    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372322    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372325    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372327    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372330    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372332    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372335    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372337    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372340    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372342    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372350    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:57:59.372354 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372353    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372355    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372358    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372361    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372364    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372366    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372369    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372371    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372374    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372376    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372379    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372381    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372384    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372386    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372389    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372391    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372393    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372396    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372398    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372401    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:57:59.372838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372404    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372406    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372409    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372411    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372414    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372416    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372420    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372422    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372424    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372427    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372429    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372431    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372434    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372436    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372440    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372443    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372445    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372448    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372450    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372453    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:57:59.373363 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372455    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372458    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372460    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372463    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372465    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372468    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372472    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372476    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372479    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372481    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372484    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372487    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372490    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372494    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372498    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372501    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372503    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372507    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372510    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:57:59.373875 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372512    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372515    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372517    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372520    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372523    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372526    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372528    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372531    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372534    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372536    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372539    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372541    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372544    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372546    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372550    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372554    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372557    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372559    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.372562    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:57:59.374340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373256    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373262    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373265    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373268    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373270    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373273    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373275    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373278    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373281    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373283    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373285    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373288    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373292    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373294    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373297    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373299    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373302    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373304    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373307    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373309    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:57:59.374824 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373312    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373315    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373318    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373320    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373323    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373326    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373328    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373331    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373334    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373336    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373338    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373341    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373345    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373348    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373350    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373353    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373355    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373358    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373361    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373364    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:57:59.375307 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373366    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373369    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373372    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373374    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373379    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373390    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373394    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373396    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373400    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373402    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373405    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373407    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373411    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373413    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373416    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373418    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373421    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373424    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373426    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373428    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:57:59.375838 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373431    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373433    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373436    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373440    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373443    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373446    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373449    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373453    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373455    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373458    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373460    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373463    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373466    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373468    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373470    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373473    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373475    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373478    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373481    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:57:59.376326 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373483    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373488    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373491    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373493    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373496    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373499    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373501    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373504    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373506    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373509    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373511    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373514    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373516    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373519    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373521    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373524    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373526    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373529    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373531    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:57:59.376889 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373534    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373897    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373902    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373905    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373908    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373911    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373913    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373916    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373918    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373921    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373924    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373926    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373929    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373932    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373937    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373939    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373942    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373944    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373947    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373949    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:57:59.377350 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373952    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373955    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373957    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373960    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373963    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373965    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373968    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373971    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373973    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373976    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373978    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373981    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373983    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373986    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373989    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373991    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373993    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373996    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.373998    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374001    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:57:59.377858 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374003    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374005    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374008    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374010    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374012    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374015    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374021    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374023    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374026    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374028    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374031    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374033    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374036    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374039    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374041    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374043    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374046    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374048    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374051    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374053    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:57:59.378349 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374056    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374058    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374063    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374066    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374069    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374072    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374075    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374078    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374080    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374082    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374085    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374087    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374089    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374092    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374094    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374097    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374099    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374101    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374106    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:57:59.378848 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374108    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374111    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374114    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374116    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374119    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374123    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374126    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374129    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374133    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374135    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374138    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374140    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374143    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374145    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374148    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374150    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374154    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374156    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:57:59.379304 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374159    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374163    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374166    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374239    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374270    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374275    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374279    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374283    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374287    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374291    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374295    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374298    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374301    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374305    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374310    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374313    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374316    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374319    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374322    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374324    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374327    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374331    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374334    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374337    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 03:57:59.380171 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374340    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374344    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374347    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374351    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374354    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374357    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374361    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374364    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374367    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374371    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374374    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374378    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374381    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374383    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374386    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374389    2580 flags.go:64] FLAG: --enable-server="true"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374392    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374396    2580 flags.go:64] FLAG: --event-burst="100"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374399    2580 flags.go:64] FLAG: --event-qps="50"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374401    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374404    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374408    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374411    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374415    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374418    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 03:57:59.381156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374421    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374423    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374426    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374429    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374431    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374434    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374437    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374439    2580 flags.go:64] FLAG: --feature-gates=""
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374443    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374445    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374448    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374451    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374454    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374457    2580 flags.go:64] FLAG: --help="false"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374459    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374462    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374469    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374472    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374476    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374479    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374482    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374484    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374487    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 03:57:59.382145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374490    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374492    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374495    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374498    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374501    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374504    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374507    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374509    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374513    2580 flags.go:64] FLAG: --lock-file=""
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374516    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374519    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374522    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374529    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374531    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374534    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374537    2580 flags.go:64] FLAG: --logging-format="text"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374540    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374543    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374546    2580 flags.go:64] FLAG: --manifest-url=""
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374549    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374553    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374556    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374559    2580 flags.go:64] FLAG: --max-pods="110"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374562    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374565    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 03:57:59.382814 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374570    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374573    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374576    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374578    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374581    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374588    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374591    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374594    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374597    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374601    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374606    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374609    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374612    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374615    2580 flags.go:64] FLAG: --port="10250"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374618    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374621    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d7ee7bd227363fdb"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374625    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374628    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374631    2580 flags.go:64] FLAG: --register-node="true"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374634    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374637    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374640    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374643    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374646    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374649    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 03:57:59.383450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374651    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374654    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374657    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374660    2580 flags.go:64] FLAG: --runonce="false"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374662    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374665    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374668    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374672    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374675    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374678    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374681    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374684    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374687    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374690    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374693    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374695    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374698    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374701    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374703    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374708    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374711    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374713    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374717    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374720    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374723    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 03:57:59.384092 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374725    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374728    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374731    2580 flags.go:64] FLAG: --v="2"
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374734    2580 flags.go:64] FLAG: --version="false"
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374738    2580 flags.go:64] FLAG: --vmodule=""
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374741    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.374744    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374874    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374878    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374881    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374884    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374886    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374889    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374891    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374895    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374897    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374900    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374902    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374904    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374907    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374909    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:57:59.384979 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374911    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374914    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374917    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374919    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374921    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374924    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374927    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374930    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374932    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374936    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374940    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374943    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374946    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374948    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374951    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374953    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374956    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374958    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374961    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:57:59.385615 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374963    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374965    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374969    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374973    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374977    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374980    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374984    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374987    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374990    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374993    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374995    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.374999    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375001    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375004    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375007    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375009    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375011    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375014    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375016    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:57:59.386117 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375020    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375022    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375025    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375027    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375029    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375032    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375035    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375038    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375040    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375043    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375045    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375047    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375050    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375052    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375054    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375057    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375060    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375062    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375065    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375069    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:57:59.386599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375071    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375073    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375076    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375078    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375081    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375083    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375085    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375088    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375090    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375092    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375095    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375097    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375101    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375103    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375105    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375108    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375110    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375113    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375115    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375118    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:57:59.387112 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375120    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375123    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375126    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375128    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375131    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375133    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.375135    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.375141    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.385113    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.385156    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.385184    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.385193    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385584    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385599    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:57:59.387599 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385602    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385605    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385609    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385613    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385616    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385618    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385621    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385624    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385626    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385631    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385636    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385639    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385641    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385644    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385647    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385649    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385652    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385654    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385657    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:57:59.387966 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385659    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385662    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385664    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385667    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385670    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385672    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385674    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385677    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385684    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385687    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385690    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385692    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385695    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385698    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385701    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385704    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385706    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385709    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385712    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385714    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:57:59.388422 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385717    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385720    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385722    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385726    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385730    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385733    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385736    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385739    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385742    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385744    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385747    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385749    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385752    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385754    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385757    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385760    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385782    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385785    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385788    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385790    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:57:59.388948 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385793    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385796    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385798    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385801    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385804    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385806    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385809    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385812    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385814    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385817    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385821    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385823    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385826    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385828    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385832    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385835    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385838    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385840    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385843    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:57:59.389434 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385846    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385848    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385851    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385854    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385857    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385859    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385862    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385865    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385867    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385870    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385872    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385875    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385877    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385880    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385883    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385885    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385888    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385891    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:57:59.390032 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.385894    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.385899    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386009    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386013    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386016    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386019    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386021    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386024    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386026    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386029    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386031    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386034    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386036    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386039    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386043    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 03:57:59.390465 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386045    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386048    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386050    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386052    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386055    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386057    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386059    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386062    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386064    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386067    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386069    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386072    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386075    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386078    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386080    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386083    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386086    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386088    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386091    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386093    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 03:57:59.390850 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386096    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386098    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386100    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386103    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386105    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386107    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386110    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386112    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386115    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386117    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386121    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386123    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386126    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386129    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386131    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386134    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386137    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386139    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386142    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386144    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 03:57:59.391365 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386146    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386149    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386151    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386153    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386156    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386158    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386161    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386164    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386166    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386169    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386171    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386173    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386176    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386196    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386199    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386202    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386204    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386207    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386210    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 03:57:59.391861 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386213    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386217    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386220    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386223    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386225    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386228    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386231    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386234    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386236    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386240    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386243    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386245    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386248    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386250    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386253    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386256    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386258    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386260    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 03:57:59.392322 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386263    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 03:57:59.392792 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386266    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 03:57:59.392792 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386268    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 03:57:59.392792 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386271    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 03:57:59.392792 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386273    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 03:57:59.392792 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386276    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 03:57:59.392792 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386280    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 03:57:59.392792 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386283    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 03:57:59.392792 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:57:59.386286    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 03:57:59.392792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.386290    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 03:57:59.392792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.386396    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 03:57:59.392792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.388914    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 03:57:59.392792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.390002    2580 server.go:1019] "Starting client certificate rotation"
Sep 09 03:57:59.392792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.390096    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 03:57:59.393121 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.390123    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 03:57:59.393581 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.393566    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 03:57:59.394737 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.394718    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 03:57:59.406512 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.406493    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 09 03:57:59.411582 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.411566    2580 log.go:25] "Validated CRI v1 image API"
Sep 09 03:57:59.412390 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.412374    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 03:57:59.416851 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.416828    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 adf4726c-3390-44f9-9ec1-0ca88bdc38cd:/dev/nvme0n1p3 e7656246-ef0f-4d2c-b5e6-a9413e31b669:/dev/nvme0n1p4]
Sep 09 03:57:59.416922 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.416849    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 03:57:59.418326 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.418308    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 03:57:59.423037 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.422923    2580 manager.go:216] Machine: {Timestamp:2026-09-09 03:57:59.42150798 +0000 UTC m=+0.371198628 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098638 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e478eaa2d22cad341ef0e5f4a23d2 SystemUUID:ec2e478e-aa2d-22ca-d341-ef0e5f4a23d2 BootID:d8979cad-c670-4472-8cf0-9fd2044834ac Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:68:44:17:fb:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:68:44:17:fb:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:da:7b:17:dd:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 03:57:59.423037 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.423030    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 03:57:59.423190 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.423133    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 03:57:59.426926 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.426901    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 03:57:59.427066 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.426930    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-198.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 03:57:59.427783 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.427758    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 03:57:59.427820 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.427789    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 03:57:59.427820 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.427798    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 03:57:59.428652 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.428643    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 03:57:59.430597 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.430587    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 03:57:59.441262 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.441244    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qk7wg"
Sep 09 03:57:59.442124 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.442110    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 03:57:59.442273 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.442261    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 03:57:59.442314 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.442303    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 03:57:59.442347 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.442319    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 03:57:59.442347 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.442338    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 03:57:59.442436 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.442353    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 09 03:57:59.442436 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.442366    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 03:57:59.442847 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.442809    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 03:57:59.442847 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.442836    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 03:57:59.446527 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.446496    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 03:57:59.449142 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.449122    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qk7wg"
Sep 09 03:57:59.449488 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.449475    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 03:57:59.449537 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.449506    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 03:57:59.449682 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.449672    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 03:57:59.449717 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.449687    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 03:57:59.449717 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.449693    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 03:57:59.449717 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.449698    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 03:57:59.449717 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.449706    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 03:57:59.449717 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.449711    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 03:57:59.450402 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.450393    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 03:57:59.450402 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.450402    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 03:57:59.450477 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.450409    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 03:57:59.450477 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.450415    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 03:57:59.450477 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.450423    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 03:57:59.450477 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.450430    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 03:57:59.451305 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.451295    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 03:57:59.451305 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.451305    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 03:57:59.453695 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.453677    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-198.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 03:57:59.454122 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.454098    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-198.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 03:57:59.454122 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.454098    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 03:57:59.455340 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.455324    2580 server.go:1268] "Started kubelet"
Sep 09 03:57:59.455438 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.455389    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 03:57:59.455491 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.455455    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 03:57:59.455534 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.455392    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 03:57:59.456107 ip-10-0-142-198 systemd[1]: Started Kubernetes Kubelet.
Sep 09 03:57:59.457182 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.457000    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 03:57:59.457532 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.457515    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 03:57:59.459802 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.459782    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 03:57:59.461607 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.461591    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 03:57:59.461732 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.460536    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-198.ec2.internal.18d38af9184b4adb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-198.ec2.internal,UID:ip-10-0-142-198.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-198.ec2.internal,},FirstTimestamp:2026-09-09 03:57:59.455296219 +0000 UTC m=+0.404986863,LastTimestamp:2026-09-09 03:57:59.455296219 +0000 UTC m=+0.404986863,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-198.ec2.internal,}"
Sep 09 03:57:59.462259 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.462239    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 03:57:59.462442 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.462420    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:57:59.462932 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.462916    2580 factory.go:55] Registering systemd factory
Sep 09 03:57:59.462932 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.462935    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 09 03:57:59.463089 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.463073    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 03:57:59.463089 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.463085    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 03:57:59.463185 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.463097    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 03:57:59.463185 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.463175    2580 factory.go:153] Registering CRI-O factory
Sep 09 03:57:59.463251 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.463187    2580 factory.go:223] Registration of the crio container factory successfully
Sep 09 03:57:59.463251 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.463219    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 03:57:59.463251 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.463226    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 03:57:59.463251 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.463252    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 03:57:59.463407 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.463266    2580 factory.go:103] Registering Raw factory
Sep 09 03:57:59.463407 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.463277    2580 manager.go:1196] Started watching for new ooms in manager
Sep 09 03:57:59.463707 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.463694    2580 manager.go:349] Starting recovery of all containers
Sep 09 03:57:59.472034 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.472006    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:57:59.475866 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.475821    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-198.ec2.internal\" not found" node="ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.476750 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.476734    2580 manager.go:354] Recovery completed
Sep 09 03:57:59.480851 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.480838    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:57:59.482699 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.482684    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:57:59.482777 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.482712    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:57:59.482777 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.482722    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:57:59.483279 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.483263    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 03:57:59.483279 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.483277    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 03:57:59.483412 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.483297    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 03:57:59.483412 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.483308    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:57:59.483965 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.483951    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:57:59.484021 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.483976    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:57:59.484021 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.483986    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:57:59.485453 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.485440    2580 policy_none.go:49] "None policy: Start"
Sep 09 03:57:59.485513 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.485459    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 03:57:59.485513 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.485473    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 03:57:59.487340 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.487328    2580 policy_none.go:47] "Start"
Sep 09 03:57:59.523584 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.523569    2580 manager.go:337] "Starting Device Plugin manager"
Sep 09 03:57:59.523671 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.523618    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 03:57:59.523671 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.523631    2580 server.go:87] "Starting device plugin registration server"
Sep 09 03:57:59.523990 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.523978    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 03:57:59.524041 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.523993    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 03:57:59.524125 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.524076    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 03:57:59.524212 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.524195    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 03:57:59.524212 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.524209    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 03:57:59.524642 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.524621    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 03:57:59.524733 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.524674    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:57:59.587357 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.587323    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 03:57:59.589526 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.588635    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 03:57:59.589526 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.588677    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 03:57:59.589526 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.588701    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 03:57:59.589526 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.588735    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 03:57:59.591210 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.591193    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:57:59.624268 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.624189    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:57:59.625000 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.624985    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:57:59.625068 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.625012    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:57:59.625068 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.625023    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:57:59.625068 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.625044    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.631553 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.631540    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.631611 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.631559    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-198.ec2.internal\": node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:57:59.645077 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.645056    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:57:59.689375 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.689337    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-198.ec2.internal"]
Sep 09 03:57:59.689458 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.689414    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:57:59.690250 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.690235    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:57:59.690321 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.690267    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:57:59.690321 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.690281    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:57:59.692638 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.692625    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:57:59.692761 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.692749    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.692827 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.692797    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:57:59.693321 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.693300    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:57:59.693395 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.693328    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:57:59.693395 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.693302    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:57:59.693395 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.693373    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:57:59.693395 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.693338    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:57:59.693395 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.693387    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:57:59.695623 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.695611    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.695681 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.695633    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 03:57:59.696298 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.696283    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 03:57:59.696383 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.696309    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 03:57:59.696383 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.696321    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeHasSufficientPID"
Sep 09 03:57:59.717501 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.717480    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-198.ec2.internal\" not found" node="ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.721040 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.721025    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-198.ec2.internal\" not found" node="ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.745930 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.745904    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:57:59.764622 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.764601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/643c72097878f1443f4703f561b5b845-config\") pod \"kube-apiserver-proxy-ip-10-0-142-198.ec2.internal\" (UID: \"643c72097878f1443f4703f561b5b845\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.764751 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.764632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6eb4d3f7a360a75325b5b6c8577884f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal\" (UID: \"6eb4d3f7a360a75325b5b6c8577884f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.764751 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.764660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6eb4d3f7a360a75325b5b6c8577884f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal\" (UID: \"6eb4d3f7a360a75325b5b6c8577884f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.846416 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.846388    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:57:59.865822 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.865798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6eb4d3f7a360a75325b5b6c8577884f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal\" (UID: \"6eb4d3f7a360a75325b5b6c8577884f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.865897 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.865827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6eb4d3f7a360a75325b5b6c8577884f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal\" (UID: \"6eb4d3f7a360a75325b5b6c8577884f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.865897 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.865844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/643c72097878f1443f4703f561b5b845-config\") pod \"kube-apiserver-proxy-ip-10-0-142-198.ec2.internal\" (UID: \"643c72097878f1443f4703f561b5b845\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.865897 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.865876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/643c72097878f1443f4703f561b5b845-config\") pod \"kube-apiserver-proxy-ip-10-0-142-198.ec2.internal\" (UID: \"643c72097878f1443f4703f561b5b845\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.865996 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.865890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6eb4d3f7a360a75325b5b6c8577884f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal\" (UID: \"6eb4d3f7a360a75325b5b6c8577884f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.865996 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:57:59.865900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6eb4d3f7a360a75325b5b6c8577884f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal\" (UID: \"6eb4d3f7a360a75325b5b6c8577884f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal"
Sep 09 03:57:59.947252 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:57:59.947173    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:58:00.020696 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:00.020660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal"
Sep 09 03:58:00.023874 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:00.023858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-198.ec2.internal"
Sep 09 03:58:00.047584 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:00.047553    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:58:00.148140 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:00.148097    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:58:00.248638 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:00.248570    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:58:00.349241 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:00.349215    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:58:00.389695 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:00.389663    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 03:58:00.390133 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:00.389842    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 03:58:00.390133 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:00.389846    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 03:58:00.450224 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:00.450200    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:58:00.454542 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:00.454506    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 03:52:59 +0000 UTC" deadline="2028-04-27 12:18:24.688471802 +0000 UTC"
Sep 09 03:58:00.454609 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:00.454543    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14312h20m24.233932834s"
Sep 09 03:58:00.550674 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:00.550614    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:58:00.603383 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:58:00.603353    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb4d3f7a360a75325b5b6c8577884f6.slice/crio-f3ca1ca96f3e98d50f405efaba5f50407b46a4839340ce5496c0b16dccfde199 WatchSource:0}: Error finding container f3ca1ca96f3e98d50f405efaba5f50407b46a4839340ce5496c0b16dccfde199: Status 404 returned error can't find the container with id f3ca1ca96f3e98d50f405efaba5f50407b46a4839340ce5496c0b16dccfde199
Sep 09 03:58:00.603739 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:58:00.603716    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643c72097878f1443f4703f561b5b845.slice/crio-95701d7d398be8c01ac2ea3040f72e2575506e06b48da9c061d195f7a0543bd3 WatchSource:0}: Error finding container 95701d7d398be8c01ac2ea3040f72e2575506e06b48da9c061d195f7a0543bd3: Status 404 returned error can't find the container with id 95701d7d398be8c01ac2ea3040f72e2575506e06b48da9c061d195f7a0543bd3
Sep 09 03:58:00.608243 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:00.608219    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:58:00.651089 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:00.651067    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:58:00.651818 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:00.651799    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:58:00.751460 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:00.751421    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:58:00.851993 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:00.851962    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:58:00.952260 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:00.952223    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-198.ec2.internal\" not found"
Sep 09 03:58:00.985219 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:00.985196    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:58:01.062941 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.062907    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal"
Sep 09 03:58:01.076164 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.076138    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:58:01.077064 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.077046    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-198.ec2.internal"
Sep 09 03:58:01.088275 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.088256    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 03:58:01.443617 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.443583    2580 apiserver.go:52] "Watching apiserver"
Sep 09 03:58:01.455327 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.454869    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 03:58:01.456486 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.456454    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-s6f48","kube-system/konnectivity-agent-6j4fw","kube-system/kube-apiserver-proxy-ip-10-0-142-198.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb","openshift-image-registry/node-ca-6h7v6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal","openshift-multus/multus-additional-cni-plugins-x5fkn","openshift-multus/multus-wjdpl","openshift-multus/network-metrics-daemon-xwl9d","kube-system/global-pull-secret-syncer-brsc7","openshift-cluster-node-tuning-operator/tuned-g5ngq","openshift-dns/node-resolver-h6bnp","openshift-network-diagnostics/network-check-target-qhb72","openshift-network-operator/iptables-alerter-55thf"]
Sep 09 03:58:01.460890 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.460116    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.463091 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.461822    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 03:58:01.463753 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.463411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.464722 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.464232    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 03:58:01.464722 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.464260    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 03:58:01.464722 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.464436    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 03:58:01.464722 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.464472    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x8zjl\""
Sep 09 03:58:01.464722 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.464559    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 03:58:01.464722 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.464616    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 03:58:01.465750 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.465730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.466333 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.466201    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:58:01.466333 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.466225    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h7965\""
Sep 09 03:58:01.466551 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.466384    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 03:58:01.468324 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.468294    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.468964 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.468946    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 03:58:01.469219 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.469198    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dkvrf\""
Sep 09 03:58:01.470397 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.470378    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.471382 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.471362    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 03:58:01.471628 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.471613    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-26d7m\""
Sep 09 03:58:01.471988 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.471967    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 03:58:01.472112 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.471972    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-modprobe-d\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-run\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-var-lib-kubelet\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-tuned\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rxrj\" (UniqueName: \"kubernetes.io/projected/1c4180d7-3001-4994-a152-a1a98bd27aa1-kube-api-access-6rxrj\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-run-netns\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-run-multus-certs\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-kubernetes\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-cnibin\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-run-k8s-cni-cncf-io\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-var-lib-cni-multus\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-hostroot\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-system-cni-dir\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.475425 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-cnibin\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-sysconfig\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-sysctl-d\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-lib-modules\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-multus-daemon-config\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-etc-kubernetes\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473874    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-sysctl-conf\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-cni-binary-copy\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-var-lib-cni-bin\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xj5g\" (UniqueName: \"kubernetes.io/projected/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-kube-api-access-7xj5g\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.473982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-host\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-system-cni-dir\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-multus-cni-dir\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-var-lib-kubelet\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.476160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txhqq\" (UniqueName: \"kubernetes.io/projected/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-kube-api-access-txhqq\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-systemd\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-sys\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c4180d7-3001-4994-a152-a1a98bd27aa1-tmp\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:01.474208    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474211    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-multus-socket-dir-parent\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474264    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-multus-conf-dir\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474576    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474591    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.474612    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.475219    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.475869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tc4d\""
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.476520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:01.477011 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:01.476587    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:01.479368 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.478855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6j4fw"
Sep 09 03:58:01.481750 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.481722    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:01.481873 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:01.481816    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:01.482033 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.482014    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vtkl7\""
Sep 09 03:58:01.482101 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.482041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 03:58:01.482304 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.482210    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 03:58:01.485600 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.485582    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6h7v6"
Sep 09 03:58:01.488988 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.488255    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h6bnp"
Sep 09 03:58:01.488988 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.488817    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 03:58:01.488988 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.488900    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 03:58:01.488988 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.488915    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 03:58:01.489226 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.489147    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r9xnc\""
Sep 09 03:58:01.491784 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.490942    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 03:58:01.491784 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.490994    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-55thf"
Sep 09 03:58:01.491784 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.491660    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 03:58:01.496760 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.496061    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jjf5g\""
Sep 09 03:58:01.496760 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.496445    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 03:58:01.496760 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.496528    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7zlmw\""
Sep 09 03:58:01.496760 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.496548    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 03:58:01.496760 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.496631    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 03:58:01.499950 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.499930    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9zxsf"
Sep 09 03:58:01.506936 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.506916    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9zxsf"
Sep 09 03:58:01.564502 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.564475    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 03:58:01.575341 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f88a2c9f-c508-4bab-8a3a-eb3197e93ccb-host\") pod \"node-ca-6h7v6\" (UID: \"f88a2c9f-c508-4bab-8a3a-eb3197e93ccb\") " pod="openshift-image-registry/node-ca-6h7v6"
Sep 09 03:58:01.575489 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9vlz\" (UniqueName: \"kubernetes.io/projected/f88a2c9f-c508-4bab-8a3a-eb3197e93ccb-kube-api-access-w9vlz\") pod \"node-ca-6h7v6\" (UID: \"f88a2c9f-c508-4bab-8a3a-eb3197e93ccb\") " pod="openshift-image-registry/node-ca-6h7v6"
Sep 09 03:58:01.575489 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-systemd-units\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.575489 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93076315-afaf-43ce-a72c-104751d6d4e2-kubelet-config\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:01.575489 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-system-cni-dir\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.575489 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-multus-cni-dir\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.575753 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-var-lib-kubelet\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.575753 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-socket-dir\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.575753 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a433ebe-7762-4b8a-b605-418518973e05-ovn-node-metrics-cert\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.575753 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93076315-afaf-43ce-a72c-104751d6d4e2-dbus\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:01.575753 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.575753 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-sys\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.575753 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c4180d7-3001-4994-a152-a1a98bd27aa1-tmp\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.575753 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-multus-socket-dir-parent\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.575753 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-multus-conf-dir\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.575753 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-kubelet\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-modprobe-d\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-registration-dir\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-device-dir\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-var-lib-kubelet\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-tuned\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rxrj\" (UniqueName: \"kubernetes.io/projected/1c4180d7-3001-4994-a152-a1a98bd27aa1-kube-api-access-6rxrj\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.575988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzfbf\" (UniqueName: \"kubernetes.io/projected/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-kube-api-access-bzfbf\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3-agent-certs\") pod \"konnectivity-agent-6j4fw\" (UID: \"22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3\") " pod="kube-system/konnectivity-agent-6j4fw"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-kubernetes\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-cnibin\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-run-k8s-cni-cncf-io\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-var-lib-cni-multus\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-etc-selinux\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.576302 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-run-openvswitch\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1537ac59-5afb-4843-aa47-39dbbaf34a47-hosts-file\") pod \"node-resolver-h6bnp\" (UID: \"1537ac59-5afb-4843-aa47-39dbbaf34a47\") " pod="openshift-dns/node-resolver-h6bnp"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-system-cni-dir\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs\") pod \"network-metrics-daemon-xwl9d\" (UID: \"57ca0680-7caf-465f-bedd-44a747cc90af\") " pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46cf1ce2-20f8-4818-9e37-83b7a68260b7-iptables-alerter-script\") pod \"iptables-alerter-55thf\" (UID: \"46cf1ce2-20f8-4818-9e37-83b7a68260b7\") " pod="openshift-network-operator/iptables-alerter-55thf"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a433ebe-7762-4b8a-b605-418518973e05-ovnkube-config\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-sysconfig\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-lib-modules\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-run-systemd\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-cni-binary-copy\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f88a2c9f-c508-4bab-8a3a-eb3197e93ccb-serviceca\") pod \"node-ca-6h7v6\" (UID: \"f88a2c9f-c508-4bab-8a3a-eb3197e93ccb\") " pod="openshift-image-registry/node-ca-6h7v6"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-var-lib-openvswitch\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a433ebe-7762-4b8a-b605-418518973e05-env-overrides\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3-konnectivity-ca\") pod \"konnectivity-agent-6j4fw\" (UID: \"22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3\") " pod="kube-system/konnectivity-agent-6j4fw"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-host\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46cf1ce2-20f8-4818-9e37-83b7a68260b7-host-slash\") pod \"iptables-alerter-55thf\" (UID: \"46cf1ce2-20f8-4818-9e37-83b7a68260b7\") " pod="openshift-network-operator/iptables-alerter-55thf"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v5hl\" (UniqueName: \"kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl\") pod \"network-check-target-qhb72\" (UID: \"2988c802-f1a1-4199-9131-ecca43a5f4df\") " pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:01.577156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6rdg\" (UniqueName: \"kubernetes.io/projected/1537ac59-5afb-4843-aa47-39dbbaf34a47-kube-api-access-z6rdg\") pod \"node-resolver-h6bnp\" (UID: \"1537ac59-5afb-4843-aa47-39dbbaf34a47\") " pod="openshift-dns/node-resolver-h6bnp"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txhqq\" (UniqueName: \"kubernetes.io/projected/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-kube-api-access-txhqq\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-systemd\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjtk\" (UniqueName: \"kubernetes.io/projected/57ca0680-7caf-465f-bedd-44a747cc90af-kube-api-access-dxjtk\") pod \"network-metrics-daemon-xwl9d\" (UID: \"57ca0680-7caf-465f-bedd-44a747cc90af\") " pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-etc-openvswitch\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-cni-netd\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1537ac59-5afb-4843-aa47-39dbbaf34a47-tmp-dir\") pod \"node-resolver-h6bnp\" (UID: \"1537ac59-5afb-4843-aa47-39dbbaf34a47\") " pod="openshift-dns/node-resolver-h6bnp"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-run\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-run-netns\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-run-multus-certs\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.576989    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-run-ovn-kubernetes\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-cni-bin\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-hostroot\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qmxk\" (UniqueName: \"kubernetes.io/projected/46cf1ce2-20f8-4818-9e37-83b7a68260b7-kube-api-access-6qmxk\") pod \"iptables-alerter-55thf\" (UID: \"46cf1ce2-20f8-4818-9e37-83b7a68260b7\") " pod="openshift-network-operator/iptables-alerter-55thf"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577068    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-sys-fs\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-run-netns\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.577995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-node-log\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a433ebe-7762-4b8a-b605-418518973e05-ovnkube-script-lib\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwhfd\" (UniqueName: \"kubernetes.io/projected/2a433ebe-7762-4b8a-b605-418518973e05-kube-api-access-xwhfd\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-slash\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-log-socket\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-cnibin\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-sysctl-d\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-multus-daemon-config\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-etc-kubernetes\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-run-ovn\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-sysctl-conf\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-var-lib-cni-bin\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.577545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xj5g\" (UniqueName: \"kubernetes.io/projected/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-kube-api-access-7xj5g\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.578079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-system-cni-dir\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.578792 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.578156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-multus-cni-dir\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.578199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-var-lib-kubelet\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.578422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-host\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.578588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-cnibin\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.578602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-multus-socket-dir-parent\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.578646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-multus-conf-dir\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.578680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-hostroot\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.578714    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-sysctl-d\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.578722    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-var-lib-kubelet\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.578761    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-run-k8s-cni-cncf-io\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.578759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-modprobe-d\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-kubernetes\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-sys\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-sysconfig\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-system-cni-dir\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-systemd\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-cnibin\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.579558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-multus-daemon-config\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.580348 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-var-lib-cni-multus\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.580348 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-run\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.580348 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579392    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-run-netns\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.580348 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-etc-kubernetes\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.580348 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-run-multus-certs\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.580348 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-host-var-lib-cni-bin\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.580348 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-sysctl-conf\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.580348 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579628    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 03:58:01.580348 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.580348 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.579801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c4180d7-3001-4994-a152-a1a98bd27aa1-lib-modules\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.580348 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.580012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.580348 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.580019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-cni-binary-copy\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.580348 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.580150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.584423 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.584402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c4180d7-3001-4994-a152-a1a98bd27aa1-etc-tuned\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.584830 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.584762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c4180d7-3001-4994-a152-a1a98bd27aa1-tmp\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.592593 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.589185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rxrj\" (UniqueName: \"kubernetes.io/projected/1c4180d7-3001-4994-a152-a1a98bd27aa1-kube-api-access-6rxrj\") pod \"tuned-g5ngq\" (UID: \"1c4180d7-3001-4994-a152-a1a98bd27aa1\") " pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.592593 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.592543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txhqq\" (UniqueName: \"kubernetes.io/projected/aa72aab2-7d1d-4c62-83e3-a01afac7ee1c-kube-api-access-txhqq\") pod \"multus-additional-cni-plugins-x5fkn\" (UID: \"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c\") " pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.595247 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.595202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xj5g\" (UniqueName: \"kubernetes.io/projected/cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea-kube-api-access-7xj5g\") pod \"multus-wjdpl\" (UID: \"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea\") " pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.598996 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.598893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal" event={"ID":"6eb4d3f7a360a75325b5b6c8577884f6","Type":"ContainerStarted","Data":"f3ca1ca96f3e98d50f405efaba5f50407b46a4839340ce5496c0b16dccfde199"}
Sep 09 03:58:01.602957 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.602934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-198.ec2.internal" event={"ID":"643c72097878f1443f4703f561b5b845","Type":"ContainerStarted","Data":"95701d7d398be8c01ac2ea3040f72e2575506e06b48da9c061d195f7a0543bd3"}
Sep 09 03:58:01.673967 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.673938    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:58:01.678038 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.678009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-etc-selinux\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.678174 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.678049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-run-openvswitch\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.678174 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.678078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1537ac59-5afb-4843-aa47-39dbbaf34a47-hosts-file\") pod \"node-resolver-h6bnp\" (UID: \"1537ac59-5afb-4843-aa47-39dbbaf34a47\") " pod="openshift-dns/node-resolver-h6bnp"
Sep 09 03:58:01.678174 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.678106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs\") pod \"network-metrics-daemon-xwl9d\" (UID: \"57ca0680-7caf-465f-bedd-44a747cc90af\") " pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:01.678174 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.678145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46cf1ce2-20f8-4818-9e37-83b7a68260b7-iptables-alerter-script\") pod \"iptables-alerter-55thf\" (UID: \"46cf1ce2-20f8-4818-9e37-83b7a68260b7\") " pod="openshift-network-operator/iptables-alerter-55thf"
Sep 09 03:58:01.678174 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.678165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a433ebe-7762-4b8a-b605-418518973e05-ovnkube-config\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.678353 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.678192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-run-systemd\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.679865 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.678475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f88a2c9f-c508-4bab-8a3a-eb3197e93ccb-serviceca\") pod \"node-ca-6h7v6\" (UID: \"f88a2c9f-c508-4bab-8a3a-eb3197e93ccb\") " pod="openshift-image-registry/node-ca-6h7v6"
Sep 09 03:58:01.679865 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.678524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-var-lib-openvswitch\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.679865 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.678564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a433ebe-7762-4b8a-b605-418518973e05-env-overrides\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.679865 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.678597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3-konnectivity-ca\") pod \"konnectivity-agent-6j4fw\" (UID: \"22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3\") " pod="kube-system/konnectivity-agent-6j4fw"
Sep 09 03:58:01.683903 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.678623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46cf1ce2-20f8-4818-9e37-83b7a68260b7-host-slash\") pod \"iptables-alerter-55thf\" (UID: \"46cf1ce2-20f8-4818-9e37-83b7a68260b7\") " pod="openshift-network-operator/iptables-alerter-55thf"
Sep 09 03:58:01.684024 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.683961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5hl\" (UniqueName: \"kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl\") pod \"network-check-target-qhb72\" (UID: \"2988c802-f1a1-4199-9131-ecca43a5f4df\") " pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:01.684024 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.683985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-run-openvswitch\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.684024 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.684008    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6rdg\" (UniqueName: \"kubernetes.io/projected/1537ac59-5afb-4843-aa47-39dbbaf34a47-kube-api-access-z6rdg\") pod \"node-resolver-h6bnp\" (UID: \"1537ac59-5afb-4843-aa47-39dbbaf34a47\") " pod="openshift-dns/node-resolver-h6bnp"
Sep 09 03:58:01.684250 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.684050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjtk\" (UniqueName: \"kubernetes.io/projected/57ca0680-7caf-465f-bedd-44a747cc90af-kube-api-access-dxjtk\") pod \"network-metrics-daemon-xwl9d\" (UID: \"57ca0680-7caf-465f-bedd-44a747cc90af\") " pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:01.684250 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.684058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1537ac59-5afb-4843-aa47-39dbbaf34a47-hosts-file\") pod \"node-resolver-h6bnp\" (UID: \"1537ac59-5afb-4843-aa47-39dbbaf34a47\") " pod="openshift-dns/node-resolver-h6bnp"
Sep 09 03:58:01.684250 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.684087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-etc-openvswitch\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.684250 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.684115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-cni-netd\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.684250 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:01.684147    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:01.684250 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:01.684234    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs podName:57ca0680-7caf-465f-bedd-44a747cc90af nodeName:}" failed. No retries permitted until 2026-09-09 03:58:02.184213403 +0000 UTC m=+3.133904061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs") pod "network-metrics-daemon-xwl9d" (UID: "57ca0680-7caf-465f-bedd-44a747cc90af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.684868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/46cf1ce2-20f8-4818-9e37-83b7a68260b7-iptables-alerter-script\") pod \"iptables-alerter-55thf\" (UID: \"46cf1ce2-20f8-4818-9e37-83b7a68260b7\") " pod="openshift-network-operator/iptables-alerter-55thf"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.685002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1537ac59-5afb-4843-aa47-39dbbaf34a47-tmp-dir\") pod \"node-resolver-h6bnp\" (UID: \"1537ac59-5afb-4843-aa47-39dbbaf34a47\") " pod="openshift-dns/node-resolver-h6bnp"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.685154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-etc-selinux\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.685417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a433ebe-7762-4b8a-b605-418518973e05-ovnkube-config\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.684147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1537ac59-5afb-4843-aa47-39dbbaf34a47-tmp-dir\") pod \"node-resolver-h6bnp\" (UID: \"1537ac59-5afb-4843-aa47-39dbbaf34a47\") " pod="openshift-dns/node-resolver-h6bnp"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.685476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-run-ovn-kubernetes\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.685508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-cni-bin\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.685539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qmxk\" (UniqueName: \"kubernetes.io/projected/46cf1ce2-20f8-4818-9e37-83b7a68260b7-kube-api-access-6qmxk\") pod \"iptables-alerter-55thf\" (UID: \"46cf1ce2-20f8-4818-9e37-83b7a68260b7\") " pod="openshift-network-operator/iptables-alerter-55thf"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.685563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-sys-fs\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.685585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-run-netns\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.685605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-node-log\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.685760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-etc-openvswitch\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.685673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a433ebe-7762-4b8a-b605-418518973e05-ovnkube-script-lib\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.685937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-cni-bin\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.685987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwhfd\" (UniqueName: \"kubernetes.io/projected/2a433ebe-7762-4b8a-b605-418518973e05-kube-api-access-xwhfd\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-cni-netd\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688186 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-run-ovn-kubernetes\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-slash\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-log-socket\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-slash\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-run-ovn\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-sys-fs\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-log-socket\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-run-ovn\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686447    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f88a2c9f-c508-4bab-8a3a-eb3197e93ccb-host\") pod \"node-ca-6h7v6\" (UID: \"f88a2c9f-c508-4bab-8a3a-eb3197e93ccb\") " pod="openshift-image-registry/node-ca-6h7v6"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9vlz\" (UniqueName: \"kubernetes.io/projected/f88a2c9f-c508-4bab-8a3a-eb3197e93ccb-kube-api-access-w9vlz\") pod \"node-ca-6h7v6\" (UID: \"f88a2c9f-c508-4bab-8a3a-eb3197e93ccb\") " pod="openshift-image-registry/node-ca-6h7v6"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a433ebe-7762-4b8a-b605-418518973e05-ovnkube-script-lib\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-systemd-units\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93076315-afaf-43ce-a72c-104751d6d4e2-kubelet-config\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-systemd-units\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-run-systemd\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93076315-afaf-43ce-a72c-104751d6d4e2-kubelet-config\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f88a2c9f-c508-4bab-8a3a-eb3197e93ccb-host\") pod \"node-ca-6h7v6\" (UID: \"f88a2c9f-c508-4bab-8a3a-eb3197e93ccb\") " pod="openshift-image-registry/node-ca-6h7v6"
Sep 09 03:58:01.688942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-socket-dir\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686783    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-socket-dir\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:01.686909    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.686917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-run-netns\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:01.686974    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret podName:93076315-afaf-43ce-a72c-104751d6d4e2 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:02.186955025 +0000 UTC m=+3.136645661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret") pod "global-pull-secret-syncer-brsc7" (UID: "93076315-afaf-43ce-a72c-104751d6d4e2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f88a2c9f-c508-4bab-8a3a-eb3197e93ccb-serviceca\") pod \"node-ca-6h7v6\" (UID: \"f88a2c9f-c508-4bab-8a3a-eb3197e93ccb\") " pod="openshift-image-registry/node-ca-6h7v6"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a433ebe-7762-4b8a-b605-418518973e05-ovn-node-metrics-cert\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93076315-afaf-43ce-a72c-104751d6d4e2-dbus\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-kubelet\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-var-lib-openvswitch\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-registration-dir\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a433ebe-7762-4b8a-b605-418518973e05-env-overrides\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-registration-dir\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-device-dir\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3-konnectivity-ca\") pod \"konnectivity-agent-6j4fw\" (UID: \"22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3\") " pod="kube-system/konnectivity-agent-6j4fw"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.689779 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-kubelet\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.690443 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.690443 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-device-dir\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.690443 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93076315-afaf-43ce-a72c-104751d6d4e2-dbus\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:01.690443 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzfbf\" (UniqueName: \"kubernetes.io/projected/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-kube-api-access-bzfbf\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.690443 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.690443 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3-agent-certs\") pod \"konnectivity-agent-6j4fw\" (UID: \"22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3\") " pod="kube-system/konnectivity-agent-6j4fw"
Sep 09 03:58:01.690443 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.687819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a433ebe-7762-4b8a-b605-418518973e05-node-log\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.690443 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.688011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46cf1ce2-20f8-4818-9e37-83b7a68260b7-host-slash\") pod \"iptables-alerter-55thf\" (UID: \"46cf1ce2-20f8-4818-9e37-83b7a68260b7\") " pod="openshift-network-operator/iptables-alerter-55thf"
Sep 09 03:58:01.690443 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.690228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a433ebe-7762-4b8a-b605-418518973e05-ovn-node-metrics-cert\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.691014 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.690990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3-agent-certs\") pod \"konnectivity-agent-6j4fw\" (UID: \"22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3\") " pod="kube-system/konnectivity-agent-6j4fw"
Sep 09 03:58:01.695605 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:01.695541    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:01.695605 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:01.695569    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:01.695605 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:01.695584    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8v5hl for pod openshift-network-diagnostics/network-check-target-qhb72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:01.695825 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:01.695642    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl podName:2988c802-f1a1-4199-9131-ecca43a5f4df nodeName:}" failed. No retries permitted until 2026-09-09 03:58:02.195624156 +0000 UTC m=+3.145314810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5hl" (UniqueName: "kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl") pod "network-check-target-qhb72" (UID: "2988c802-f1a1-4199-9131-ecca43a5f4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:01.697761 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.697735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6rdg\" (UniqueName: \"kubernetes.io/projected/1537ac59-5afb-4843-aa47-39dbbaf34a47-kube-api-access-z6rdg\") pod \"node-resolver-h6bnp\" (UID: \"1537ac59-5afb-4843-aa47-39dbbaf34a47\") " pod="openshift-dns/node-resolver-h6bnp"
Sep 09 03:58:01.701701 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.701676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9vlz\" (UniqueName: \"kubernetes.io/projected/f88a2c9f-c508-4bab-8a3a-eb3197e93ccb-kube-api-access-w9vlz\") pod \"node-ca-6h7v6\" (UID: \"f88a2c9f-c508-4bab-8a3a-eb3197e93ccb\") " pod="openshift-image-registry/node-ca-6h7v6"
Sep 09 03:58:01.702479 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.702389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwhfd\" (UniqueName: \"kubernetes.io/projected/2a433ebe-7762-4b8a-b605-418518973e05-kube-api-access-xwhfd\") pod \"ovnkube-node-s6f48\" (UID: \"2a433ebe-7762-4b8a-b605-418518973e05\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.702479 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.702435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzfbf\" (UniqueName: \"kubernetes.io/projected/f4d0c2ce-88d3-4834-8e8c-e6a125aa4038-kube-api-access-bzfbf\") pod \"aws-ebs-csi-driver-node-tz5rb\" (UID: \"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.705613 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.705588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjtk\" (UniqueName: \"kubernetes.io/projected/57ca0680-7caf-465f-bedd-44a747cc90af-kube-api-access-dxjtk\") pod \"network-metrics-daemon-xwl9d\" (UID: \"57ca0680-7caf-465f-bedd-44a747cc90af\") " pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:01.706975 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.706940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qmxk\" (UniqueName: \"kubernetes.io/projected/46cf1ce2-20f8-4818-9e37-83b7a68260b7-kube-api-access-6qmxk\") pod \"iptables-alerter-55thf\" (UID: \"46cf1ce2-20f8-4818-9e37-83b7a68260b7\") " pod="openshift-network-operator/iptables-alerter-55thf"
Sep 09 03:58:01.779104 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.779066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x5fkn"
Sep 09 03:58:01.789825 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.789794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g5ngq"
Sep 09 03:58:01.802103 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.802069    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wjdpl"
Sep 09 03:58:01.810819 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.810792    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb"
Sep 09 03:58:01.818002 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.817977    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:01.825922 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.825899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6j4fw"
Sep 09 03:58:01.833871 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.833850    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6h7v6"
Sep 09 03:58:01.842207 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.842182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h6bnp"
Sep 09 03:58:01.850204 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.850181    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-55thf"
Sep 09 03:58:01.855277 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:01.855253    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 03:58:02.192466 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:02.192431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs\") pod \"network-metrics-daemon-xwl9d\" (UID: \"57ca0680-7caf-465f-bedd-44a747cc90af\") " pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:02.192660 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:02.192492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:02.192660 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:02.192557    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:02.192660 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:02.192600    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:02.192660 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:02.192635    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs podName:57ca0680-7caf-465f-bedd-44a747cc90af nodeName:}" failed. No retries permitted until 2026-09-09 03:58:03.192615009 +0000 UTC m=+4.142305660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs") pod "network-metrics-daemon-xwl9d" (UID: "57ca0680-7caf-465f-bedd-44a747cc90af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:02.192660 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:02.192655    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret podName:93076315-afaf-43ce-a72c-104751d6d4e2 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:03.192642603 +0000 UTC m=+4.142333236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret") pod "global-pull-secret-syncer-brsc7" (UID: "93076315-afaf-43ce-a72c-104751d6d4e2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:02.293331 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:02.293295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5hl\" (UniqueName: \"kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl\") pod \"network-check-target-qhb72\" (UID: \"2988c802-f1a1-4199-9131-ecca43a5f4df\") " pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:02.293536 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:02.293480    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:02.293536 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:02.293509    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:02.293536 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:02.293520    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8v5hl for pod openshift-network-diagnostics/network-check-target-qhb72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:02.293677 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:02.293584    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl podName:2988c802-f1a1-4199-9131-ecca43a5f4df nodeName:}" failed. No retries permitted until 2026-09-09 03:58:03.293563317 +0000 UTC m=+4.243253952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5hl" (UniqueName: "kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl") pod "network-check-target-qhb72" (UID: "2988c802-f1a1-4199-9131-ecca43a5f4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:02.510658 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:02.510591    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 03:53:01 +0000 UTC" deadline="2028-06-23 00:25:44.494949262 +0000 UTC"
Sep 09 03:58:02.511036 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:02.510669    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15668h27m41.984292041s"
Sep 09 03:58:02.775791 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:58:02.775710    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa72aab2_7d1d_4c62_83e3_a01afac7ee1c.slice/crio-5b3217e11639a639d77731fcad66ed8ac6cc73d7dfb76b83e93c9a8d59aae706 WatchSource:0}: Error finding container 5b3217e11639a639d77731fcad66ed8ac6cc73d7dfb76b83e93c9a8d59aae706: Status 404 returned error can't find the container with id 5b3217e11639a639d77731fcad66ed8ac6cc73d7dfb76b83e93c9a8d59aae706
Sep 09 03:58:02.788916 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:58:02.788843    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88a2c9f_c508_4bab_8a3a_eb3197e93ccb.slice/crio-0d629f69bb4f269a1523e75394b49fba92ad8998d52b10ada9d9b66f1ab6214e WatchSource:0}: Error finding container 0d629f69bb4f269a1523e75394b49fba92ad8998d52b10ada9d9b66f1ab6214e: Status 404 returned error can't find the container with id 0d629f69bb4f269a1523e75394b49fba92ad8998d52b10ada9d9b66f1ab6214e
Sep 09 03:58:02.789852 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:58:02.789769    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1537ac59_5afb_4843_aa47_39dbbaf34a47.slice/crio-d418e310f88d84629d5570e036dc18a41f498a4b176ed9da4277b23d01b4f7fa WatchSource:0}: Error finding container d418e310f88d84629d5570e036dc18a41f498a4b176ed9da4277b23d01b4f7fa: Status 404 returned error can't find the container with id d418e310f88d84629d5570e036dc18a41f498a4b176ed9da4277b23d01b4f7fa
Sep 09 03:58:02.791351 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:58:02.791313    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cce36d_d1ae_4ae8_b3dc_30c4f8d38af3.slice/crio-b78147c1e88c31c6a97450f14a18948d7a040c394d0606e6e018dc79565f1b36 WatchSource:0}: Error finding container b78147c1e88c31c6a97450f14a18948d7a040c394d0606e6e018dc79565f1b36: Status 404 returned error can't find the container with id b78147c1e88c31c6a97450f14a18948d7a040c394d0606e6e018dc79565f1b36
Sep 09 03:58:02.792333 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:58:02.791788    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4180d7_3001_4994_a152_a1a98bd27aa1.slice/crio-0a1a364defc33c969574ed5d8e6b6bc9564053b5967843fa355abcb52fd97a7e WatchSource:0}: Error finding container 0a1a364defc33c969574ed5d8e6b6bc9564053b5967843fa355abcb52fd97a7e: Status 404 returned error can't find the container with id 0a1a364defc33c969574ed5d8e6b6bc9564053b5967843fa355abcb52fd97a7e
Sep 09 03:58:02.794340 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:58:02.794012    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d0c2ce_88d3_4834_8e8c_e6a125aa4038.slice/crio-e25593b6e794b414c53b0ba49a247b006b9e4da5955b9ae2b2e605ac343f47ed WatchSource:0}: Error finding container e25593b6e794b414c53b0ba49a247b006b9e4da5955b9ae2b2e605ac343f47ed: Status 404 returned error can't find the container with id e25593b6e794b414c53b0ba49a247b006b9e4da5955b9ae2b2e605ac343f47ed
Sep 09 03:58:02.794795 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:58:02.794755    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cf1ce2_20f8_4818_9e37_83b7a68260b7.slice/crio-4382f66832fe6e746b8c93c07ee9f7be8d9f995460522756871153febc905765 WatchSource:0}: Error finding container 4382f66832fe6e746b8c93c07ee9f7be8d9f995460522756871153febc905765: Status 404 returned error can't find the container with id 4382f66832fe6e746b8c93c07ee9f7be8d9f995460522756871153febc905765
Sep 09 03:58:02.818611 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:58:02.818350    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a433ebe_7762_4b8a_b605_418518973e05.slice/crio-864cb504b1d9c5e4ec59fd167330caaec85b6abeaa24c7ed41ea7a139ef7785b WatchSource:0}: Error finding container 864cb504b1d9c5e4ec59fd167330caaec85b6abeaa24c7ed41ea7a139ef7785b: Status 404 returned error can't find the container with id 864cb504b1d9c5e4ec59fd167330caaec85b6abeaa24c7ed41ea7a139ef7785b
Sep 09 03:58:03.200643 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.200511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:03.200643 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.200585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs\") pod \"network-metrics-daemon-xwl9d\" (UID: \"57ca0680-7caf-465f-bedd-44a747cc90af\") " pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:03.200643 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:03.200629    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:03.200967 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:03.200704    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret podName:93076315-afaf-43ce-a72c-104751d6d4e2 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:05.200682772 +0000 UTC m=+6.150373416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret") pod "global-pull-secret-syncer-brsc7" (UID: "93076315-afaf-43ce-a72c-104751d6d4e2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:03.200967 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:03.200731    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:03.200967 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:03.200812    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs podName:57ca0680-7caf-465f-bedd-44a747cc90af nodeName:}" failed. No retries permitted until 2026-09-09 03:58:05.200797438 +0000 UTC m=+6.150488090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs") pod "network-metrics-daemon-xwl9d" (UID: "57ca0680-7caf-465f-bedd-44a747cc90af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:03.301586 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.301501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5hl\" (UniqueName: \"kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl\") pod \"network-check-target-qhb72\" (UID: \"2988c802-f1a1-4199-9131-ecca43a5f4df\") " pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:03.301741 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:03.301652    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:03.301741 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:03.301674    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:03.301741 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:03.301687    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8v5hl for pod openshift-network-diagnostics/network-check-target-qhb72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:03.301892 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:03.301749    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl podName:2988c802-f1a1-4199-9131-ecca43a5f4df nodeName:}" failed. No retries permitted until 2026-09-09 03:58:05.301728887 +0000 UTC m=+6.251419527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5hl" (UniqueName: "kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl") pod "network-check-target-qhb72" (UID: "2988c802-f1a1-4199-9131-ecca43a5f4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:03.511735 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.511652    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 03:53:01 +0000 UTC" deadline="2028-02-07 20:06:06.677778023 +0000 UTC"
Sep 09 03:58:03.511735 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.511700    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12400h8m3.166083656s"
Sep 09 03:58:03.593109 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.591982    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:03.593109 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:03.592133    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:03.593109 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.592778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:03.593109 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:03.592904    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:03.593109 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.592970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:03.593109 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:03.593066    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:03.620589 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.619587    2580 generic.go:356] "Generic (PLEG): container finished" podID="6eb4d3f7a360a75325b5b6c8577884f6" containerID="5e73454b5fa86506cc33ce9fbd99d418cb1ef9a29865c56b882eea0be4bcf14b" exitCode=0
Sep 09 03:58:03.620589 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.619693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal" event={"ID":"6eb4d3f7a360a75325b5b6c8577884f6","Type":"ContainerDied","Data":"5e73454b5fa86506cc33ce9fbd99d418cb1ef9a29865c56b882eea0be4bcf14b"}
Sep 09 03:58:03.632918 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.632408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-198.ec2.internal" event={"ID":"643c72097878f1443f4703f561b5b845","Type":"ContainerStarted","Data":"71f16cbe54aed7ff35f6360ffb0b3bbc539e82bc7361edc7b4056ee5ec45b478"}
Sep 09 03:58:03.645916 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.645883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48" event={"ID":"2a433ebe-7762-4b8a-b605-418518973e05","Type":"ContainerStarted","Data":"864cb504b1d9c5e4ec59fd167330caaec85b6abeaa24c7ed41ea7a139ef7785b"}
Sep 09 03:58:03.656981 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.656861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb" event={"ID":"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038","Type":"ContainerStarted","Data":"e25593b6e794b414c53b0ba49a247b006b9e4da5955b9ae2b2e605ac343f47ed"}
Sep 09 03:58:03.663750 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.663692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-55thf" event={"ID":"46cf1ce2-20f8-4818-9e37-83b7a68260b7","Type":"ContainerStarted","Data":"4382f66832fe6e746b8c93c07ee9f7be8d9f995460522756871153febc905765"}
Sep 09 03:58:03.665754 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.665460    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-198.ec2.internal" podStartSLOduration=2.66544124 podStartE2EDuration="2.66544124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:03.66469176 +0000 UTC m=+4.614382420" watchObservedRunningTime="2026-09-09 03:58:03.66544124 +0000 UTC m=+4.615131900"
Sep 09 03:58:03.666447 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.666418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g5ngq" event={"ID":"1c4180d7-3001-4994-a152-a1a98bd27aa1","Type":"ContainerStarted","Data":"0a1a364defc33c969574ed5d8e6b6bc9564053b5967843fa355abcb52fd97a7e"}
Sep 09 03:58:03.669858 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.669820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6j4fw" event={"ID":"22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3","Type":"ContainerStarted","Data":"b78147c1e88c31c6a97450f14a18948d7a040c394d0606e6e018dc79565f1b36"}
Sep 09 03:58:03.675684 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.675657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5fkn" event={"ID":"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c","Type":"ContainerStarted","Data":"5b3217e11639a639d77731fcad66ed8ac6cc73d7dfb76b83e93c9a8d59aae706"}
Sep 09 03:58:03.694024 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.693989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wjdpl" event={"ID":"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea","Type":"ContainerStarted","Data":"6fd4cb8bb9fb06e42549832fa68a754e614ced68c7b52fcf179fcbaa4b7decc7"}
Sep 09 03:58:03.705806 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.705737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6h7v6" event={"ID":"f88a2c9f-c508-4bab-8a3a-eb3197e93ccb","Type":"ContainerStarted","Data":"0d629f69bb4f269a1523e75394b49fba92ad8998d52b10ada9d9b66f1ab6214e"}
Sep 09 03:58:03.730254 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:03.730215    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h6bnp" event={"ID":"1537ac59-5afb-4843-aa47-39dbbaf34a47","Type":"ContainerStarted","Data":"d418e310f88d84629d5570e036dc18a41f498a4b176ed9da4277b23d01b4f7fa"}
Sep 09 03:58:04.763077 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:04.761476    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal" event={"ID":"6eb4d3f7a360a75325b5b6c8577884f6","Type":"ContainerStarted","Data":"b8ad5a5eb0f495d46ef9f1073b64826176454ffcd375085d6ac22aa676223fc7"}
Sep 09 03:58:05.218588 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:05.217725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:05.218588 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:05.217825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs\") pod \"network-metrics-daemon-xwl9d\" (UID: \"57ca0680-7caf-465f-bedd-44a747cc90af\") " pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:05.218588 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:05.217981    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:05.218588 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:05.218044    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs podName:57ca0680-7caf-465f-bedd-44a747cc90af nodeName:}" failed. No retries permitted until 2026-09-09 03:58:09.218025209 +0000 UTC m=+10.167715847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs") pod "network-metrics-daemon-xwl9d" (UID: "57ca0680-7caf-465f-bedd-44a747cc90af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:05.218588 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:05.218453    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:05.218588 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:05.218502    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret podName:93076315-afaf-43ce-a72c-104751d6d4e2 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:09.218487724 +0000 UTC m=+10.168178361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret") pod "global-pull-secret-syncer-brsc7" (UID: "93076315-afaf-43ce-a72c-104751d6d4e2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:05.319074 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:05.319034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5hl\" (UniqueName: \"kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl\") pod \"network-check-target-qhb72\" (UID: \"2988c802-f1a1-4199-9131-ecca43a5f4df\") " pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:05.319238 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:05.319192    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:05.319238 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:05.319214    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:05.319238 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:05.319227    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8v5hl for pod openshift-network-diagnostics/network-check-target-qhb72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:05.319399 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:05.319285    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl podName:2988c802-f1a1-4199-9131-ecca43a5f4df nodeName:}" failed. No retries permitted until 2026-09-09 03:58:09.319267265 +0000 UTC m=+10.268957920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5hl" (UniqueName: "kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl") pod "network-check-target-qhb72" (UID: "2988c802-f1a1-4199-9131-ecca43a5f4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:05.590597 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:05.589879    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:05.590597 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:05.590000    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:05.590597 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:05.590451    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:05.590597 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:05.590544    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:05.591228 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:05.591053    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:05.591228 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:05.591165    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:07.590890 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:07.590852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:07.591274 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:07.590965    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:07.591274 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:07.591253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:07.591366 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:07.591338    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:07.591538 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:07.591522    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:07.591657 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:07.591616    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:09.254900 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:09.254855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:09.255374 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:09.254939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs\") pod \"network-metrics-daemon-xwl9d\" (UID: \"57ca0680-7caf-465f-bedd-44a747cc90af\") " pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:09.255374 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:09.255058    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:09.255374 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:09.255125    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs podName:57ca0680-7caf-465f-bedd-44a747cc90af nodeName:}" failed. No retries permitted until 2026-09-09 03:58:17.255104384 +0000 UTC m=+18.204795039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs") pod "network-metrics-daemon-xwl9d" (UID: "57ca0680-7caf-465f-bedd-44a747cc90af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:09.255551 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:09.255522    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:09.255603 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:09.255569    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret podName:93076315-afaf-43ce-a72c-104751d6d4e2 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:17.255555293 +0000 UTC m=+18.205245929 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret") pod "global-pull-secret-syncer-brsc7" (UID: "93076315-afaf-43ce-a72c-104751d6d4e2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:09.355959 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:09.355925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5hl\" (UniqueName: \"kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl\") pod \"network-check-target-qhb72\" (UID: \"2988c802-f1a1-4199-9131-ecca43a5f4df\") " pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:09.356169 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:09.356076    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:09.356169 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:09.356094    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:09.356169 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:09.356103    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8v5hl for pod openshift-network-diagnostics/network-check-target-qhb72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:09.356169 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:09.356157    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl podName:2988c802-f1a1-4199-9131-ecca43a5f4df nodeName:}" failed. No retries permitted until 2026-09-09 03:58:17.356143269 +0000 UTC m=+18.305833915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5hl" (UniqueName: "kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl") pod "network-check-target-qhb72" (UID: "2988c802-f1a1-4199-9131-ecca43a5f4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:09.590445 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:09.590418    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:09.590672 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:09.590515    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:09.590909 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:09.590891    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:09.590998 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:09.590982    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:09.591145 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:09.591048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:09.591145 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:09.591117    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:11.590471 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:11.590442    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:11.590944 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:11.590587    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:11.590944 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:11.590442    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:11.591046 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:11.590981    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:11.591046 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:11.591036    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:11.591158 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:11.591082    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:13.589109 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:13.589071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:13.589534 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:13.589078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:13.589534 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:13.589205    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:13.589534 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:13.589223    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:13.589534 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:13.589316    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:13.589534 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:13.589440    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:15.589533 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:15.589351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:15.590003 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:15.589354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:15.590003 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:15.589367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:15.590003 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:15.589712    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:15.590003 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:15.589600    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:15.590003 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:15.589825    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:17.312421 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:17.312388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:17.312887 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:17.312456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs\") pod \"network-metrics-daemon-xwl9d\" (UID: \"57ca0680-7caf-465f-bedd-44a747cc90af\") " pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:17.312887 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:17.312541    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:17.312887 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:17.312594    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:17.312887 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:17.312612    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret podName:93076315-afaf-43ce-a72c-104751d6d4e2 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:33.31259221 +0000 UTC m=+34.262282861 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret") pod "global-pull-secret-syncer-brsc7" (UID: "93076315-afaf-43ce-a72c-104751d6d4e2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:17.312887 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:17.312645    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs podName:57ca0680-7caf-465f-bedd-44a747cc90af nodeName:}" failed. No retries permitted until 2026-09-09 03:58:33.312628568 +0000 UTC m=+34.262319209 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs") pod "network-metrics-daemon-xwl9d" (UID: "57ca0680-7caf-465f-bedd-44a747cc90af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:17.413650 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:17.413616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5hl\" (UniqueName: \"kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl\") pod \"network-check-target-qhb72\" (UID: \"2988c802-f1a1-4199-9131-ecca43a5f4df\") " pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:17.413835 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:17.413741    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:17.413835 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:17.413791    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:17.413835 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:17.413805    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8v5hl for pod openshift-network-diagnostics/network-check-target-qhb72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:17.413990 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:17.413862    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl podName:2988c802-f1a1-4199-9131-ecca43a5f4df nodeName:}" failed. No retries permitted until 2026-09-09 03:58:33.413847965 +0000 UTC m=+34.363538622 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5hl" (UniqueName: "kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl") pod "network-check-target-qhb72" (UID: "2988c802-f1a1-4199-9131-ecca43a5f4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:17.589916 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:17.589884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:17.590099 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:17.590025    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:17.590427 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:17.590405    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:17.590537 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:17.590482    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:17.590669 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:17.590643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:17.590751 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:17.590740    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:19.590347 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:19.590306    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:19.590347 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:19.590342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:19.590882 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:19.590421    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:19.590882 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:19.590451    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:19.590882 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:19.590495    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:19.591498 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:19.591471    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:21.589404 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.589184    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:21.590121 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.589190    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:21.590121 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:21.589491    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:21.590121 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.589228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:21.590121 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:21.589566    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:21.590121 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:21.589674    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:21.800938 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.800851    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 03:58:21.801229 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.801206    2580 generic.go:356] "Generic (PLEG): container finished" podID="2a433ebe-7762-4b8a-b605-418518973e05" containerID="761f6a50e07afde03d620fd58fef11dbfe82f6aa21316b86b8f468d793110651" exitCode=1
Sep 09 03:58:21.801308 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.801278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48" event={"ID":"2a433ebe-7762-4b8a-b605-418518973e05","Type":"ContainerStarted","Data":"b9501084a0b28de04365784618eea948e3ce3119683ada607468c3f6321b695a"}
Sep 09 03:58:21.801381 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.801368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48" event={"ID":"2a433ebe-7762-4b8a-b605-418518973e05","Type":"ContainerStarted","Data":"5d708e2bf4eeaf3f6af894aa3b739c0468e5d8183537e06a2df3f957b48167c9"}
Sep 09 03:58:21.801442 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.801412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48" event={"ID":"2a433ebe-7762-4b8a-b605-418518973e05","Type":"ContainerStarted","Data":"4e94c023be7e1a3ec66c7197f4ced5e6372c9b1ef5998116e1e17fd7efb1458e"}
Sep 09 03:58:21.801496 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.801457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48" event={"ID":"2a433ebe-7762-4b8a-b605-418518973e05","Type":"ContainerStarted","Data":"2a516a72d6243dd0bae18ede16adeb84826652db22d505486cd923b911baaf81"}
Sep 09 03:58:21.801541 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.801499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48" event={"ID":"2a433ebe-7762-4b8a-b605-418518973e05","Type":"ContainerDied","Data":"761f6a50e07afde03d620fd58fef11dbfe82f6aa21316b86b8f468d793110651"}
Sep 09 03:58:21.801581 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.801548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48" event={"ID":"2a433ebe-7762-4b8a-b605-418518973e05","Type":"ContainerStarted","Data":"5288fee4ceabda8b84774f7f390434eeed45c269c382e0bfbde64c1e0ab228f5"}
Sep 09 03:58:21.802997 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.802969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb" event={"ID":"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038","Type":"ContainerStarted","Data":"6d63f506c6a837ef2afb5af0da9331d3493bdd95c8a53bdc515095450fa20929"}
Sep 09 03:58:21.804569 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.804548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g5ngq" event={"ID":"1c4180d7-3001-4994-a152-a1a98bd27aa1","Type":"ContainerStarted","Data":"8f4677bf2e188795719d356fd2e6c34badc1b4ba5584759785c8c2f2aebb775b"}
Sep 09 03:58:21.806080 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.806062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6j4fw" event={"ID":"22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3","Type":"ContainerStarted","Data":"58dbb33995497eb230d3b8dfc5de083ffabd0ba05fa5de172fdd3dad12f4ea68"}
Sep 09 03:58:21.807736 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.807680    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa72aab2-7d1d-4c62-83e3-a01afac7ee1c" containerID="379d1cb00571f7fb435af750cdd9956eaa2802e885f5151d0f830d2d7b5fafd3" exitCode=0
Sep 09 03:58:21.807859 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.807760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5fkn" event={"ID":"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c","Type":"ContainerDied","Data":"379d1cb00571f7fb435af750cdd9956eaa2802e885f5151d0f830d2d7b5fafd3"}
Sep 09 03:58:21.809440 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.809329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wjdpl" event={"ID":"cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea","Type":"ContainerStarted","Data":"78c892f562af4ae9f8ea1defb62fe6f52e9a42c83e0837934bd88229960597d7"}
Sep 09 03:58:21.811094 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.811065    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6h7v6" event={"ID":"f88a2c9f-c508-4bab-8a3a-eb3197e93ccb","Type":"ContainerStarted","Data":"eefa111185ceedbe722b728f416101ad4e50aac461f5f8717fb0cb527b1b78c3"}
Sep 09 03:58:21.812613 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.812582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h6bnp" event={"ID":"1537ac59-5afb-4843-aa47-39dbbaf34a47","Type":"ContainerStarted","Data":"c84f4a8827f1b00c795a08bbcb7e3e0fac0b520ac271e17044d1db0fa9f6486d"}
Sep 09 03:58:21.833214 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.833165    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-198.ec2.internal" podStartSLOduration=20.833153126 podStartE2EDuration="20.833153126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:04.779175517 +0000 UTC m=+5.728866172" watchObservedRunningTime="2026-09-09 03:58:21.833153126 +0000 UTC m=+22.782843779"
Sep 09 03:58:21.833548 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.833515    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g5ngq" podStartSLOduration=4.821527624 podStartE2EDuration="22.833505843s" totalImagesPullingTime="18.011978219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:21.832981339 +0000 UTC m=+22.782671993" watchObservedRunningTime="2026-09-09 03:58:21.833505843 +0000 UTC m=+22.783196501"
Sep 09 03:58:21.867410 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.867378    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6h7v6" podStartSLOduration=3.832004243 podStartE2EDuration="21.867365323s" totalImagesPullingTime="18.03536108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:21.848248722 +0000 UTC m=+22.797939376" watchObservedRunningTime="2026-09-09 03:58:21.867365323 +0000 UTC m=+22.817055977"
Sep 09 03:58:21.867695 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.867676    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h6bnp" podStartSLOduration=3.834585907 podStartE2EDuration="21.867670244s" totalImagesPullingTime="18.033084337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:21.866491922 +0000 UTC m=+22.816182576" watchObservedRunningTime="2026-09-09 03:58:21.867670244 +0000 UTC m=+22.817360898"
Sep 09 03:58:21.907724 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.907688    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wjdpl" podStartSLOduration=4.855782789 podStartE2EDuration="22.90767593s" totalImagesPullingTime="18.051893141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:21.907451833 +0000 UTC m=+22.857142489" watchObservedRunningTime="2026-09-09 03:58:21.90767593 +0000 UTC m=+22.857366583"
Sep 09 03:58:21.926998 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.926954    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6j4fw" podStartSLOduration=8.186752596 podStartE2EDuration="21.926938439s" totalImagesPullingTime="13.740185843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:21.926571428 +0000 UTC m=+22.876262082" watchObservedRunningTime="2026-09-09 03:58:21.926938439 +0000 UTC m=+22.876629094"
Sep 09 03:58:21.992761 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:21.992742    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 03:58:22.534797 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:22.534610    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T03:58:21.992757142Z","UUID":"7ad84113-a68a-4b42-a676-37e7586b42e3","Handler":null,"Name":"","Endpoint":""}
Sep 09 03:58:22.536485 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:22.536437    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 03:58:22.536485 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:22.536475    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 03:58:22.817623 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:22.817385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb" event={"ID":"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038","Type":"ContainerStarted","Data":"fffb74d1b2ad8bea0e1fcfdccb9d321db353f2d5ec12d0f24d43e6a5d45bf29e"}
Sep 09 03:58:22.819283 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:22.818998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-55thf" event={"ID":"46cf1ce2-20f8-4818-9e37-83b7a68260b7","Type":"ContainerStarted","Data":"06621fd43adb6d855c5abdd2aa04fcaaf037856a722272dd3740e8b0a597b37c"}
Sep 09 03:58:22.835045 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:22.834997    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-55thf" podStartSLOduration=4.824829034 podStartE2EDuration="22.834978036s" totalImagesPullingTime="18.010149002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:22.834823102 +0000 UTC m=+23.784513758" watchObservedRunningTime="2026-09-09 03:58:22.834978036 +0000 UTC m=+23.784668692"
Sep 09 03:58:23.592975 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:23.592947    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:23.593153 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:23.592986    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:23.593153 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:23.593058    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:23.593153 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:23.593078    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:23.593153 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:23.593126    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:23.593356 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:23.593205    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:23.824367 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:23.824287    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 03:58:23.824829 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:23.824658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48" event={"ID":"2a433ebe-7762-4b8a-b605-418518973e05","Type":"ContainerStarted","Data":"674ca654b21510994b33040081a0e4876cc3e9ef287dd1d4f71682585c4f8e67"}
Sep 09 03:58:23.826696 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:23.826666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb" event={"ID":"f4d0c2ce-88d3-4834-8e8c-e6a125aa4038","Type":"ContainerStarted","Data":"c82b9afaf784d4c30ce8e605ecd652eaf759414b3585f6e66ff13c065dc905f6"}
Sep 09 03:58:23.846603 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:23.846547    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tz5rb" podStartSLOduration=5.213133932 podStartE2EDuration="24.846530301s" totalImagesPullingTime="19.633396369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:57:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:23.84593779 +0000 UTC m=+24.795628445" watchObservedRunningTime="2026-09-09 03:58:23.846530301 +0000 UTC m=+24.796220956"
Sep 09 03:58:25.590292 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:25.589843    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:25.590292 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:25.589843    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:25.590292 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:25.589972    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:25.590292 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:25.589966    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:25.590292 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:25.590075    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:25.590292 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:25.590157    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:25.799539 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:25.799499    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6j4fw"
Sep 09 03:58:25.800472 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:25.800451    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6j4fw"
Sep 09 03:58:25.835158 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:25.835087    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 03:58:25.835833 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:25.835804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48" event={"ID":"2a433ebe-7762-4b8a-b605-418518973e05","Type":"ContainerStarted","Data":"ce7868ead9b753cdaf0b5c434d806496f5024463de3846455de67bfd09694d68"}
Sep 09 03:58:25.836409 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:25.836389    2580 scope.go:117] "RemoveContainer" containerID="761f6a50e07afde03d620fd58fef11dbfe82f6aa21316b86b8f468d793110651"
Sep 09 03:58:25.836525 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:25.836423    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6j4fw"
Sep 09 03:58:25.836573 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:25.836561    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6j4fw"
Sep 09 03:58:27.589447 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.589287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:27.590247 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.589310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:27.590247 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:27.589557    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:27.590247 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.589361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:27.590247 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:27.589601    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:27.590247 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:27.589694    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:27.642052 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.642023    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-brsc7"]
Sep 09 03:58:27.644706 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.644630    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qhb72"]
Sep 09 03:58:27.645554 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.645535    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwl9d"]
Sep 09 03:58:27.842361 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.842291    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5fkn" event={"ID":"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c","Type":"ContainerStarted","Data":"139ecf825f97bdab4fae70a109b5429cd1d18e18077e8fa68205ef8a057b3dca"}
Sep 09 03:58:27.846446 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.846428    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 03:58:27.846968 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.846879    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48" event={"ID":"2a433ebe-7762-4b8a-b605-418518973e05","Type":"ContainerStarted","Data":"c09acc409f532c0fe95f2e0adf575558b0d8d8a4d47638737ec9be938b97cc03"}
Sep 09 03:58:27.846968 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.846932    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:27.847138 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.846898    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:27.847332 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:27.847255    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:27.847451 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.847435    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:27.847737 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.847502    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:27.847737 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.847545    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:27.847737 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:27.847660    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:27.847897 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.847887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:27.847987 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:27.847971    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:27.863966 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.863945    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:27.878849 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.878827    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:58:27.928634 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:27.928593    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48" podStartSLOduration=10.904281735 podStartE2EDuration="28.928579956s" totalImagesPullingTime="18.024298221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:27.895707363 +0000 UTC m=+28.845398014" watchObservedRunningTime="2026-09-09 03:58:27.928579956 +0000 UTC m=+28.878270610"
Sep 09 03:58:28.851138 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:28.851102    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa72aab2-7d1d-4c62-83e3-a01afac7ee1c" containerID="139ecf825f97bdab4fae70a109b5429cd1d18e18077e8fa68205ef8a057b3dca" exitCode=0
Sep 09 03:58:28.851585 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:28.851175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5fkn" event={"ID":"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c","Type":"ContainerDied","Data":"139ecf825f97bdab4fae70a109b5429cd1d18e18077e8fa68205ef8a057b3dca"}
Sep 09 03:58:29.216206 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:29.216127    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:29.590911 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:29.590691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:29.591117 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:29.590811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:29.591208 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:29.590848    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:29.591301 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:29.590987    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:29.591301 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:29.591118    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:29.591301 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:29.591271    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:29.768730 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:29.768700    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:30.858011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:30.857978    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa72aab2-7d1d-4c62-83e3-a01afac7ee1c" containerID="271200ae5084258a41f248dd63d4ef981c518907edab7814a50e941e999246cd" exitCode=0
Sep 09 03:58:30.858483 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:30.858021    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5fkn" event={"ID":"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c","Type":"ContainerDied","Data":"271200ae5084258a41f248dd63d4ef981c518907edab7814a50e941e999246cd"}
Sep 09 03:58:31.589810 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:31.589777    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:31.589990 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:31.589777    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:31.589990 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:31.589874    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:31.589990 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:31.589908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:31.590181 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:31.590016    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:31.590181 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:31.590109    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:32.864054 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:32.864021    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa72aab2-7d1d-4c62-83e3-a01afac7ee1c" containerID="6e734aac02022be20edf3c9bac0887051cadb3fc46013e252266016f1d044e6e" exitCode=0
Sep 09 03:58:32.864469 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:32.864075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5fkn" event={"ID":"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c","Type":"ContainerDied","Data":"6e734aac02022be20edf3c9bac0887051cadb3fc46013e252266016f1d044e6e"}
Sep 09 03:58:33.034503 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.034473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:33.091174 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.091143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:33.329253 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.329179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:33.329253 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.329221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs\") pod \"network-metrics-daemon-xwl9d\" (UID: \"57ca0680-7caf-465f-bedd-44a747cc90af\") " pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:33.329461 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:33.329322    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:33.329461 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:33.329346    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:33.329461 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:33.329374    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs podName:57ca0680-7caf-465f-bedd-44a747cc90af nodeName:}" failed. No retries permitted until 2026-09-09 03:59:05.329359576 +0000 UTC m=+66.279050208 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs") pod "network-metrics-daemon-xwl9d" (UID: "57ca0680-7caf-465f-bedd-44a747cc90af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 03:58:33.329461 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:33.329416    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret podName:93076315-afaf-43ce-a72c-104751d6d4e2 nodeName:}" failed. No retries permitted until 2026-09-09 03:59:05.329388988 +0000 UTC m=+66.279079627 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret") pod "global-pull-secret-syncer-brsc7" (UID: "93076315-afaf-43ce-a72c-104751d6d4e2") : object "kube-system"/"original-pull-secret" not registered
Sep 09 03:58:33.429781 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.429735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5hl\" (UniqueName: \"kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl\") pod \"network-check-target-qhb72\" (UID: \"2988c802-f1a1-4199-9131-ecca43a5f4df\") " pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:33.429924 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:33.429905    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 03:58:33.429969 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:33.429928    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 03:58:33.429969 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:33.429939    2580 projected.go:196] Error preparing data for projected volume kube-api-access-8v5hl for pod openshift-network-diagnostics/network-check-target-qhb72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:33.430043 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:33.429988    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl podName:2988c802-f1a1-4199-9131-ecca43a5f4df nodeName:}" failed. No retries permitted until 2026-09-09 03:59:05.429975638 +0000 UTC m=+66.379666274 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v5hl" (UniqueName: "kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl") pod "network-check-target-qhb72" (UID: "2988c802-f1a1-4199-9131-ecca43a5f4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 03:58:33.589930 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.589901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:33.590097 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.589938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:33.590097 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.589902    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:33.590097 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:33.590020    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhb72" podUID="2988c802-f1a1-4199-9131-ecca43a5f4df"
Sep 09 03:58:33.590097 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:33.590089    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brsc7" podUID="93076315-afaf-43ce-a72c-104751d6d4e2"
Sep 09 03:58:33.590303 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:33.590163    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwl9d" podUID="57ca0680-7caf-465f-bedd-44a747cc90af"
Sep 09 03:58:33.870533 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.870453    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-198.ec2.internal" event="NodeReady"
Sep 09 03:58:33.870968 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.870592    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 03:58:33.904459 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.904429    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d7d794f54-vbjrc"]
Sep 09 03:58:33.927390 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.927361    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vlhhj"]
Sep 09 03:58:33.927540 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.927491    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:33.929890 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.929866    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rtwn6\""
Sep 09 03:58:33.930218 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.930197    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 03:58:33.930442 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.930426    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 03:58:33.931997 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.930470    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 03:58:33.936140 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.936119    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 03:58:33.946715 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.946687    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vhd4r"]
Sep 09 03:58:33.946853 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.946834    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vlhhj"
Sep 09 03:58:33.949915 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.949886    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 03:58:33.950300 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.950050    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 03:58:33.950300 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.950094    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2ccw8\""
Sep 09 03:58:33.950300 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.950111    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 03:58:33.966312 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.966292    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d7d794f54-vbjrc"]
Sep 09 03:58:33.966409 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.966323    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vlhhj"]
Sep 09 03:58:33.966409 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.966339    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vhd4r"]
Sep 09 03:58:33.966409 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.966403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:33.969049 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.969032    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nrdh9\""
Sep 09 03:58:33.969430 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.969411    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 03:58:33.969659 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:33.969646    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 03:58:34.032961 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.032931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-trusted-ca\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.032961 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.032960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmkmv\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-kube-api-access-mmkmv\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.033148 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.033016    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98eb9f21-8704-4f75-b980-19a618f95afc-cert\") pod \"ingress-canary-vlhhj\" (UID: \"98eb9f21-8704-4f75-b980-19a618f95afc\") " pod="openshift-ingress-canary/ingress-canary-vlhhj"
Sep 09 03:58:34.033148 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.033032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.033148 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.033060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khz8n\" (UniqueName: \"kubernetes.io/projected/98eb9f21-8704-4f75-b980-19a618f95afc-kube-api-access-khz8n\") pod \"ingress-canary-vlhhj\" (UID: \"98eb9f21-8704-4f75-b980-19a618f95afc\") " pod="openshift-ingress-canary/ingress-canary-vlhhj"
Sep 09 03:58:34.033148 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.033074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-ca-trust-extracted\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.033389 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.033179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-installation-pull-secrets\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.033389 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.033229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-image-registry-private-configuration\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.033389 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.033278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-certificates\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.033389 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.033321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-bound-sa-token\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.049366 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.049334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:34.105419 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.105382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:34.134456 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmb4x\" (UniqueName: \"kubernetes.io/projected/f850a46e-0840-481a-8f86-0b3877277387-kube-api-access-nmb4x\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:34.134456 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98eb9f21-8704-4f75-b980-19a618f95afc-cert\") pod \"ingress-canary-vlhhj\" (UID: \"98eb9f21-8704-4f75-b980-19a618f95afc\") " pod="openshift-ingress-canary/ingress-canary-vlhhj"
Sep 09 03:58:34.134644 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.134644 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khz8n\" (UniqueName: \"kubernetes.io/projected/98eb9f21-8704-4f75-b980-19a618f95afc-kube-api-access-khz8n\") pod \"ingress-canary-vlhhj\" (UID: \"98eb9f21-8704-4f75-b980-19a618f95afc\") " pod="openshift-ingress-canary/ingress-canary-vlhhj"
Sep 09 03:58:34.134644 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-ca-trust-extracted\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.134644 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-installation-pull-secrets\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.134644 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f850a46e-0840-481a-8f86-0b3877277387-tmp-dir\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:34.134644 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:34.134632    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:58:34.134962 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-image-registry-private-configuration\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.134962 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:34.134656    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d7d794f54-vbjrc: secret "image-registry-tls" not found
Sep 09 03:58:34.134962 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:34.134962 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:34.134721    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls podName:73fd3b9c-8485-41cc-a1cd-0e338d60a49a nodeName:}" failed. No retries permitted until 2026-09-09 03:58:34.634700185 +0000 UTC m=+35.584390831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls") pod "image-registry-d7d794f54-vbjrc" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a") : secret "image-registry-tls" not found
Sep 09 03:58:34.134962 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-certificates\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.134962 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-bound-sa-token\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.134962 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-trusted-ca\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.134962 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmkmv\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-kube-api-access-mmkmv\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.134962 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.134906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f850a46e-0840-481a-8f86-0b3877277387-config-volume\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:34.135362 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.135260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-ca-trust-extracted\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.135708 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.135682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-certificates\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.136193 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.136152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-trusted-ca\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.139846 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.139823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-installation-pull-secrets\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.139846 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.139841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98eb9f21-8704-4f75-b980-19a618f95afc-cert\") pod \"ingress-canary-vlhhj\" (UID: \"98eb9f21-8704-4f75-b980-19a618f95afc\") " pod="openshift-ingress-canary/ingress-canary-vlhhj"
Sep 09 03:58:34.139987 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.139909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-image-registry-private-configuration\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.144682 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.144659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khz8n\" (UniqueName: \"kubernetes.io/projected/98eb9f21-8704-4f75-b980-19a618f95afc-kube-api-access-khz8n\") pod \"ingress-canary-vlhhj\" (UID: \"98eb9f21-8704-4f75-b980-19a618f95afc\") " pod="openshift-ingress-canary/ingress-canary-vlhhj"
Sep 09 03:58:34.145221 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.145190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-bound-sa-token\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.145221 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.145194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmkmv\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-kube-api-access-mmkmv\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.235477 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.235445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f850a46e-0840-481a-8f86-0b3877277387-tmp-dir\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:34.235678 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.235506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:34.235678 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.235551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f850a46e-0840-481a-8f86-0b3877277387-config-volume\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:34.235678 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.235578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmb4x\" (UniqueName: \"kubernetes.io/projected/f850a46e-0840-481a-8f86-0b3877277387-kube-api-access-nmb4x\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:34.235873 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:34.235684    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:58:34.235873 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:34.235757    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls podName:f850a46e-0840-481a-8f86-0b3877277387 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:34.735737081 +0000 UTC m=+35.685427733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls") pod "dns-default-vhd4r" (UID: "f850a46e-0840-481a-8f86-0b3877277387") : secret "dns-default-metrics-tls" not found
Sep 09 03:58:34.235979 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.235870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f850a46e-0840-481a-8f86-0b3877277387-tmp-dir\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:34.236166 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.236144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f850a46e-0840-481a-8f86-0b3877277387-config-volume\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:34.246440 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.246406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmb4x\" (UniqueName: \"kubernetes.io/projected/f850a46e-0840-481a-8f86-0b3877277387-kube-api-access-nmb4x\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:34.257290 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.257272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vlhhj"
Sep 09 03:58:34.417134 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.416911    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:34.473909 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.473873    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:34.587101 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.587060    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vlhhj"]
Sep 09 03:58:34.592031 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:58:34.591999    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98eb9f21_8704_4f75_b980_19a618f95afc.slice/crio-537c002766bef253dee81e1644e162897f0a0afe420fb1a1d512b668c88a7882 WatchSource:0}: Error finding container 537c002766bef253dee81e1644e162897f0a0afe420fb1a1d512b668c88a7882: Status 404 returned error can't find the container with id 537c002766bef253dee81e1644e162897f0a0afe420fb1a1d512b668c88a7882
Sep 09 03:58:34.639903 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.639873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:34.640039 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:34.640013    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:58:34.640039 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:34.640035    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d7d794f54-vbjrc: secret "image-registry-tls" not found
Sep 09 03:58:34.640130 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:34.640105    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls podName:73fd3b9c-8485-41cc-a1cd-0e338d60a49a nodeName:}" failed. No retries permitted until 2026-09-09 03:58:35.640082595 +0000 UTC m=+36.589773232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls") pod "image-registry-d7d794f54-vbjrc" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a") : secret "image-registry-tls" not found
Sep 09 03:58:34.741301 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.741214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:34.741457 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:34.741370    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:58:34.741457 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:34.741434    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls podName:f850a46e-0840-481a-8f86-0b3877277387 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:35.741417716 +0000 UTC m=+36.691108348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls") pod "dns-default-vhd4r" (UID: "f850a46e-0840-481a-8f86-0b3877277387") : secret "dns-default-metrics-tls" not found
Sep 09 03:58:34.788606 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.788574    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:34.850697 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.850670    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:34.869412 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:34.869377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vlhhj" event={"ID":"98eb9f21-8704-4f75-b980-19a618f95afc","Type":"ContainerStarted","Data":"537c002766bef253dee81e1644e162897f0a0afe420fb1a1d512b668c88a7882"}
Sep 09 03:58:35.589204 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:35.589176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:58:35.589919 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:35.589176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:58:35.589919 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:35.589176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:58:35.595279 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:35.595255    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 03:58:35.595423 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:35.595370    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 03:58:35.595423 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:35.595377    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 03:58:35.596706 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:35.596685    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 03:58:35.596834 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:35.596806    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-krf5h\""
Sep 09 03:58:35.596920 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:35.596904    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-slrsq\""
Sep 09 03:58:35.650349 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:35.650326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:35.650495 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:35.650462    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:58:35.650495 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:35.650478    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d7d794f54-vbjrc: secret "image-registry-tls" not found
Sep 09 03:58:35.650602 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:35.650550    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls podName:73fd3b9c-8485-41cc-a1cd-0e338d60a49a nodeName:}" failed. No retries permitted until 2026-09-09 03:58:37.650529755 +0000 UTC m=+38.600220413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls") pod "image-registry-d7d794f54-vbjrc" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a") : secret "image-registry-tls" not found
Sep 09 03:58:35.751476 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:35.751346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:35.751476 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:35.751473    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:58:35.751689 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:35.751548    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls podName:f850a46e-0840-481a-8f86-0b3877277387 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:37.75152826 +0000 UTC m=+38.701218915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls") pod "dns-default-vhd4r" (UID: "f850a46e-0840-481a-8f86-0b3877277387") : secret "dns-default-metrics-tls" not found
Sep 09 03:58:37.668129 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:37.668099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:37.668649 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:37.668289    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:58:37.668649 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:37.668308    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d7d794f54-vbjrc: secret "image-registry-tls" not found
Sep 09 03:58:37.668649 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:37.668369    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls podName:73fd3b9c-8485-41cc-a1cd-0e338d60a49a nodeName:}" failed. No retries permitted until 2026-09-09 03:58:41.668348457 +0000 UTC m=+42.618039088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls") pod "image-registry-d7d794f54-vbjrc" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a") : secret "image-registry-tls" not found
Sep 09 03:58:37.768994 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:37.768956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:37.769167 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:37.769107    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:58:37.769235 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:37.769185    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls podName:f850a46e-0840-481a-8f86-0b3877277387 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:41.769161923 +0000 UTC m=+42.718852556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls") pod "dns-default-vhd4r" (UID: "f850a46e-0840-481a-8f86-0b3877277387") : secret "dns-default-metrics-tls" not found
Sep 09 03:58:38.883528 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:38.883329    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa72aab2-7d1d-4c62-83e3-a01afac7ee1c" containerID="1ddb88ff242af9d4b863d9dc097c00a890ee7cb98a9ae54003b7359db564a466" exitCode=0
Sep 09 03:58:38.884073 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:38.883407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5fkn" event={"ID":"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c","Type":"ContainerDied","Data":"1ddb88ff242af9d4b863d9dc097c00a890ee7cb98a9ae54003b7359db564a466"}
Sep 09 03:58:38.885185 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:38.885159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vlhhj" event={"ID":"98eb9f21-8704-4f75-b980-19a618f95afc","Type":"ContainerStarted","Data":"fd5f9c8ca604dd1a4dbd9449b3b36384d4025d47d618908520f158e950e28283"}
Sep 09 03:58:38.926613 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:38.926565    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vlhhj" podStartSLOduration=2.292304945 podStartE2EDuration="5.926551326s" totalImagesPullingTime="3.634246381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:38.925981871 +0000 UTC m=+39.875672528" watchObservedRunningTime="2026-09-09 03:58:38.926551326 +0000 UTC m=+39.876241971"
Sep 09 03:58:39.361408 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:39.361381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:39.416461 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:39.416427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:39.889894 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:39.889865    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa72aab2-7d1d-4c62-83e3-a01afac7ee1c" containerID="86ec065b58557edc3dc5b4ee058531299b654b65831b570df5a399c4f6dd5745" exitCode=0
Sep 09 03:58:39.890291 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:39.889938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5fkn" event={"ID":"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c","Type":"ContainerDied","Data":"86ec065b58557edc3dc5b4ee058531299b654b65831b570df5a399c4f6dd5745"}
Sep 09 03:58:40.895656 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:40.895622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x5fkn" event={"ID":"aa72aab2-7d1d-4c62-83e3-a01afac7ee1c","Type":"ContainerStarted","Data":"683d3aece404f32f399085ad93d849d8ba49e78e447a1d09d47c0e0f58f14797"}
Sep 09 03:58:40.923036 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:40.922990    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x5fkn" podStartSLOduration=10.54205677 podStartE2EDuration="41.922976264s" totalImagesPullingTime="31.380919494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:57:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:40.920957851 +0000 UTC m=+41.870648504" watchObservedRunningTime="2026-09-09 03:58:40.922976264 +0000 UTC m=+41.872666918"
Sep 09 03:58:41.046688 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:41.046643    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:41.146663 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:41.146584    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:41.203717 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:41.203688    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:41.697687 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:41.697634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:41.697910 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:41.697806    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:58:41.697910 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:41.697823    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d7d794f54-vbjrc: secret "image-registry-tls" not found
Sep 09 03:58:41.697910 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:41.697877    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls podName:73fd3b9c-8485-41cc-a1cd-0e338d60a49a nodeName:}" failed. No retries permitted until 2026-09-09 03:58:49.697863716 +0000 UTC m=+50.647554348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls") pod "image-registry-d7d794f54-vbjrc" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a") : secret "image-registry-tls" not found
Sep 09 03:58:41.798913 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:41.798884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:41.799076 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:41.799055    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:58:41.799150 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:41.799138    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls podName:f850a46e-0840-481a-8f86-0b3877277387 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:49.799115391 +0000 UTC m=+50.748806037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls") pod "dns-default-vhd4r" (UID: "f850a46e-0840-481a-8f86-0b3877277387") : secret "dns-default-metrics-tls" not found
Sep 09 03:58:43.117964 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:43.117926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:43.182682 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:43.182654    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log" limit=1024
Sep 09 03:58:44.220018 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.219982    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ngtdv"]
Sep 09 03:58:44.226364 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.226346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.229388 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.229367    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 03:58:44.229685 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.229672    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 03:58:44.230804 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.230784    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4kqm2\""
Sep 09 03:58:44.230892 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.230799    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 03:58:44.231362 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.231349    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 03:58:44.281646 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.281608    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ngtdv"]
Sep 09 03:58:44.318466 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.318428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a18afeda-adec-4705-b816-f8bdac47e629-crio-socket\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.318466 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.318467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl8ns\" (UniqueName: \"kubernetes.io/projected/a18afeda-adec-4705-b816-f8bdac47e629-kube-api-access-tl8ns\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.318700 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.318527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a18afeda-adec-4705-b816-f8bdac47e629-data-volume\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.318700 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.318566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.318700 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.318619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a18afeda-adec-4705-b816-f8bdac47e629-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.418907 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.418878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a18afeda-adec-4705-b816-f8bdac47e629-crio-socket\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.419023 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.418915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl8ns\" (UniqueName: \"kubernetes.io/projected/a18afeda-adec-4705-b816-f8bdac47e629-kube-api-access-tl8ns\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.419087 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.419067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a18afeda-adec-4705-b816-f8bdac47e629-data-volume\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.419122 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.419092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a18afeda-adec-4705-b816-f8bdac47e629-crio-socket\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.419165 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.419123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.419203 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.419166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a18afeda-adec-4705-b816-f8bdac47e629-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.419258 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:44.419245    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:58:44.419370 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:44.419268    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:58:44.419370 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:44.419295    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls podName:a18afeda-adec-4705-b816-f8bdac47e629 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:44.91927958 +0000 UTC m=+45.868970212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngtdv" (UID: "a18afeda-adec-4705-b816-f8bdac47e629") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:58:44.419370 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:44.419313    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a18afeda-adec-4705-b816-f8bdac47e629-kube-rbac-proxy-cm podName:a18afeda-adec-4705-b816-f8bdac47e629 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:44.919302341 +0000 UTC m=+45.868992994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a18afeda-adec-4705-b816-f8bdac47e629-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ngtdv" (UID: "a18afeda-adec-4705-b816-f8bdac47e629") : configmap "kube-rbac-proxy" not found
Sep 09 03:58:44.419522 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.419407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a18afeda-adec-4705-b816-f8bdac47e629-data-volume\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.429077 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.429056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl8ns\" (UniqueName: \"kubernetes.io/projected/a18afeda-adec-4705-b816-f8bdac47e629-kube-api-access-tl8ns\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.922282 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.922253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a18afeda-adec-4705-b816-f8bdac47e629-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.922471 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:44.922344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:44.922471 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:44.922378    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 03:58:44.922471 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:44.922430    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:58:44.922471 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:44.922442    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a18afeda-adec-4705-b816-f8bdac47e629-kube-rbac-proxy-cm podName:a18afeda-adec-4705-b816-f8bdac47e629 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:45.922426676 +0000 UTC m=+46.872117311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a18afeda-adec-4705-b816-f8bdac47e629-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ngtdv" (UID: "a18afeda-adec-4705-b816-f8bdac47e629") : configmap "kube-rbac-proxy" not found
Sep 09 03:58:44.922471 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:44.922469    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls podName:a18afeda-adec-4705-b816-f8bdac47e629 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:45.922459038 +0000 UTC m=+46.872149670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngtdv" (UID: "a18afeda-adec-4705-b816-f8bdac47e629") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:58:45.517809 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.517750    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s9k24"]
Sep 09 03:58:45.520609 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.520595    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s9k24"
Sep 09 03:58:45.523461 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.523439    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vmtjm\""
Sep 09 03:58:45.523556 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.523487    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 03:58:45.524324 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.524310    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 03:58:45.526233 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.526212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndkpm\" (UniqueName: \"kubernetes.io/projected/e45776a2-d18f-4bc6-8b63-fa62745dc1a1-kube-api-access-ndkpm\") pod \"migrator-fdbb8fdff-s9k24\" (UID: \"e45776a2-d18f-4bc6-8b63-fa62745dc1a1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s9k24"
Sep 09 03:58:45.529445 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.529425    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s9k24"]
Sep 09 03:58:45.626541 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.626512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndkpm\" (UniqueName: \"kubernetes.io/projected/e45776a2-d18f-4bc6-8b63-fa62745dc1a1-kube-api-access-ndkpm\") pod \"migrator-fdbb8fdff-s9k24\" (UID: \"e45776a2-d18f-4bc6-8b63-fa62745dc1a1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s9k24"
Sep 09 03:58:45.635266 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.635245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndkpm\" (UniqueName: \"kubernetes.io/projected/e45776a2-d18f-4bc6-8b63-fa62745dc1a1-kube-api-access-ndkpm\") pod \"migrator-fdbb8fdff-s9k24\" (UID: \"e45776a2-d18f-4bc6-8b63-fa62745dc1a1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s9k24"
Sep 09 03:58:45.761601 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.761575    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h6bnp_1537ac59-5afb-4843-aa47-39dbbaf34a47/dns-node-resolver/0.log"
Sep 09 03:58:45.829359 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.829303    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s9k24"
Sep 09 03:58:45.929292 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.929259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:45.929429 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.929299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a18afeda-adec-4705-b816-f8bdac47e629-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:45.929429 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:45.929415    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:58:45.929503 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:45.929492    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls podName:a18afeda-adec-4705-b816-f8bdac47e629 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:47.929471803 +0000 UTC m=+48.879162434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngtdv" (UID: "a18afeda-adec-4705-b816-f8bdac47e629") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:58:45.929891 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:45.929873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a18afeda-adec-4705-b816-f8bdac47e629-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:46.096784 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.096738    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s9k24"]
Sep 09 03:58:46.099669 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:58:46.099648    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45776a2_d18f_4bc6_8b63_fa62745dc1a1.slice/crio-541d8db5e1ed4e6642e6bac3dc3c41663c849b75860dd21c7f0032b59a21e68f WatchSource:0}: Error finding container 541d8db5e1ed4e6642e6bac3dc3c41663c849b75860dd21c7f0032b59a21e68f: Status 404 returned error can't find the container with id 541d8db5e1ed4e6642e6bac3dc3c41663c849b75860dd21c7f0032b59a21e68f
Sep 09 03:58:46.431132 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.431052    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-tfn7q"]
Sep 09 03:58:46.434756 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.434739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q"
Sep 09 03:58:46.438681 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.438654    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 03:58:46.438813 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.438700    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xfst6\""
Sep 09 03:58:46.439564 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.439549    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 03:58:46.444834 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.444811    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-tfn7q"]
Sep 09 03:58:46.534128 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.534093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1fa279b0-1e9c-454d-b0e8-24d193ca3476-nginx-conf\") pod \"networking-console-plugin-79977f9844-tfn7q\" (UID: \"1fa279b0-1e9c-454d-b0e8-24d193ca3476\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q"
Sep 09 03:58:46.534465 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.534143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-tfn7q\" (UID: \"1fa279b0-1e9c-454d-b0e8-24d193ca3476\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q"
Sep 09 03:58:46.571273 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.571251    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6h7v6_f88a2c9f-c508-4bab-8a3a-eb3197e93ccb/node-ca/0.log"
Sep 09 03:58:46.635404 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.635368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1fa279b0-1e9c-454d-b0e8-24d193ca3476-nginx-conf\") pod \"networking-console-plugin-79977f9844-tfn7q\" (UID: \"1fa279b0-1e9c-454d-b0e8-24d193ca3476\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q"
Sep 09 03:58:46.635568 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.635433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-tfn7q\" (UID: \"1fa279b0-1e9c-454d-b0e8-24d193ca3476\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q"
Sep 09 03:58:46.635636 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:46.635602    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:58:46.635694 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:46.635677    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert podName:1fa279b0-1e9c-454d-b0e8-24d193ca3476 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:47.13565919 +0000 UTC m=+48.085349828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-tfn7q" (UID: "1fa279b0-1e9c-454d-b0e8-24d193ca3476") : secret "networking-console-plugin-cert" not found
Sep 09 03:58:46.636100 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.636077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1fa279b0-1e9c-454d-b0e8-24d193ca3476-nginx-conf\") pod \"networking-console-plugin-79977f9844-tfn7q\" (UID: \"1fa279b0-1e9c-454d-b0e8-24d193ca3476\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q"
Sep 09 03:58:46.911700 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:46.911665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s9k24" event={"ID":"e45776a2-d18f-4bc6-8b63-fa62745dc1a1","Type":"ContainerStarted","Data":"541d8db5e1ed4e6642e6bac3dc3c41663c849b75860dd21c7f0032b59a21e68f"}
Sep 09 03:58:47.138730 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.138706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-tfn7q\" (UID: \"1fa279b0-1e9c-454d-b0e8-24d193ca3476\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q"
Sep 09 03:58:47.138885 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:47.138868    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:58:47.138956 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:47.138918    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert podName:1fa279b0-1e9c-454d-b0e8-24d193ca3476 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:48.138905153 +0000 UTC m=+49.088595785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-tfn7q" (UID: "1fa279b0-1e9c-454d-b0e8-24d193ca3476") : secret "networking-console-plugin-cert" not found
Sep 09 03:58:47.393525 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.393488    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-79sft"]
Sep 09 03:58:47.396525 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.396505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-79sft"
Sep 09 03:58:47.399069 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.399042    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 03:58:47.399207 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.399048    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 03:58:47.400299 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.400274    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9cshh\""
Sep 09 03:58:47.400574 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.400280    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 03:58:47.400574 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.400337    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 03:58:47.404870 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.404850    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-79sft"]
Sep 09 03:58:47.440953 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.440886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58d03053-167f-4131-8a7b-cb14cf07901c-signing-cabundle\") pod \"service-ca-75d99c484-79sft\" (UID: \"58d03053-167f-4131-8a7b-cb14cf07901c\") " pod="openshift-service-ca/service-ca-75d99c484-79sft"
Sep 09 03:58:47.440953 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.440938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls8vh\" (UniqueName: \"kubernetes.io/projected/58d03053-167f-4131-8a7b-cb14cf07901c-kube-api-access-ls8vh\") pod \"service-ca-75d99c484-79sft\" (UID: \"58d03053-167f-4131-8a7b-cb14cf07901c\") " pod="openshift-service-ca/service-ca-75d99c484-79sft"
Sep 09 03:58:47.441096 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.441022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58d03053-167f-4131-8a7b-cb14cf07901c-signing-key\") pod \"service-ca-75d99c484-79sft\" (UID: \"58d03053-167f-4131-8a7b-cb14cf07901c\") " pod="openshift-service-ca/service-ca-75d99c484-79sft"
Sep 09 03:58:47.542235 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.542195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls8vh\" (UniqueName: \"kubernetes.io/projected/58d03053-167f-4131-8a7b-cb14cf07901c-kube-api-access-ls8vh\") pod \"service-ca-75d99c484-79sft\" (UID: \"58d03053-167f-4131-8a7b-cb14cf07901c\") " pod="openshift-service-ca/service-ca-75d99c484-79sft"
Sep 09 03:58:47.542649 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.542256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58d03053-167f-4131-8a7b-cb14cf07901c-signing-key\") pod \"service-ca-75d99c484-79sft\" (UID: \"58d03053-167f-4131-8a7b-cb14cf07901c\") " pod="openshift-service-ca/service-ca-75d99c484-79sft"
Sep 09 03:58:47.542649 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.542306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58d03053-167f-4131-8a7b-cb14cf07901c-signing-cabundle\") pod \"service-ca-75d99c484-79sft\" (UID: \"58d03053-167f-4131-8a7b-cb14cf07901c\") " pod="openshift-service-ca/service-ca-75d99c484-79sft"
Sep 09 03:58:47.542943 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.542924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58d03053-167f-4131-8a7b-cb14cf07901c-signing-cabundle\") pod \"service-ca-75d99c484-79sft\" (UID: \"58d03053-167f-4131-8a7b-cb14cf07901c\") " pod="openshift-service-ca/service-ca-75d99c484-79sft"
Sep 09 03:58:47.544862 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.544843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58d03053-167f-4131-8a7b-cb14cf07901c-signing-key\") pod \"service-ca-75d99c484-79sft\" (UID: \"58d03053-167f-4131-8a7b-cb14cf07901c\") " pod="openshift-service-ca/service-ca-75d99c484-79sft"
Sep 09 03:58:47.551161 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.551143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls8vh\" (UniqueName: \"kubernetes.io/projected/58d03053-167f-4131-8a7b-cb14cf07901c-kube-api-access-ls8vh\") pod \"service-ca-75d99c484-79sft\" (UID: \"58d03053-167f-4131-8a7b-cb14cf07901c\") " pod="openshift-service-ca/service-ca-75d99c484-79sft"
Sep 09 03:58:47.561517 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.561501    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vlhhj_98eb9f21-8704-4f75-b980-19a618f95afc/serve-healthcheck-canary/0.log"
Sep 09 03:58:47.706128 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.706066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-79sft"
Sep 09 03:58:47.915674 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.915638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s9k24" event={"ID":"e45776a2-d18f-4bc6-8b63-fa62745dc1a1","Type":"ContainerStarted","Data":"63671e5b06abd3b2be673e740c9a72f6e8397d66e6699a69809ef67503355f4d"}
Sep 09 03:58:47.915849 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.915713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s9k24" event={"ID":"e45776a2-d18f-4bc6-8b63-fa62745dc1a1","Type":"ContainerStarted","Data":"460d38495d576e02f67da0f13cae8c20cd12ab8e6fe99f9385133d557e4bd2b2"}
Sep 09 03:58:47.933309 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.933264    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-s9k24" podStartSLOduration=1.929468505 podStartE2EDuration="2.93324941s" totalImagesPullingTime="1.003780905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:47.933200545 +0000 UTC m=+48.882891198" watchObservedRunningTime="2026-09-09 03:58:47.93324941 +0000 UTC m=+48.882940058"
Sep 09 03:58:47.947614 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.947589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:47.947812 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:47.947792    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:58:47.947867 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:47.947840    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls podName:a18afeda-adec-4705-b816-f8bdac47e629 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:51.947827693 +0000 UTC m=+52.897518325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngtdv" (UID: "a18afeda-adec-4705-b816-f8bdac47e629") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:58:47.975914 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:47.975859    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-79sft"]
Sep 09 03:58:47.979048 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:58:47.979026    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d03053_167f_4131_8a7b_cb14cf07901c.slice/crio-e48c682903a342c590bff62d070e0a3d87b2e0a2558080439de0070eeb3162b0 WatchSource:0}: Error finding container e48c682903a342c590bff62d070e0a3d87b2e0a2558080439de0070eeb3162b0: Status 404 returned error can't find the container with id e48c682903a342c590bff62d070e0a3d87b2e0a2558080439de0070eeb3162b0
Sep 09 03:58:48.149843 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:48.149808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-tfn7q\" (UID: \"1fa279b0-1e9c-454d-b0e8-24d193ca3476\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q"
Sep 09 03:58:48.150015 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:48.150001    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:58:48.150097 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:48.150084    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert podName:1fa279b0-1e9c-454d-b0e8-24d193ca3476 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:50.150062701 +0000 UTC m=+51.099753336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-tfn7q" (UID: "1fa279b0-1e9c-454d-b0e8-24d193ca3476") : secret "networking-console-plugin-cert" not found
Sep 09 03:58:48.919738 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:48.919703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-79sft" event={"ID":"58d03053-167f-4131-8a7b-cb14cf07901c","Type":"ContainerStarted","Data":"e48c682903a342c590bff62d070e0a3d87b2e0a2558080439de0070eeb3162b0"}
Sep 09 03:58:49.763629 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:49.763605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:58:49.763825 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:49.763805    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 03:58:49.763887 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:49.763830    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d7d794f54-vbjrc: secret "image-registry-tls" not found
Sep 09 03:58:49.763921 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:49.763897    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls podName:73fd3b9c-8485-41cc-a1cd-0e338d60a49a nodeName:}" failed. No retries permitted until 2026-09-09 03:59:05.763874314 +0000 UTC m=+66.713564966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls") pod "image-registry-d7d794f54-vbjrc" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a") : secret "image-registry-tls" not found
Sep 09 03:58:49.864842 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:49.864806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:58:49.864983 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:49.864949    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 03:58:49.865024 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:49.865017    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls podName:f850a46e-0840-481a-8f86-0b3877277387 nodeName:}" failed. No retries permitted until 2026-09-09 03:59:05.86499983 +0000 UTC m=+66.814690462 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls") pod "dns-default-vhd4r" (UID: "f850a46e-0840-481a-8f86-0b3877277387") : secret "dns-default-metrics-tls" not found
Sep 09 03:58:49.924306 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:49.924273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-79sft" event={"ID":"58d03053-167f-4131-8a7b-cb14cf07901c","Type":"ContainerStarted","Data":"faf48d498fab0e234226eb596c476d9b0f1ff05efae58171070e97a0a748ca83"}
Sep 09 03:58:49.969782 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:49.969724    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-79sft" podStartSLOduration=1.201768486 podStartE2EDuration="2.969709808s" totalImagesPullingTime="1.767941322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:58:49.969279337 +0000 UTC m=+50.918969994" watchObservedRunningTime="2026-09-09 03:58:49.969709808 +0000 UTC m=+50.919400462"
Sep 09 03:58:50.166702 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:50.166666    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-tfn7q\" (UID: \"1fa279b0-1e9c-454d-b0e8-24d193ca3476\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q"
Sep 09 03:58:50.166869 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:50.166830    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:58:50.166908 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:50.166903    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert podName:1fa279b0-1e9c-454d-b0e8-24d193ca3476 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:54.16687995 +0000 UTC m=+55.116570582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-tfn7q" (UID: "1fa279b0-1e9c-454d-b0e8-24d193ca3476") : secret "networking-console-plugin-cert" not found
Sep 09 03:58:51.980940 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:51.980900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:58:51.981591 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:51.981088    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 03:58:51.981591 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:51.981146    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls podName:a18afeda-adec-4705-b816-f8bdac47e629 nodeName:}" failed. No retries permitted until 2026-09-09 03:58:59.981129152 +0000 UTC m=+60.930819803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ngtdv" (UID: "a18afeda-adec-4705-b816-f8bdac47e629") : secret "insights-runtime-extractor-tls" not found
Sep 09 03:58:54.199393 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:54.199343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-tfn7q\" (UID: \"1fa279b0-1e9c-454d-b0e8-24d193ca3476\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q"
Sep 09 03:58:54.199794 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:54.199495    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 03:58:54.199794 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:58:54.199561    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert podName:1fa279b0-1e9c-454d-b0e8-24d193ca3476 nodeName:}" failed. No retries permitted until 2026-09-09 03:59:02.19954533 +0000 UTC m=+63.149235966 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-tfn7q" (UID: "1fa279b0-1e9c-454d-b0e8-24d193ca3476") : secret "networking-console-plugin-cert" not found
Sep 09 03:58:59.874049 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:58:59.874019    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6f48"
Sep 09 03:59:00.045965 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:00.045934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:59:00.048558 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:00.048534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a18afeda-adec-4705-b816-f8bdac47e629-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngtdv\" (UID: \"a18afeda-adec-4705-b816-f8bdac47e629\") " pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:59:00.139414 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:00.139328    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4kqm2\""
Sep 09 03:59:00.145898 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:00.145876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ngtdv"
Sep 09 03:59:00.448701 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:00.448624    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ngtdv"]
Sep 09 03:59:00.453169 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:59:00.453142    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18afeda_adec_4705_b816_f8bdac47e629.slice/crio-acd0a06d6ee8690598ff9f70a3a12d9dd2eb78217a5a8e64401ca3a4e21cc54f WatchSource:0}: Error finding container acd0a06d6ee8690598ff9f70a3a12d9dd2eb78217a5a8e64401ca3a4e21cc54f: Status 404 returned error can't find the container with id acd0a06d6ee8690598ff9f70a3a12d9dd2eb78217a5a8e64401ca3a4e21cc54f
Sep 09 03:59:00.956117 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:00.956086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngtdv" event={"ID":"a18afeda-adec-4705-b816-f8bdac47e629","Type":"ContainerStarted","Data":"a3bfebb75db8e681a1266e2daa22b22b65954d372f68456120a63c2f082c2a08"}
Sep 09 03:59:00.956480 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:00.956160    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngtdv" event={"ID":"a18afeda-adec-4705-b816-f8bdac47e629","Type":"ContainerStarted","Data":"acd0a06d6ee8690598ff9f70a3a12d9dd2eb78217a5a8e64401ca3a4e21cc54f"}
Sep 09 03:59:01.960312 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:01.960231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngtdv" event={"ID":"a18afeda-adec-4705-b816-f8bdac47e629","Type":"ContainerStarted","Data":"398000e0495ac0dc0fd922c0404c20d5367928565ebb935d5de3e6b232454219"}
Sep 09 03:59:02.263204 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:02.263122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-tfn7q\" (UID: \"1fa279b0-1e9c-454d-b0e8-24d193ca3476\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q"
Sep 09 03:59:02.265585 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:02.265567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1fa279b0-1e9c-454d-b0e8-24d193ca3476-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-tfn7q\" (UID: \"1fa279b0-1e9c-454d-b0e8-24d193ca3476\") " pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q"
Sep 09 03:59:02.346450 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:02.346420    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xfst6\""
Sep 09 03:59:02.353918 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:02.353899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q"
Sep 09 03:59:02.645142 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:02.645115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-tfn7q"]
Sep 09 03:59:02.647716 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:59:02.647686    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa279b0_1e9c_454d_b0e8_24d193ca3476.slice/crio-69c2b67f9c2025906baeae042b5a14ba32f59560ab72683b1b1bc50b25483b67 WatchSource:0}: Error finding container 69c2b67f9c2025906baeae042b5a14ba32f59560ab72683b1b1bc50b25483b67: Status 404 returned error can't find the container with id 69c2b67f9c2025906baeae042b5a14ba32f59560ab72683b1b1bc50b25483b67
Sep 09 03:59:02.967725 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:02.967643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q" event={"ID":"1fa279b0-1e9c-454d-b0e8-24d193ca3476","Type":"ContainerStarted","Data":"69c2b67f9c2025906baeae042b5a14ba32f59560ab72683b1b1bc50b25483b67"}
Sep 09 03:59:03.972644 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:03.972612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngtdv" event={"ID":"a18afeda-adec-4705-b816-f8bdac47e629","Type":"ContainerStarted","Data":"ec734fab2e1019adceaf4b8dd0e4c55276457d0d5b0b38224a5aefbaa0ca69b7"}
Sep 09 03:59:03.997831 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:03.997543    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ngtdv" podStartSLOduration=17.409553998 podStartE2EDuration="19.997523653s" totalImagesPullingTime="2.587969655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:03.996372613 +0000 UTC m=+64.946063268" watchObservedRunningTime="2026-09-09 03:59:03.997523653 +0000 UTC m=+64.947214308"
Sep 09 03:59:04.976942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:04.976904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q" event={"ID":"1fa279b0-1e9c-454d-b0e8-24d193ca3476","Type":"ContainerStarted","Data":"630cd0d97055635ba33cf6a04680a2e5b387c60224ddbc6ca773262a8a4a53de"}
Sep 09 03:59:04.993528 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:04.993482    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-tfn7q" podStartSLOduration=17.667476342 podStartE2EDuration="18.993465775s" totalImagesPullingTime="1.325989433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:04.992853213 +0000 UTC m=+65.942543905" watchObservedRunningTime="2026-09-09 03:59:04.993465775 +0000 UTC m=+65.943156428"
Sep 09 03:59:05.388344 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.388310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:59:05.388517 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.388397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs\") pod \"network-metrics-daemon-xwl9d\" (UID: \"57ca0680-7caf-465f-bedd-44a747cc90af\") " pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:59:05.391500 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.391481    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 03:59:05.392330 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.392305    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 03:59:05.400838 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.400820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ca0680-7caf-465f-bedd-44a747cc90af-metrics-certs\") pod \"network-metrics-daemon-xwl9d\" (UID: \"57ca0680-7caf-465f-bedd-44a747cc90af\") " pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:59:05.400932 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.400885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93076315-afaf-43ce-a72c-104751d6d4e2-original-pull-secret\") pod \"global-pull-secret-syncer-brsc7\" (UID: \"93076315-afaf-43ce-a72c-104751d6d4e2\") " pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:59:05.489566 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.489541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5hl\" (UniqueName: \"kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl\") pod \"network-check-target-qhb72\" (UID: \"2988c802-f1a1-4199-9131-ecca43a5f4df\") " pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:59:05.492749 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.492726    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 03:59:05.503030 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.503009    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 03:59:05.512810 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.512785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v5hl\" (UniqueName: \"kubernetes.io/projected/2988c802-f1a1-4199-9131-ecca43a5f4df-kube-api-access-8v5hl\") pod \"network-check-target-qhb72\" (UID: \"2988c802-f1a1-4199-9131-ecca43a5f4df\") " pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:59:05.604812 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.604784    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-slrsq\""
Sep 09 03:59:05.610136 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.610123    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brsc7"
Sep 09 03:59:05.612253 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.612238    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:59:05.618709 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.618690    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-krf5h\""
Sep 09 03:59:05.626319 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.626295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwl9d"
Sep 09 03:59:05.792251 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.792216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:59:05.794727 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.794705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls\") pod \"image-registry-d7d794f54-vbjrc\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") " pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:59:05.889879 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.889849    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-brsc7"]
Sep 09 03:59:05.891839 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.891816    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qhb72"]
Sep 09 03:59:05.892957 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.892937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:59:05.893252 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:59:05.893218    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93076315_afaf_43ce_a72c_104751d6d4e2.slice/crio-eb380d0dda7a97c75cc4e221b0ecdbd85817e21a7e995a718b9bd435fddb359a WatchSource:0}: Error finding container eb380d0dda7a97c75cc4e221b0ecdbd85817e21a7e995a718b9bd435fddb359a: Status 404 returned error can't find the container with id eb380d0dda7a97c75cc4e221b0ecdbd85817e21a7e995a718b9bd435fddb359a
Sep 09 03:59:05.894745 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:59:05.894725    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2988c802_f1a1_4199_9131_ecca43a5f4df.slice/crio-640c6ed0b5b3bf4bb45109cab8e9bd29a6cc08341bd0fc2643570f8263ac49d4 WatchSource:0}: Error finding container 640c6ed0b5b3bf4bb45109cab8e9bd29a6cc08341bd0fc2643570f8263ac49d4: Status 404 returned error can't find the container with id 640c6ed0b5b3bf4bb45109cab8e9bd29a6cc08341bd0fc2643570f8263ac49d4
Sep 09 03:59:05.895736 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.895684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f850a46e-0840-481a-8f86-0b3877277387-metrics-tls\") pod \"dns-default-vhd4r\" (UID: \"f850a46e-0840-481a-8f86-0b3877277387\") " pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:59:05.916631 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.916608    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwl9d"]
Sep 09 03:59:05.919477 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:59:05.919458    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ca0680_7caf_465f_bedd_44a747cc90af.slice/crio-85505d0888b9a80dd381e75f45924dd81aca100f29184d9efb6e9b493bcbda72 WatchSource:0}: Error finding container 85505d0888b9a80dd381e75f45924dd81aca100f29184d9efb6e9b493bcbda72: Status 404 returned error can't find the container with id 85505d0888b9a80dd381e75f45924dd81aca100f29184d9efb6e9b493bcbda72
Sep 09 03:59:05.980609 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.980581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwl9d" event={"ID":"57ca0680-7caf-465f-bedd-44a747cc90af","Type":"ContainerStarted","Data":"85505d0888b9a80dd381e75f45924dd81aca100f29184d9efb6e9b493bcbda72"}
Sep 09 03:59:05.981784 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.981738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qhb72" event={"ID":"2988c802-f1a1-4199-9131-ecca43a5f4df","Type":"ContainerStarted","Data":"640c6ed0b5b3bf4bb45109cab8e9bd29a6cc08341bd0fc2643570f8263ac49d4"}
Sep 09 03:59:05.982876 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:05.982851    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-brsc7" event={"ID":"93076315-afaf-43ce-a72c-104751d6d4e2","Type":"ContainerStarted","Data":"eb380d0dda7a97c75cc4e221b0ecdbd85817e21a7e995a718b9bd435fddb359a"}
Sep 09 03:59:06.043326 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:06.043299    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rtwn6\""
Sep 09 03:59:06.050641 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:06.050623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:59:06.079705 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:06.079685    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nrdh9\""
Sep 09 03:59:06.086696 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:06.086674    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:59:06.326218 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:06.326098    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d7d794f54-vbjrc"]
Sep 09 03:59:06.331346 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:59:06.331315    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fd3b9c_8485_41cc_a1cd_0e338d60a49a.slice/crio-62560b102b2030e9d1c9dab7ee8a7fa8503a8097f05bc15c00410ad9a724fb6e WatchSource:0}: Error finding container 62560b102b2030e9d1c9dab7ee8a7fa8503a8097f05bc15c00410ad9a724fb6e: Status 404 returned error can't find the container with id 62560b102b2030e9d1c9dab7ee8a7fa8503a8097f05bc15c00410ad9a724fb6e
Sep 09 03:59:06.363743 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:06.363720    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vhd4r"]
Sep 09 03:59:06.369637 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:59:06.369604    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf850a46e_0840_481a_8f86_0b3877277387.slice/crio-b25085dd9731e6380076cf45b96d2640a00ea260c298272ec0bcf8fdd1024c93 WatchSource:0}: Error finding container b25085dd9731e6380076cf45b96d2640a00ea260c298272ec0bcf8fdd1024c93: Status 404 returned error can't find the container with id b25085dd9731e6380076cf45b96d2640a00ea260c298272ec0bcf8fdd1024c93
Sep 09 03:59:06.989246 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:06.989205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vhd4r" event={"ID":"f850a46e-0840-481a-8f86-0b3877277387","Type":"ContainerStarted","Data":"b25085dd9731e6380076cf45b96d2640a00ea260c298272ec0bcf8fdd1024c93"}
Sep 09 03:59:06.991221 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:06.991188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d7d794f54-vbjrc" event={"ID":"73fd3b9c-8485-41cc-a1cd-0e338d60a49a","Type":"ContainerStarted","Data":"6371a8e5a926740afbcc24e020997490de3306f564d58af8e836df092b15a807"}
Sep 09 03:59:06.991346 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:06.991245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d7d794f54-vbjrc" event={"ID":"73fd3b9c-8485-41cc-a1cd-0e338d60a49a","Type":"ContainerStarted","Data":"62560b102b2030e9d1c9dab7ee8a7fa8503a8097f05bc15c00410ad9a724fb6e"}
Sep 09 03:59:06.991738 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:06.991680    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:59:07.014002 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:07.013786    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d7d794f54-vbjrc" podStartSLOduration=35.013743907 podStartE2EDuration="35.013743907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:07.013167612 +0000 UTC m=+67.962858277" watchObservedRunningTime="2026-09-09 03:59:07.013743907 +0000 UTC m=+67.963434564"
Sep 09 03:59:08.007205 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:08.007169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwl9d" event={"ID":"57ca0680-7caf-465f-bedd-44a747cc90af","Type":"ContainerStarted","Data":"b79289b93156fa5e9e6450c701dab46f56f3fe01f26695eaa3fba76fc9272964"}
Sep 09 03:59:08.007718 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:08.007255    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwl9d" event={"ID":"57ca0680-7caf-465f-bedd-44a747cc90af","Type":"ContainerStarted","Data":"b92dae16ff19fcd08092f40a25a5c8ef2954ca7e4000d99da60f2e30ebe65781"}
Sep 09 03:59:08.025463 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:08.025393    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xwl9d" podStartSLOduration=67.930393183 podStartE2EDuration="1m9.025374206s" totalImagesPullingTime="1.094981023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:08.023457243 +0000 UTC m=+68.973147898" watchObservedRunningTime="2026-09-09 03:59:08.025374206 +0000 UTC m=+68.975064861"
Sep 09 03:59:09.324277 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.324241    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cngkj"]
Sep 09 03:59:09.327305 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.327285    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cngkj"
Sep 09 03:59:09.329864 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.329844    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5x77q\""
Sep 09 03:59:09.332942 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.332920    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 03:59:09.334922 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.334906    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d7d794f54-vbjrc"]
Sep 09 03:59:09.337202 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.337180    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cngkj"]
Sep 09 03:59:09.427941 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.427904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/003d3d43-36cc-4be1-9caf-785bfd0725c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-cngkj\" (UID: \"003d3d43-36cc-4be1-9caf-785bfd0725c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cngkj"
Sep 09 03:59:09.432155 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.432126    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55bd5d57b-h568l"]
Sep 09 03:59:09.435340 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.435315    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.455142 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.455115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55bd5d57b-h568l"]
Sep 09 03:59:09.528310 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.528271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77411f50-af38-4530-ac82-8d64ced36874-registry-certificates\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.528493 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.528325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cndvk\" (UniqueName: \"kubernetes.io/projected/77411f50-af38-4530-ac82-8d64ced36874-kube-api-access-cndvk\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.528493 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.528352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77411f50-af38-4530-ac82-8d64ced36874-bound-sa-token\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.528493 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.528457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77411f50-af38-4530-ac82-8d64ced36874-installation-pull-secrets\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.528493 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.528487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77411f50-af38-4530-ac82-8d64ced36874-ca-trust-extracted\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.528640 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.528519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77411f50-af38-4530-ac82-8d64ced36874-registry-tls\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.528640 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.528544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/003d3d43-36cc-4be1-9caf-785bfd0725c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-cngkj\" (UID: \"003d3d43-36cc-4be1-9caf-785bfd0725c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cngkj"
Sep 09 03:59:09.528640 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.528563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77411f50-af38-4530-ac82-8d64ced36874-image-registry-private-configuration\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.528640 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.528579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77411f50-af38-4530-ac82-8d64ced36874-trusted-ca\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.531529 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.531507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/003d3d43-36cc-4be1-9caf-785bfd0725c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-cngkj\" (UID: \"003d3d43-36cc-4be1-9caf-785bfd0725c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cngkj"
Sep 09 03:59:09.629411 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.629372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77411f50-af38-4530-ac82-8d64ced36874-installation-pull-secrets\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.629594 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.629418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77411f50-af38-4530-ac82-8d64ced36874-ca-trust-extracted\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.629594 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.629454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77411f50-af38-4530-ac82-8d64ced36874-registry-tls\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.629594 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.629485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77411f50-af38-4530-ac82-8d64ced36874-image-registry-private-configuration\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.629594 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.629512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77411f50-af38-4530-ac82-8d64ced36874-trusted-ca\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.629594 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.629544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77411f50-af38-4530-ac82-8d64ced36874-registry-certificates\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.629594 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.629567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cndvk\" (UniqueName: \"kubernetes.io/projected/77411f50-af38-4530-ac82-8d64ced36874-kube-api-access-cndvk\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.629594 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.629595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77411f50-af38-4530-ac82-8d64ced36874-bound-sa-token\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.629910 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.629859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77411f50-af38-4530-ac82-8d64ced36874-ca-trust-extracted\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.630900 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.630877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77411f50-af38-4530-ac82-8d64ced36874-trusted-ca\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.631005 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.630956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77411f50-af38-4530-ac82-8d64ced36874-registry-certificates\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.632750 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.632726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77411f50-af38-4530-ac82-8d64ced36874-image-registry-private-configuration\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.632918 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.632902    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77411f50-af38-4530-ac82-8d64ced36874-registry-tls\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.635056 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.635035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77411f50-af38-4530-ac82-8d64ced36874-installation-pull-secrets\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.639003 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.638984    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cngkj"
Sep 09 03:59:09.642395 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.642368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cndvk\" (UniqueName: \"kubernetes.io/projected/77411f50-af38-4530-ac82-8d64ced36874-kube-api-access-cndvk\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.646987 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.646967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77411f50-af38-4530-ac82-8d64ced36874-bound-sa-token\") pod \"image-registry-55bd5d57b-h568l\" (UID: \"77411f50-af38-4530-ac82-8d64ced36874\") " pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:09.745760 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:09.745688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:11.328297 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:11.328265    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cngkj"]
Sep 09 03:59:11.331132 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:59:11.331105    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003d3d43_36cc_4be1_9caf_785bfd0725c0.slice/crio-31fc897853dff5b08d69db7f95d86512e473d67d0800a7ac4948cc0bb9642b5a WatchSource:0}: Error finding container 31fc897853dff5b08d69db7f95d86512e473d67d0800a7ac4948cc0bb9642b5a: Status 404 returned error can't find the container with id 31fc897853dff5b08d69db7f95d86512e473d67d0800a7ac4948cc0bb9642b5a
Sep 09 03:59:11.338679 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:11.338656    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55bd5d57b-h568l"]
Sep 09 03:59:11.341315 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:59:11.341283    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77411f50_af38_4530_ac82_8d64ced36874.slice/crio-8ecd2a0ac7a27e0a022f5d03c069f239e279b1bcc8196535c64b6aa0e130476f WatchSource:0}: Error finding container 8ecd2a0ac7a27e0a022f5d03c069f239e279b1bcc8196535c64b6aa0e130476f: Status 404 returned error can't find the container with id 8ecd2a0ac7a27e0a022f5d03c069f239e279b1bcc8196535c64b6aa0e130476f
Sep 09 03:59:12.023620 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.023578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qhb72" event={"ID":"2988c802-f1a1-4199-9131-ecca43a5f4df","Type":"ContainerStarted","Data":"2dfbca641a41e13b14b10c0f45c1b63140fe91edab7794ed3f223293183d42da"}
Sep 09 03:59:12.023802 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.023701    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:59:12.025404 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.025376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-brsc7" event={"ID":"93076315-afaf-43ce-a72c-104751d6d4e2","Type":"ContainerStarted","Data":"7e8be258aad1a73b49c0ab96deab95fe73e9e07ebae51208bb9209bef46b26cf"}
Sep 09 03:59:12.026865 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.026800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cngkj" event={"ID":"003d3d43-36cc-4be1-9caf-785bfd0725c0","Type":"ContainerStarted","Data":"31fc897853dff5b08d69db7f95d86512e473d67d0800a7ac4948cc0bb9642b5a"}
Sep 09 03:59:12.029128 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.029107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vhd4r" event={"ID":"f850a46e-0840-481a-8f86-0b3877277387","Type":"ContainerStarted","Data":"e9d19af72dc26cd4184a5d9dcbf45f443f56be35f0f9df3fd7d7496969ec443d"}
Sep 09 03:59:12.029223 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.029150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vhd4r" event={"ID":"f850a46e-0840-481a-8f86-0b3877277387","Type":"ContainerStarted","Data":"65b8466ca15ff122bba29339f9d4785f019e4dcbad449ac5bf7e6c0384e87515"}
Sep 09 03:59:12.029223 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.029180    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:59:12.030678 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.030658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bd5d57b-h568l" event={"ID":"77411f50-af38-4530-ac82-8d64ced36874","Type":"ContainerStarted","Data":"600efdd1d4016a67ef2311cd49505e76fbd568bdfb0111c860f7d0054149d948"}
Sep 09 03:59:12.030781 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.030697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bd5d57b-h568l" event={"ID":"77411f50-af38-4530-ac82-8d64ced36874","Type":"ContainerStarted","Data":"8ecd2a0ac7a27e0a022f5d03c069f239e279b1bcc8196535c64b6aa0e130476f"}
Sep 09 03:59:12.030781 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.030749    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:12.042549 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.042506    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qhb72" podStartSLOduration=67.882417416 podStartE2EDuration="1m13.04249317s" totalImagesPullingTime="5.160075754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:12.042149709 +0000 UTC m=+72.991840363" watchObservedRunningTime="2026-09-09 03:59:12.04249317 +0000 UTC m=+72.992183824"
Sep 09 03:59:12.058566 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.058525    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-brsc7" podStartSLOduration=66.891889011 podStartE2EDuration="1m12.058512921s" totalImagesPullingTime="5.16662391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:12.058331718 +0000 UTC m=+73.008022371" watchObservedRunningTime="2026-09-09 03:59:12.058512921 +0000 UTC m=+73.008203554"
Sep 09 03:59:12.080471 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.080432    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55bd5d57b-h568l" podStartSLOduration=3.080419423 podStartE2EDuration="3.080419423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:12.079610503 +0000 UTC m=+73.029301169" watchObservedRunningTime="2026-09-09 03:59:12.080419423 +0000 UTC m=+73.030110078"
Sep 09 03:59:12.097296 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:12.097252    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vhd4r" podStartSLOduration=34.418937001 podStartE2EDuration="39.097239241s" totalImagesPullingTime="4.67830224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:12.096475088 +0000 UTC m=+73.046165742" watchObservedRunningTime="2026-09-09 03:59:12.097239241 +0000 UTC m=+73.046929895"
Sep 09 03:59:13.035888 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:13.035784    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cngkj" event={"ID":"003d3d43-36cc-4be1-9caf-785bfd0725c0","Type":"ContainerStarted","Data":"a4559b0d5f25bea89db31a23d23c35fbff7a2ec29d54d79578668857f8a5b7ff"}
Sep 09 03:59:13.036320 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:13.035905    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cngkj"
Sep 09 03:59:13.040788 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:13.040754    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cngkj"
Sep 09 03:59:13.054740 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:13.054688    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-cngkj" podStartSLOduration=2.787512025 podStartE2EDuration="4.054675444s" totalImagesPullingTime="1.267163419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:13.053423161 +0000 UTC m=+74.003113827" watchObservedRunningTime="2026-09-09 03:59:13.054675444 +0000 UTC m=+74.004366098"
Sep 09 03:59:17.863936 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.862289    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lp9vx"]
Sep 09 03:59:17.868082 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.867958    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:17.871134 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.870938    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 03:59:17.871134 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.870947    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 03:59:17.871134 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.870957    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 03:59:17.871387 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.871240    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 03:59:17.871500 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.871472    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 03:59:17.872335 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.872312    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rxzhr\""
Sep 09 03:59:17.872427 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.872376    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 03:59:17.991667 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.991629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-metrics-client-ca\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:17.991842 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.991677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcnkd\" (UniqueName: \"kubernetes.io/projected/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-kube-api-access-wcnkd\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:17.991842 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.991704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-tls\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:17.991842 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.991730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-wtmp\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:17.991842 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.991794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-accelerators-collector-config\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:17.992059 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.991852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-root\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:17.992059 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.991896    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-sys\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:17.992059 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.991964    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-textfile\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:17.992059 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:17.992008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.093310 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.093276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-textfile\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.093468 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.093329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.093468 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.093356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-metrics-client-ca\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.093468 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.093382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcnkd\" (UniqueName: \"kubernetes.io/projected/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-kube-api-access-wcnkd\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.093468 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.093408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-tls\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.093468 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.093433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-wtmp\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.093468 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.093458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-accelerators-collector-config\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.093739 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.093496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-root\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.093739 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.093537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-sys\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.093739 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.093618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-sys\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.093739 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.093621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-textfile\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.093739 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:59:18.093703    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 03:59:18.094011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.093752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-root\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.094011 ip-10-0-142-198 kubenswrapper[2580]: E0909 03:59:18.093788    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-tls podName:fb17bdc5-dc3f-4873-a01a-5c9d11e36536 nodeName:}" failed. No retries permitted until 2026-09-09 03:59:18.593737394 +0000 UTC m=+79.543428032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-tls") pod "node-exporter-lp9vx" (UID: "fb17bdc5-dc3f-4873-a01a-5c9d11e36536") : secret "node-exporter-tls" not found
Sep 09 03:59:18.094011 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.093921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-wtmp\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.094162 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.094149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-accelerators-collector-config\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.094434 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.094416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-metrics-client-ca\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.096160 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.096142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.104393 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.104368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcnkd\" (UniqueName: \"kubernetes.io/projected/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-kube-api-access-wcnkd\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.596342 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.596300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-tls\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.598704 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.598681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fb17bdc5-dc3f-4873-a01a-5c9d11e36536-node-exporter-tls\") pod \"node-exporter-lp9vx\" (UID: \"fb17bdc5-dc3f-4873-a01a-5c9d11e36536\") " pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.779156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:18.779113    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lp9vx"
Sep 09 03:59:18.790209 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:59:18.790182    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb17bdc5_dc3f_4873_a01a_5c9d11e36536.slice/crio-32c17b7cbecfc794a756bddc04545828fff40dc02aa285074bba9ea85f6a0293 WatchSource:0}: Error finding container 32c17b7cbecfc794a756bddc04545828fff40dc02aa285074bba9ea85f6a0293: Status 404 returned error can't find the container with id 32c17b7cbecfc794a756bddc04545828fff40dc02aa285074bba9ea85f6a0293
Sep 09 03:59:19.057316 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:19.057226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lp9vx" event={"ID":"fb17bdc5-dc3f-4873-a01a-5c9d11e36536","Type":"ContainerStarted","Data":"32c17b7cbecfc794a756bddc04545828fff40dc02aa285074bba9ea85f6a0293"}
Sep 09 03:59:20.062269 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:20.062178    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb17bdc5-dc3f-4873-a01a-5c9d11e36536" containerID="a8da60b2c14b240a12cb9e723fe887b2ec21091ff77d0380d8f8c4b335e4f621" exitCode=0
Sep 09 03:59:20.062269 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:20.062237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lp9vx" event={"ID":"fb17bdc5-dc3f-4873-a01a-5c9d11e36536","Type":"ContainerDied","Data":"a8da60b2c14b240a12cb9e723fe887b2ec21091ff77d0380d8f8c4b335e4f621"}
Sep 09 03:59:21.068230 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:21.068110    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lp9vx" event={"ID":"fb17bdc5-dc3f-4873-a01a-5c9d11e36536","Type":"ContainerStarted","Data":"ac108400b7f6ed9513ff294070b6924df7280dc3f213babf851de0a1fac91641"}
Sep 09 03:59:21.068230 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:21.068164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lp9vx" event={"ID":"fb17bdc5-dc3f-4873-a01a-5c9d11e36536","Type":"ContainerStarted","Data":"b495dfbfa99bff5a2d0961b8299cf35c3ba613c311460f2c3bd6a0d888103e0b"}
Sep 09 03:59:21.100161 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:21.100108    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lp9vx" podStartSLOduration=3.040495661 podStartE2EDuration="4.100088443s" totalImagesPullingTime="990.248054ms" totalInitContainerRuntime="69.344728ms" isStatefulPod=true podCreationTimestamp="2026-09-09 03:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:21.097489357 +0000 UTC m=+82.047180027" watchObservedRunningTime="2026-09-09 03:59:21.100088443 +0000 UTC m=+82.049779098"
Sep 09 03:59:22.038641 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:22.038609    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vhd4r"
Sep 09 03:59:24.521143 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:24.521111    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-n4s2s"]
Sep 09 03:59:24.525738 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:24.525722    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-n4s2s"
Sep 09 03:59:24.528270 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:24.528246    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 03:59:24.528377 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:24.528249    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-2wfw7\""
Sep 09 03:59:24.528723 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:24.528710    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 03:59:24.534556 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:24.534535    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-n4s2s"]
Sep 09 03:59:24.640707 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:24.640669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6k6x\" (UniqueName: \"kubernetes.io/projected/0f6327e8-5787-4a47-accc-50e18269c70c-kube-api-access-h6k6x\") pod \"downloads-85d98c998f-n4s2s\" (UID: \"0f6327e8-5787-4a47-accc-50e18269c70c\") " pod="openshift-console/downloads-85d98c998f-n4s2s"
Sep 09 03:59:24.741952 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:24.741918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6k6x\" (UniqueName: \"kubernetes.io/projected/0f6327e8-5787-4a47-accc-50e18269c70c-kube-api-access-h6k6x\") pod \"downloads-85d98c998f-n4s2s\" (UID: \"0f6327e8-5787-4a47-accc-50e18269c70c\") " pod="openshift-console/downloads-85d98c998f-n4s2s"
Sep 09 03:59:24.751327 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:24.751292    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6k6x\" (UniqueName: \"kubernetes.io/projected/0f6327e8-5787-4a47-accc-50e18269c70c-kube-api-access-h6k6x\") pod \"downloads-85d98c998f-n4s2s\" (UID: \"0f6327e8-5787-4a47-accc-50e18269c70c\") " pod="openshift-console/downloads-85d98c998f-n4s2s"
Sep 09 03:59:24.835670 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:24.835592    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-n4s2s"
Sep 09 03:59:25.109728 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:25.109706    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-n4s2s"]
Sep 09 03:59:25.111694 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:59:25.111669    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6327e8_5787_4a47_accc_50e18269c70c.slice/crio-d231bff57512e0e2868aa4c783127ff411dfb323dd63aa62d5bdc86d3c33304d WatchSource:0}: Error finding container d231bff57512e0e2868aa4c783127ff411dfb323dd63aa62d5bdc86d3c33304d: Status 404 returned error can't find the container with id d231bff57512e0e2868aa4c783127ff411dfb323dd63aa62d5bdc86d3c33304d
Sep 09 03:59:26.087188 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:26.087148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-n4s2s" event={"ID":"0f6327e8-5787-4a47-accc-50e18269c70c","Type":"ContainerStarted","Data":"d231bff57512e0e2868aa4c783127ff411dfb323dd63aa62d5bdc86d3c33304d"}
Sep 09 03:59:29.341690 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:29.341661    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:59:30.577454 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.577422    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d7df9cd6f-8v78t"]
Sep 09 03:59:30.580995 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.580976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.583511 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.583488    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 03:59:30.584847 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.584823    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rc74s\""
Sep 09 03:59:30.585045 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.584934    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 03:59:30.585045 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.584937    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 03:59:30.585193 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.585059    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 03:59:30.585253 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.585229    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 03:59:30.595553 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.595531    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d7df9cd6f-8v78t"]
Sep 09 03:59:30.689653 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.689621    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-console-config\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.689893 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.689691    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6bfc970-4187-4490-ac05-1106d09477ed-console-oauth-config\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.689893 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.689719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v42v\" (UniqueName: \"kubernetes.io/projected/b6bfc970-4187-4490-ac05-1106d09477ed-kube-api-access-2v42v\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.689893 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.689756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-oauth-serving-cert\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.689893 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.689845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6bfc970-4187-4490-ac05-1106d09477ed-console-serving-cert\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.689893 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.689889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-service-ca\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.790899 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.790857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6bfc970-4187-4490-ac05-1106d09477ed-console-oauth-config\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.790899 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.790896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v42v\" (UniqueName: \"kubernetes.io/projected/b6bfc970-4187-4490-ac05-1106d09477ed-kube-api-access-2v42v\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.791130 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.790934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-oauth-serving-cert\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.791130 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.790963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6bfc970-4187-4490-ac05-1106d09477ed-console-serving-cert\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.791130 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.790988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-service-ca\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.791130 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.791020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-console-config\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.791931 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.791880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-console-config\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.791931 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.791903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-oauth-serving-cert\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.792124 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.792042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-service-ca\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.793783 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.793746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6bfc970-4187-4490-ac05-1106d09477ed-console-oauth-config\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.794102 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.794081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6bfc970-4187-4490-ac05-1106d09477ed-console-serving-cert\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.800889 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.800869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v42v\" (UniqueName: \"kubernetes.io/projected/b6bfc970-4187-4490-ac05-1106d09477ed-kube-api-access-2v42v\") pod \"console-7d7df9cd6f-8v78t\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") " pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:30.892603 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:30.892569    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:31.178269 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:31.178238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d7df9cd6f-8v78t"]
Sep 09 03:59:31.181537 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:59:31.181511    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bfc970_4187_4490_ac05_1106d09477ed.slice/crio-0d3e9211d495276864c2bedeec7dd69b9172843d00907cb291aa3d9862d2f530 WatchSource:0}: Error finding container 0d3e9211d495276864c2bedeec7dd69b9172843d00907cb291aa3d9862d2f530: Status 404 returned error can't find the container with id 0d3e9211d495276864c2bedeec7dd69b9172843d00907cb291aa3d9862d2f530
Sep 09 03:59:32.112342 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:32.112301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7df9cd6f-8v78t" event={"ID":"b6bfc970-4187-4490-ac05-1106d09477ed","Type":"ContainerStarted","Data":"0d3e9211d495276864c2bedeec7dd69b9172843d00907cb291aa3d9862d2f530"}
Sep 09 03:59:33.042143 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:33.042108    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55bd5d57b-h568l"
Sep 09 03:59:34.360722 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:34.360660    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d7d794f54-vbjrc" podUID="73fd3b9c-8485-41cc-a1cd-0e338d60a49a" containerName="registry" containerID="cri-o://6371a8e5a926740afbcc24e020997490de3306f564d58af8e836df092b15a807" gracePeriod=30
Sep 09 03:59:35.127418 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:35.127387    2580 generic.go:356] "Generic (PLEG): container finished" podID="73fd3b9c-8485-41cc-a1cd-0e338d60a49a" containerID="6371a8e5a926740afbcc24e020997490de3306f564d58af8e836df092b15a807" exitCode=0
Sep 09 03:59:35.127586 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:35.127469    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d7d794f54-vbjrc" event={"ID":"73fd3b9c-8485-41cc-a1cd-0e338d60a49a","Type":"ContainerDied","Data":"6371a8e5a926740afbcc24e020997490de3306f564d58af8e836df092b15a807"}
Sep 09 03:59:35.129363 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:35.129340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7df9cd6f-8v78t" event={"ID":"b6bfc970-4187-4490-ac05-1106d09477ed","Type":"ContainerStarted","Data":"bab2f3432db566bcf62ef736a41b8d99fd678627b3d69383e5c57f4617683a67"}
Sep 09 03:59:35.152985 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:35.152945    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d7df9cd6f-8v78t" podStartSLOduration=1.967928604 podStartE2EDuration="5.152930518s" totalImagesPullingTime="3.185001914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:35.152696067 +0000 UTC m=+96.102386725" watchObservedRunningTime="2026-09-09 03:59:35.152930518 +0000 UTC m=+96.102621154"
Sep 09 03:59:40.031722 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.031683    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c99c55cb4-vh5hw"]
Sep 09 03:59:40.038385 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.038349    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.048796 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.048091    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c99c55cb4-vh5hw"]
Sep 09 03:59:40.048796 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.048525    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 03:59:40.168536 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.168499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-service-ca\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.168720 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.168553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fzfw\" (UniqueName: \"kubernetes.io/projected/5f634788-4228-470b-a461-29696863418b-kube-api-access-9fzfw\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.168720 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.168616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-console-config\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.168720 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.168666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-trusted-ca-bundle\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.168720 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.168695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-oauth-serving-cert\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.168890 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.168733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f634788-4228-470b-a461-29696863418b-console-oauth-config\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.168890 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.168791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f634788-4228-470b-a461-29696863418b-console-serving-cert\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.269849 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.269797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-console-config\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.270039 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.269878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-trusted-ca-bundle\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.270039 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.269904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-oauth-serving-cert\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.270039 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.269930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f634788-4228-470b-a461-29696863418b-console-oauth-config\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.270738 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.270270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f634788-4228-470b-a461-29696863418b-console-serving-cert\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.270738 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.270325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-service-ca\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.270738 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.270369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fzfw\" (UniqueName: \"kubernetes.io/projected/5f634788-4228-470b-a461-29696863418b-kube-api-access-9fzfw\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.270738 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.270601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-console-config\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.270738 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.270651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-oauth-serving-cert\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.271042 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.270825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-trusted-ca-bundle\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.271255 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.271229    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-service-ca\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.272851 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.272827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f634788-4228-470b-a461-29696863418b-console-oauth-config\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.273175 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.273154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f634788-4228-470b-a461-29696863418b-console-serving-cert\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.280261 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.280241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fzfw\" (UniqueName: \"kubernetes.io/projected/5f634788-4228-470b-a461-29696863418b-kube-api-access-9fzfw\") pod \"console-6c99c55cb4-vh5hw\" (UID: \"5f634788-4228-470b-a461-29696863418b\") " pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.351138 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.351107    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:40.893254 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.893215    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:40.893539 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.893518    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:40.898835 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:40.898816    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:41.161141 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.161056    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 03:59:41.639589 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.639558    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:59:41.784022 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.783987    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls\") pod \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") "
Sep 09 03:59:41.784022 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.784024    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-image-registry-private-configuration\") pod \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") "
Sep 09 03:59:41.784241 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.784059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-trusted-ca\") pod \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") "
Sep 09 03:59:41.784241 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.784087    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-installation-pull-secrets\") pod \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") "
Sep 09 03:59:41.784241 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.784113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-certificates\") pod \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") "
Sep 09 03:59:41.784241 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.784147    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmkmv\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-kube-api-access-mmkmv\") pod \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") "
Sep 09 03:59:41.784241 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.784202    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-ca-trust-extracted\") pod \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") "
Sep 09 03:59:41.784241 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.784232    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-bound-sa-token\") pod \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\" (UID: \"73fd3b9c-8485-41cc-a1cd-0e338d60a49a\") "
Sep 09 03:59:41.784631 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.784594    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "73fd3b9c-8485-41cc-a1cd-0e338d60a49a" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:59:41.784631 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.784605    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "73fd3b9c-8485-41cc-a1cd-0e338d60a49a" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:59:41.786746 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.786722    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "73fd3b9c-8485-41cc-a1cd-0e338d60a49a" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:59:41.786938 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.786912    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "73fd3b9c-8485-41cc-a1cd-0e338d60a49a" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:59:41.787004 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.786946    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-kube-api-access-mmkmv" (OuterVolumeSpecName: "kube-api-access-mmkmv") pod "73fd3b9c-8485-41cc-a1cd-0e338d60a49a" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a"). InnerVolumeSpecName "kube-api-access-mmkmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:59:41.787053 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.787001    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "73fd3b9c-8485-41cc-a1cd-0e338d60a49a" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:59:41.787156 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.787140    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "73fd3b9c-8485-41cc-a1cd-0e338d60a49a" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:59:41.792876 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.792853    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "73fd3b9c-8485-41cc-a1cd-0e338d60a49a" (UID: "73fd3b9c-8485-41cc-a1cd-0e338d60a49a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:59:41.874205 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.874176    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c99c55cb4-vh5hw"]
Sep 09 03:59:41.885049 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.885030    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-ca-trust-extracted\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 03:59:41.885109 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.885050    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-bound-sa-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 03:59:41.885109 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.885062    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 03:59:41.885109 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.885070    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-image-registry-private-configuration\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 03:59:41.885109 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.885078    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-trusted-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 03:59:41.885109 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.885086    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-installation-pull-secrets\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 03:59:41.885109 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.885097    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-registry-certificates\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 03:59:41.885109 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:41.885111    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmkmv\" (UniqueName: \"kubernetes.io/projected/73fd3b9c-8485-41cc-a1cd-0e338d60a49a-kube-api-access-mmkmv\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 03:59:41.919370 ip-10-0-142-198 kubenswrapper[2580]: W0909 03:59:41.919347    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f634788_4228_470b_a461_29696863418b.slice/crio-33c46a064557a8d0365458018d71f99c8f131a395e92ed9a628432763f90f986 WatchSource:0}: Error finding container 33c46a064557a8d0365458018d71f99c8f131a395e92ed9a628432763f90f986: Status 404 returned error can't find the container with id 33c46a064557a8d0365458018d71f99c8f131a395e92ed9a628432763f90f986
Sep 09 03:59:42.162024 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:42.161934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c99c55cb4-vh5hw" event={"ID":"5f634788-4228-470b-a461-29696863418b","Type":"ContainerStarted","Data":"6ff61980922149e2826c05d395dc1e46c1656ba3f11a0a7debf9729889628192"}
Sep 09 03:59:42.162024 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:42.161992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c99c55cb4-vh5hw" event={"ID":"5f634788-4228-470b-a461-29696863418b","Type":"ContainerStarted","Data":"33c46a064557a8d0365458018d71f99c8f131a395e92ed9a628432763f90f986"}
Sep 09 03:59:42.163542 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:42.163516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-n4s2s" event={"ID":"0f6327e8-5787-4a47-accc-50e18269c70c","Type":"ContainerStarted","Data":"85870096a48d2b6f6e29c3764225ec5caaf3461de216af79aae8ec8a7857ae55"}
Sep 09 03:59:42.163687 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:42.163671    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-n4s2s"
Sep 09 03:59:42.164978 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:42.164956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d7d794f54-vbjrc" event={"ID":"73fd3b9c-8485-41cc-a1cd-0e338d60a49a","Type":"ContainerDied","Data":"62560b102b2030e9d1c9dab7ee8a7fa8503a8097f05bc15c00410ad9a724fb6e"}
Sep 09 03:59:42.165088 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:42.164979    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d7d794f54-vbjrc"
Sep 09 03:59:42.165088 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:42.165006    2580 scope.go:117] "RemoveContainer" containerID="6371a8e5a926740afbcc24e020997490de3306f564d58af8e836df092b15a807"
Sep 09 03:59:42.165236 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:42.165215    2580 patch_prober.go:28] interesting pod/downloads-85d98c998f-n4s2s container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused" start-of-body=
Sep 09 03:59:42.165296 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:42.165272    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-n4s2s" podUID="0f6327e8-5787-4a47-accc-50e18269c70c" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused"
Sep 09 03:59:42.191632 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:42.191588    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c99c55cb4-vh5hw" podStartSLOduration=2.191574907 podStartE2EDuration="2.191574907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:42.190106441 +0000 UTC m=+103.139797088" watchObservedRunningTime="2026-09-09 03:59:42.191574907 +0000 UTC m=+103.141265558"
Sep 09 03:59:42.208745 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:42.208721    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d7d794f54-vbjrc"]
Sep 09 03:59:42.212574 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:42.212556    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d7d794f54-vbjrc"]
Sep 09 03:59:43.039473 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:43.039438    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qhb72"
Sep 09 03:59:43.056632 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:43.056569    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-n4s2s" podStartSLOduration=2.183671589 podStartE2EDuration="19.056547101s" totalImagesPullingTime="16.872875512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:59:42.236327235 +0000 UTC m=+103.186017889" watchObservedRunningTime="2026-09-09 03:59:43.056547101 +0000 UTC m=+104.006237756"
Sep 09 03:59:43.192571 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:43.192537    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-n4s2s"
Sep 09 03:59:43.595007 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:43.594970    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73fd3b9c-8485-41cc-a1cd-0e338d60a49a" path="/var/lib/kubelet/pods/73fd3b9c-8485-41cc-a1cd-0e338d60a49a/volumes"
Sep 09 03:59:44.335801 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:44.335743    2580 patch_prober.go:28] interesting pod/image-registry-d7d794f54-vbjrc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.133.0.6:5000/healthz\": context deadline exceeded" start-of-body=
Sep 09 03:59:44.336385 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:44.336345    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-d7d794f54-vbjrc" podUID="73fd3b9c-8485-41cc-a1cd-0e338d60a49a" containerName="registry" probeResult="failure" output="Get \"https://10.133.0.6:5000/healthz\": context deadline exceeded"
Sep 09 03:59:50.352031 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:50.351992    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:50.352591 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:50.352066    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:50.357359 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:50.357331    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:51.207168 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:51.207138    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 03:59:51.265345 ip-10-0-142-198 kubenswrapper[2580]: I0909 03:59:51.265313    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d7df9cd6f-8v78t"]
Sep 09 04:00:16.295541 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.295497    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d7df9cd6f-8v78t" podUID="b6bfc970-4187-4490-ac05-1106d09477ed" containerName="console" containerID="cri-o://bab2f3432db566bcf62ef736a41b8d99fd678627b3d69383e5c57f4617683a67" gracePeriod=15
Sep 09 04:00:16.562654 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.562600    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 04:00:16.619025 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.618994    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6bfc970-4187-4490-ac05-1106d09477ed-console-oauth-config\") pod \"b6bfc970-4187-4490-ac05-1106d09477ed\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") "
Sep 09 04:00:16.619025 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.619029    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6bfc970-4187-4490-ac05-1106d09477ed-console-serving-cert\") pod \"b6bfc970-4187-4490-ac05-1106d09477ed\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") "
Sep 09 04:00:16.619239 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.619050    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-oauth-serving-cert\") pod \"b6bfc970-4187-4490-ac05-1106d09477ed\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") "
Sep 09 04:00:16.619239 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.619096    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v42v\" (UniqueName: \"kubernetes.io/projected/b6bfc970-4187-4490-ac05-1106d09477ed-kube-api-access-2v42v\") pod \"b6bfc970-4187-4490-ac05-1106d09477ed\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") "
Sep 09 04:00:16.619239 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.619112    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-service-ca\") pod \"b6bfc970-4187-4490-ac05-1106d09477ed\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") "
Sep 09 04:00:16.619239 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.619132    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-console-config\") pod \"b6bfc970-4187-4490-ac05-1106d09477ed\" (UID: \"b6bfc970-4187-4490-ac05-1106d09477ed\") "
Sep 09 04:00:16.619591 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.619556    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-console-config" (OuterVolumeSpecName: "console-config") pod "b6bfc970-4187-4490-ac05-1106d09477ed" (UID: "b6bfc970-4187-4490-ac05-1106d09477ed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:00:16.619591 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.619570    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-service-ca" (OuterVolumeSpecName: "service-ca") pod "b6bfc970-4187-4490-ac05-1106d09477ed" (UID: "b6bfc970-4187-4490-ac05-1106d09477ed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:00:16.619591 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.619554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b6bfc970-4187-4490-ac05-1106d09477ed" (UID: "b6bfc970-4187-4490-ac05-1106d09477ed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:00:16.621490 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.621462    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bfc970-4187-4490-ac05-1106d09477ed-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b6bfc970-4187-4490-ac05-1106d09477ed" (UID: "b6bfc970-4187-4490-ac05-1106d09477ed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:16.621580 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.621505    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bfc970-4187-4490-ac05-1106d09477ed-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b6bfc970-4187-4490-ac05-1106d09477ed" (UID: "b6bfc970-4187-4490-ac05-1106d09477ed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:00:16.621580 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.621537    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bfc970-4187-4490-ac05-1106d09477ed-kube-api-access-2v42v" (OuterVolumeSpecName: "kube-api-access-2v42v") pod "b6bfc970-4187-4490-ac05-1106d09477ed" (UID: "b6bfc970-4187-4490-ac05-1106d09477ed"). InnerVolumeSpecName "kube-api-access-2v42v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:00:16.719532 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.719503    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6bfc970-4187-4490-ac05-1106d09477ed-console-oauth-config\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:00:16.719532 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.719526    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6bfc970-4187-4490-ac05-1106d09477ed-console-serving-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:00:16.719532 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.719535    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-oauth-serving-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:00:16.719718 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.719543    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v42v\" (UniqueName: \"kubernetes.io/projected/b6bfc970-4187-4490-ac05-1106d09477ed-kube-api-access-2v42v\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:00:16.719718 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.719551    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:00:16.719718 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:16.719559    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6bfc970-4187-4490-ac05-1106d09477ed-console-config\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:00:17.297302 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:17.297267    2580 generic.go:356] "Generic (PLEG): container finished" podID="b6bfc970-4187-4490-ac05-1106d09477ed" containerID="bab2f3432db566bcf62ef736a41b8d99fd678627b3d69383e5c57f4617683a67" exitCode=0
Sep 09 04:00:17.297302 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:17.297309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7df9cd6f-8v78t" event={"ID":"b6bfc970-4187-4490-ac05-1106d09477ed","Type":"ContainerDied","Data":"bab2f3432db566bcf62ef736a41b8d99fd678627b3d69383e5c57f4617683a67"}
Sep 09 04:00:17.297891 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:17.297346    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7df9cd6f-8v78t" event={"ID":"b6bfc970-4187-4490-ac05-1106d09477ed","Type":"ContainerDied","Data":"0d3e9211d495276864c2bedeec7dd69b9172843d00907cb291aa3d9862d2f530"}
Sep 09 04:00:17.297891 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:17.297366    2580 scope.go:117] "RemoveContainer" containerID="bab2f3432db566bcf62ef736a41b8d99fd678627b3d69383e5c57f4617683a67"
Sep 09 04:00:17.297891 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:17.297310    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7df9cd6f-8v78t"
Sep 09 04:00:17.305803 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:17.305761    2580 scope.go:117] "RemoveContainer" containerID="bab2f3432db566bcf62ef736a41b8d99fd678627b3d69383e5c57f4617683a67"
Sep 09 04:00:17.306100 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:00:17.306073    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab2f3432db566bcf62ef736a41b8d99fd678627b3d69383e5c57f4617683a67\": container with ID starting with bab2f3432db566bcf62ef736a41b8d99fd678627b3d69383e5c57f4617683a67 not found: ID does not exist" containerID="bab2f3432db566bcf62ef736a41b8d99fd678627b3d69383e5c57f4617683a67"
Sep 09 04:00:17.306149 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:17.306111    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab2f3432db566bcf62ef736a41b8d99fd678627b3d69383e5c57f4617683a67"} err="failed to get container status \"bab2f3432db566bcf62ef736a41b8d99fd678627b3d69383e5c57f4617683a67\": rpc error: code = NotFound desc = could not find container \"bab2f3432db566bcf62ef736a41b8d99fd678627b3d69383e5c57f4617683a67\": container with ID starting with bab2f3432db566bcf62ef736a41b8d99fd678627b3d69383e5c57f4617683a67 not found: ID does not exist"
Sep 09 04:00:17.318664 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:17.318645    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d7df9cd6f-8v78t"]
Sep 09 04:00:17.328269 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:17.328250    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d7df9cd6f-8v78t"]
Sep 09 04:00:17.593548 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:17.593523    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6bfc970-4187-4490-ac05-1106d09477ed" path="/var/lib/kubelet/pods/b6bfc970-4187-4490-ac05-1106d09477ed/volumes"
Sep 09 04:00:49.338063 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.338024    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68594544b8-kczrm"]
Sep 09 04:00:49.338501 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.338336    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6bfc970-4187-4490-ac05-1106d09477ed" containerName="console"
Sep 09 04:00:49.338501 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.338349    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6bfc970-4187-4490-ac05-1106d09477ed" containerName="console"
Sep 09 04:00:49.338501 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.338372    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73fd3b9c-8485-41cc-a1cd-0e338d60a49a" containerName="registry"
Sep 09 04:00:49.338501 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.338379    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="73fd3b9c-8485-41cc-a1cd-0e338d60a49a" containerName="registry"
Sep 09 04:00:49.338501 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.338420    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="73fd3b9c-8485-41cc-a1cd-0e338d60a49a" containerName="registry"
Sep 09 04:00:49.338501 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.338430    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6bfc970-4187-4490-ac05-1106d09477ed" containerName="console"
Sep 09 04:00:49.342543 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.342527    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.354682 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.354656    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68594544b8-kczrm"]
Sep 09 04:00:49.461346 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.461316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-oauth-serving-cert\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.461346 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.461350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-config\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.461575 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.461385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-trusted-ca-bundle\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.461575 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.461435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lc8h\" (UniqueName: \"kubernetes.io/projected/53fa2568-55bb-4c15-aa30-2cd8527e43cb-kube-api-access-4lc8h\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.461575 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.461491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-serving-cert\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.461575 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.461519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-oauth-config\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.461575 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.461536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-service-ca\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.561854 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.561822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-oauth-config\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.561854 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.561853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-service-ca\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.562100 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.561883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-oauth-serving-cert\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.562100 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.561904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-config\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.562100 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.561932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-trusted-ca-bundle\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.562100 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.561969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lc8h\" (UniqueName: \"kubernetes.io/projected/53fa2568-55bb-4c15-aa30-2cd8527e43cb-kube-api-access-4lc8h\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.562100 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.562009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-serving-cert\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.562793 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.562743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-service-ca\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.562959 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.562844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-config\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.563056 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.562960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-oauth-serving-cert\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.563142 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.563117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-trusted-ca-bundle\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.564940 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.564919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-oauth-config\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.564940 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.564935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-serving-cert\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.572105 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.572083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lc8h\" (UniqueName: \"kubernetes.io/projected/53fa2568-55bb-4c15-aa30-2cd8527e43cb-kube-api-access-4lc8h\") pod \"console-68594544b8-kczrm\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") " pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.650971 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.650887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:49.925342 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:49.925266    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68594544b8-kczrm"]
Sep 09 04:00:49.928311 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:00:49.928275    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fa2568_55bb_4c15_aa30_2cd8527e43cb.slice/crio-b652f4fc968e6051755cec0ded13acf9c5f59f0125559a93421054279fcb58d5 WatchSource:0}: Error finding container b652f4fc968e6051755cec0ded13acf9c5f59f0125559a93421054279fcb58d5: Status 404 returned error can't find the container with id b652f4fc968e6051755cec0ded13acf9c5f59f0125559a93421054279fcb58d5
Sep 09 04:00:50.416791 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:50.416750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68594544b8-kczrm" event={"ID":"53fa2568-55bb-4c15-aa30-2cd8527e43cb","Type":"ContainerStarted","Data":"1ddb28ba1f256af57f6ea522c58a50a3ce6abe4320095fb19bd6eb1f8f757f88"}
Sep 09 04:00:50.417149 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:50.416807    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68594544b8-kczrm" event={"ID":"53fa2568-55bb-4c15-aa30-2cd8527e43cb","Type":"ContainerStarted","Data":"b652f4fc968e6051755cec0ded13acf9c5f59f0125559a93421054279fcb58d5"}
Sep 09 04:00:50.436105 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:50.436061    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68594544b8-kczrm" podStartSLOduration=1.436044752 podStartE2EDuration="1.436044752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:00:50.434587472 +0000 UTC m=+171.384278126" watchObservedRunningTime="2026-09-09 04:00:50.436044752 +0000 UTC m=+171.385735406"
Sep 09 04:00:59.651809 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:59.651757    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:59.652201 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:59.651830    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:00:59.656414 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:00:59.656391    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:01:00.456851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:00.456824    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:01:00.503235 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:00.503203    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c99c55cb4-vh5hw"]
Sep 09 04:01:25.527706 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.527646    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c99c55cb4-vh5hw" podUID="5f634788-4228-470b-a461-29696863418b" containerName="console" containerID="cri-o://6ff61980922149e2826c05d395dc1e46c1656ba3f11a0a7debf9729889628192" gracePeriod=15
Sep 09 04:01:25.763840 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.763819    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 04:01:25.916209 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.916181    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-service-ca\") pod \"5f634788-4228-470b-a461-29696863418b\" (UID: \"5f634788-4228-470b-a461-29696863418b\") "
Sep 09 04:01:25.916401 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.916220    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-trusted-ca-bundle\") pod \"5f634788-4228-470b-a461-29696863418b\" (UID: \"5f634788-4228-470b-a461-29696863418b\") "
Sep 09 04:01:25.916401 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.916240    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-oauth-serving-cert\") pod \"5f634788-4228-470b-a461-29696863418b\" (UID: \"5f634788-4228-470b-a461-29696863418b\") "
Sep 09 04:01:25.916401 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.916265    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fzfw\" (UniqueName: \"kubernetes.io/projected/5f634788-4228-470b-a461-29696863418b-kube-api-access-9fzfw\") pod \"5f634788-4228-470b-a461-29696863418b\" (UID: \"5f634788-4228-470b-a461-29696863418b\") "
Sep 09 04:01:25.916401 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.916280    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f634788-4228-470b-a461-29696863418b-console-oauth-config\") pod \"5f634788-4228-470b-a461-29696863418b\" (UID: \"5f634788-4228-470b-a461-29696863418b\") "
Sep 09 04:01:25.916401 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.916309    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f634788-4228-470b-a461-29696863418b-console-serving-cert\") pod \"5f634788-4228-470b-a461-29696863418b\" (UID: \"5f634788-4228-470b-a461-29696863418b\") "
Sep 09 04:01:25.916401 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.916340    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-console-config\") pod \"5f634788-4228-470b-a461-29696863418b\" (UID: \"5f634788-4228-470b-a461-29696863418b\") "
Sep 09 04:01:25.916696 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.916656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-service-ca" (OuterVolumeSpecName: "service-ca") pod "5f634788-4228-470b-a461-29696863418b" (UID: "5f634788-4228-470b-a461-29696863418b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:01:25.916836 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.916808    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5f634788-4228-470b-a461-29696863418b" (UID: "5f634788-4228-470b-a461-29696863418b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:01:25.916912 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.916826    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5f634788-4228-470b-a461-29696863418b" (UID: "5f634788-4228-470b-a461-29696863418b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:01:25.916962 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.916941    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-console-config" (OuterVolumeSpecName: "console-config") pod "5f634788-4228-470b-a461-29696863418b" (UID: "5f634788-4228-470b-a461-29696863418b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:01:25.918693 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.918665    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f634788-4228-470b-a461-29696863418b-kube-api-access-9fzfw" (OuterVolumeSpecName: "kube-api-access-9fzfw") pod "5f634788-4228-470b-a461-29696863418b" (UID: "5f634788-4228-470b-a461-29696863418b"). InnerVolumeSpecName "kube-api-access-9fzfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:01:25.918817 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.918698    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f634788-4228-470b-a461-29696863418b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5f634788-4228-470b-a461-29696863418b" (UID: "5f634788-4228-470b-a461-29696863418b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:01:25.918817 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:25.918739    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f634788-4228-470b-a461-29696863418b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5f634788-4228-470b-a461-29696863418b" (UID: "5f634788-4228-470b-a461-29696863418b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:01:26.016829 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.016798    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:01:26.016829 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.016825    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-trusted-ca-bundle\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:01:26.016829 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.016835    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-oauth-serving-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:01:26.017037 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.016844    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fzfw\" (UniqueName: \"kubernetes.io/projected/5f634788-4228-470b-a461-29696863418b-kube-api-access-9fzfw\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:01:26.017037 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.016853    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f634788-4228-470b-a461-29696863418b-console-oauth-config\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:01:26.017037 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.016862    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f634788-4228-470b-a461-29696863418b-console-serving-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:01:26.017037 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.016869    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f634788-4228-470b-a461-29696863418b-console-config\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:01:26.553911 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.553874    2580 generic.go:356] "Generic (PLEG): container finished" podID="5f634788-4228-470b-a461-29696863418b" containerID="6ff61980922149e2826c05d395dc1e46c1656ba3f11a0a7debf9729889628192" exitCode=0
Sep 09 04:01:26.554363 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.553922    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c99c55cb4-vh5hw"
Sep 09 04:01:26.554363 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.553966    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c99c55cb4-vh5hw" event={"ID":"5f634788-4228-470b-a461-29696863418b","Type":"ContainerDied","Data":"6ff61980922149e2826c05d395dc1e46c1656ba3f11a0a7debf9729889628192"}
Sep 09 04:01:26.554363 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.554028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c99c55cb4-vh5hw" event={"ID":"5f634788-4228-470b-a461-29696863418b","Type":"ContainerDied","Data":"33c46a064557a8d0365458018d71f99c8f131a395e92ed9a628432763f90f986"}
Sep 09 04:01:26.554363 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.554056    2580 scope.go:117] "RemoveContainer" containerID="6ff61980922149e2826c05d395dc1e46c1656ba3f11a0a7debf9729889628192"
Sep 09 04:01:26.562834 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.562820    2580 scope.go:117] "RemoveContainer" containerID="6ff61980922149e2826c05d395dc1e46c1656ba3f11a0a7debf9729889628192"
Sep 09 04:01:26.563109 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:01:26.563089    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff61980922149e2826c05d395dc1e46c1656ba3f11a0a7debf9729889628192\": container with ID starting with 6ff61980922149e2826c05d395dc1e46c1656ba3f11a0a7debf9729889628192 not found: ID does not exist" containerID="6ff61980922149e2826c05d395dc1e46c1656ba3f11a0a7debf9729889628192"
Sep 09 04:01:26.563157 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.563118    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff61980922149e2826c05d395dc1e46c1656ba3f11a0a7debf9729889628192"} err="failed to get container status \"6ff61980922149e2826c05d395dc1e46c1656ba3f11a0a7debf9729889628192\": rpc error: code = NotFound desc = could not find container \"6ff61980922149e2826c05d395dc1e46c1656ba3f11a0a7debf9729889628192\": container with ID starting with 6ff61980922149e2826c05d395dc1e46c1656ba3f11a0a7debf9729889628192 not found: ID does not exist"
Sep 09 04:01:26.573966 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.573940    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c99c55cb4-vh5hw"]
Sep 09 04:01:26.576995 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:26.576975    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c99c55cb4-vh5hw"]
Sep 09 04:01:27.593384 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:27.593355    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f634788-4228-470b-a461-29696863418b" path="/var/lib/kubelet/pods/5f634788-4228-470b-a461-29696863418b/volumes"
Sep 09 04:01:39.745039 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:39.745006    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"]
Sep 09 04:01:39.745441 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:39.745254    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f634788-4228-470b-a461-29696863418b" containerName="console"
Sep 09 04:01:39.745441 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:39.745265    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f634788-4228-470b-a461-29696863418b" containerName="console"
Sep 09 04:01:39.745441 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:39.745310    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f634788-4228-470b-a461-29696863418b" containerName="console"
Sep 09 04:01:39.750131 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:39.750106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"
Sep 09 04:01:39.752526 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:39.752505    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 04:01:39.753614 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:39.753596    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wqsjv\""
Sep 09 04:01:39.753691 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:39.753596    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 04:01:39.756167 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:39.756147    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"]
Sep 09 04:01:39.913631 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:39.913590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9\" (UID: \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"
Sep 09 04:01:39.913631 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:39.913631    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9\" (UID: \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"
Sep 09 04:01:39.913877 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:39.913661    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjrtw\" (UniqueName: \"kubernetes.io/projected/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-kube-api-access-kjrtw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9\" (UID: \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"
Sep 09 04:01:40.014556 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:40.014457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9\" (UID: \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"
Sep 09 04:01:40.014556 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:40.014495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9\" (UID: \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"
Sep 09 04:01:40.014556 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:40.014521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjrtw\" (UniqueName: \"kubernetes.io/projected/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-kube-api-access-kjrtw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9\" (UID: \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"
Sep 09 04:01:40.014878 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:40.014850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9\" (UID: \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"
Sep 09 04:01:40.014935 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:40.014905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9\" (UID: \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"
Sep 09 04:01:40.023329 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:40.023300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjrtw\" (UniqueName: \"kubernetes.io/projected/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-kube-api-access-kjrtw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9\" (UID: \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"
Sep 09 04:01:40.060092 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:40.060060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"
Sep 09 04:01:40.329567 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:40.329543    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"]
Sep 09 04:01:40.332026 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:01:40.331995    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode062e1b9_c862_4bcf_87e4_4a9cd04d0fba.slice/crio-36fa8f7701b4abf4ffae74937b90211e48ac6bf8401d3ae20e621413438f3978 WatchSource:0}: Error finding container 36fa8f7701b4abf4ffae74937b90211e48ac6bf8401d3ae20e621413438f3978: Status 404 returned error can't find the container with id 36fa8f7701b4abf4ffae74937b90211e48ac6bf8401d3ae20e621413438f3978
Sep 09 04:01:40.604872 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:40.604840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9" event={"ID":"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba","Type":"ContainerStarted","Data":"36fa8f7701b4abf4ffae74937b90211e48ac6bf8401d3ae20e621413438f3978"}
Sep 09 04:01:46.629738 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:46.629708    2580 generic.go:356] "Generic (PLEG): container finished" podID="e062e1b9-c862-4bcf-87e4-4a9cd04d0fba" containerID="454dc8e0e56114906d509b14b57819fffae46de274c9008bdf766cb4d3a91769" exitCode=0
Sep 09 04:01:46.630123 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:46.629753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9" event={"ID":"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba","Type":"ContainerDied","Data":"454dc8e0e56114906d509b14b57819fffae46de274c9008bdf766cb4d3a91769"}
Sep 09 04:01:49.641136 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:49.641103    2580 generic.go:356] "Generic (PLEG): container finished" podID="e062e1b9-c862-4bcf-87e4-4a9cd04d0fba" containerID="d0843fc557675a44c206fbfa1ee7e3561bf8e3a473c5f15a67f458d91b23b11a" exitCode=0
Sep 09 04:01:49.641517 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:49.641154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9" event={"ID":"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba","Type":"ContainerDied","Data":"d0843fc557675a44c206fbfa1ee7e3561bf8e3a473c5f15a67f458d91b23b11a"}
Sep 09 04:01:57.675910 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:57.675875    2580 generic.go:356] "Generic (PLEG): container finished" podID="e062e1b9-c862-4bcf-87e4-4a9cd04d0fba" containerID="a51ad095c7459dadb2c4ca59edf379ee614388a3f4fe243ba03942b2c2274282" exitCode=0
Sep 09 04:01:57.676271 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:57.675935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9" event={"ID":"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba","Type":"ContainerDied","Data":"a51ad095c7459dadb2c4ca59edf379ee614388a3f4fe243ba03942b2c2274282"}
Sep 09 04:01:58.852438 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:58.852407    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"
Sep 09 04:01:58.859527 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:58.859502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-util\") pod \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\" (UID: \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\") "
Sep 09 04:01:58.859628 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:58.859540    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-bundle\") pod \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\" (UID: \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\") "
Sep 09 04:01:58.859628 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:58.859570    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjrtw\" (UniqueName: \"kubernetes.io/projected/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-kube-api-access-kjrtw\") pod \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\" (UID: \"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba\") "
Sep 09 04:01:58.860164 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:58.860138    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-bundle" (OuterVolumeSpecName: "bundle") pod "e062e1b9-c862-4bcf-87e4-4a9cd04d0fba" (UID: "e062e1b9-c862-4bcf-87e4-4a9cd04d0fba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:01:58.861922 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:58.861882    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-kube-api-access-kjrtw" (OuterVolumeSpecName: "kube-api-access-kjrtw") pod "e062e1b9-c862-4bcf-87e4-4a9cd04d0fba" (UID: "e062e1b9-c862-4bcf-87e4-4a9cd04d0fba"). InnerVolumeSpecName "kube-api-access-kjrtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:01:58.863637 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:58.863619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-util" (OuterVolumeSpecName: "util") pod "e062e1b9-c862-4bcf-87e4-4a9cd04d0fba" (UID: "e062e1b9-c862-4bcf-87e4-4a9cd04d0fba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:01:58.960426 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:58.960353    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-util\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:01:58.960426 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:58.960378    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-bundle\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:01:58.960426 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:58.960388    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjrtw\" (UniqueName: \"kubernetes.io/projected/e062e1b9-c862-4bcf-87e4-4a9cd04d0fba-kube-api-access-kjrtw\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:01:59.684853 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:59.684816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9" event={"ID":"e062e1b9-c862-4bcf-87e4-4a9cd04d0fba","Type":"ContainerDied","Data":"36fa8f7701b4abf4ffae74937b90211e48ac6bf8401d3ae20e621413438f3978"}
Sep 09 04:01:59.685019 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:59.684861    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hcfm9"
Sep 09 04:01:59.685019 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:01:59.684870    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36fa8f7701b4abf4ffae74937b90211e48ac6bf8401d3ae20e621413438f3978"
Sep 09 04:02:02.838312 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.838279    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp"]
Sep 09 04:02:02.838810 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.838645    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e062e1b9-c862-4bcf-87e4-4a9cd04d0fba" containerName="util"
Sep 09 04:02:02.838810 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.838661    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e062e1b9-c862-4bcf-87e4-4a9cd04d0fba" containerName="util"
Sep 09 04:02:02.838810 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.838678    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e062e1b9-c862-4bcf-87e4-4a9cd04d0fba" containerName="pull"
Sep 09 04:02:02.838810 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.838686    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e062e1b9-c862-4bcf-87e4-4a9cd04d0fba" containerName="pull"
Sep 09 04:02:02.838810 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.838705    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e062e1b9-c862-4bcf-87e4-4a9cd04d0fba" containerName="extract"
Sep 09 04:02:02.838810 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.838716    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e062e1b9-c862-4bcf-87e4-4a9cd04d0fba" containerName="extract"
Sep 09 04:02:02.838810 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.838798    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e062e1b9-c862-4bcf-87e4-4a9cd04d0fba" containerName="extract"
Sep 09 04:02:02.842503 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.842485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp"
Sep 09 04:02:02.845064 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.845039    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 09 04:02:02.845171 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.845107    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 09 04:02:02.845171 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.845107    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-95rj2\""
Sep 09 04:02:02.854672 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.854650    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp"]
Sep 09 04:02:02.886063 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.886032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bjsc\" (UniqueName: \"kubernetes.io/projected/05233dd2-c5b2-489d-8e7e-92b88072ec19-kube-api-access-2bjsc\") pod \"cert-manager-operator-controller-manager-55c444bf86-b8gwp\" (UID: \"05233dd2-c5b2-489d-8e7e-92b88072ec19\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp"
Sep 09 04:02:02.886063 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.886067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05233dd2-c5b2-489d-8e7e-92b88072ec19-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-b8gwp\" (UID: \"05233dd2-c5b2-489d-8e7e-92b88072ec19\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp"
Sep 09 04:02:02.986820 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.986784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bjsc\" (UniqueName: \"kubernetes.io/projected/05233dd2-c5b2-489d-8e7e-92b88072ec19-kube-api-access-2bjsc\") pod \"cert-manager-operator-controller-manager-55c444bf86-b8gwp\" (UID: \"05233dd2-c5b2-489d-8e7e-92b88072ec19\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp"
Sep 09 04:02:02.986955 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.986828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05233dd2-c5b2-489d-8e7e-92b88072ec19-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-b8gwp\" (UID: \"05233dd2-c5b2-489d-8e7e-92b88072ec19\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp"
Sep 09 04:02:02.987172 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.987158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05233dd2-c5b2-489d-8e7e-92b88072ec19-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-b8gwp\" (UID: \"05233dd2-c5b2-489d-8e7e-92b88072ec19\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp"
Sep 09 04:02:02.995280 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:02.995248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bjsc\" (UniqueName: \"kubernetes.io/projected/05233dd2-c5b2-489d-8e7e-92b88072ec19-kube-api-access-2bjsc\") pod \"cert-manager-operator-controller-manager-55c444bf86-b8gwp\" (UID: \"05233dd2-c5b2-489d-8e7e-92b88072ec19\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp"
Sep 09 04:02:03.152149 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:03.152067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp"
Sep 09 04:02:03.429559 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:03.429478    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp"]
Sep 09 04:02:03.433393 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:02:03.433366    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05233dd2_c5b2_489d_8e7e_92b88072ec19.slice/crio-f59ce336a2f2155a6ac69fbae1c06704e9c45512f13a3cbfac9429d055eac1b6 WatchSource:0}: Error finding container f59ce336a2f2155a6ac69fbae1c06704e9c45512f13a3cbfac9429d055eac1b6: Status 404 returned error can't find the container with id f59ce336a2f2155a6ac69fbae1c06704e9c45512f13a3cbfac9429d055eac1b6
Sep 09 04:02:03.700083 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:03.699997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp" event={"ID":"05233dd2-c5b2-489d-8e7e-92b88072ec19","Type":"ContainerStarted","Data":"f59ce336a2f2155a6ac69fbae1c06704e9c45512f13a3cbfac9429d055eac1b6"}
Sep 09 04:02:05.714313 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:05.714270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp" event={"ID":"05233dd2-c5b2-489d-8e7e-92b88072ec19","Type":"ContainerStarted","Data":"7c77e6940e467769a3cc60240edf4253089ff038786d262652754eeb6534ee02"}
Sep 09 04:02:05.735185 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:05.735130    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b8gwp" podStartSLOduration=2.05960707 podStartE2EDuration="3.735115995s" totalImagesPullingTime="1.675508925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:02:05.732975418 +0000 UTC m=+246.682666075" watchObservedRunningTime="2026-09-09 04:02:05.735115995 +0000 UTC m=+246.684806648"
Sep 09 04:02:07.177671 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.177636    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"]
Sep 09 04:02:07.181123 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.181108    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"
Sep 09 04:02:07.183550 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.183526    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wqsjv\""
Sep 09 04:02:07.183663 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.183563    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 04:02:07.184800 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.184783    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 04:02:07.190238 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.190214    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"]
Sep 09 04:02:07.218692 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.218664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6cwb\" (UniqueName: \"kubernetes.io/projected/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-kube-api-access-t6cwb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75\" (UID: \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"
Sep 09 04:02:07.218835 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.218695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75\" (UID: \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"
Sep 09 04:02:07.218835 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.218804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75\" (UID: \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"
Sep 09 04:02:07.320042 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.320004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75\" (UID: \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"
Sep 09 04:02:07.320211 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.320055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6cwb\" (UniqueName: \"kubernetes.io/projected/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-kube-api-access-t6cwb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75\" (UID: \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"
Sep 09 04:02:07.320211 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.320092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75\" (UID: \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"
Sep 09 04:02:07.320456 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.320433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75\" (UID: \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"
Sep 09 04:02:07.320456 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.320447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75\" (UID: \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"
Sep 09 04:02:07.328185 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.328162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6cwb\" (UniqueName: \"kubernetes.io/projected/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-kube-api-access-t6cwb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75\" (UID: \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"
Sep 09 04:02:07.490238 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.490154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"
Sep 09 04:02:07.758094 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:07.758071    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"]
Sep 09 04:02:07.760076 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:02:07.760050    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4192655c_8df5_49a8_8cd0_d7ed94ffb4d2.slice/crio-ec403cfdf592ebd0746981860985ee95a622076392572462d05cf7b71d032b79 WatchSource:0}: Error finding container ec403cfdf592ebd0746981860985ee95a622076392572462d05cf7b71d032b79: Status 404 returned error can't find the container with id ec403cfdf592ebd0746981860985ee95a622076392572462d05cf7b71d032b79
Sep 09 04:02:08.727688 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:08.727652    2580 generic.go:356] "Generic (PLEG): container finished" podID="4192655c-8df5-49a8-8cd0-d7ed94ffb4d2" containerID="e4f6e1f9ba3773f3e904073b2ff3492d3797ce14db138afe5bd02e88eb98bec0" exitCode=0
Sep 09 04:02:08.728162 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:08.727737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75" event={"ID":"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2","Type":"ContainerDied","Data":"e4f6e1f9ba3773f3e904073b2ff3492d3797ce14db138afe5bd02e88eb98bec0"}
Sep 09 04:02:08.728162 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:08.727798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75" event={"ID":"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2","Type":"ContainerStarted","Data":"ec403cfdf592ebd0746981860985ee95a622076392572462d05cf7b71d032b79"}
Sep 09 04:02:11.336908 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.336873    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf"]
Sep 09 04:02:11.339808 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.339792    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf"
Sep 09 04:02:11.342360 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.342343    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 04:02:11.342360 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.342351    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jgt5l\""
Sep 09 04:02:11.343517 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.343501    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 04:02:11.349026 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.349004    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf"]
Sep 09 04:02:11.450200 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.450172    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4cedb5c-0dc4-4949-b613-b8a93c37731f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dz5zf\" (UID: \"d4cedb5c-0dc4-4949-b613-b8a93c37731f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf"
Sep 09 04:02:11.450346 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.450214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqv7\" (UniqueName: \"kubernetes.io/projected/d4cedb5c-0dc4-4949-b613-b8a93c37731f-kube-api-access-dfqv7\") pod \"cert-manager-cainjector-559cdd8c87-dz5zf\" (UID: \"d4cedb5c-0dc4-4949-b613-b8a93c37731f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf"
Sep 09 04:02:11.551119 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.551083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqv7\" (UniqueName: \"kubernetes.io/projected/d4cedb5c-0dc4-4949-b613-b8a93c37731f-kube-api-access-dfqv7\") pod \"cert-manager-cainjector-559cdd8c87-dz5zf\" (UID: \"d4cedb5c-0dc4-4949-b613-b8a93c37731f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf"
Sep 09 04:02:11.551267 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.551150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4cedb5c-0dc4-4949-b613-b8a93c37731f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dz5zf\" (UID: \"d4cedb5c-0dc4-4949-b613-b8a93c37731f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf"
Sep 09 04:02:11.559325 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.559300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4cedb5c-0dc4-4949-b613-b8a93c37731f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dz5zf\" (UID: \"d4cedb5c-0dc4-4949-b613-b8a93c37731f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf"
Sep 09 04:02:11.559541 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.559521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqv7\" (UniqueName: \"kubernetes.io/projected/d4cedb5c-0dc4-4949-b613-b8a93c37731f-kube-api-access-dfqv7\") pod \"cert-manager-cainjector-559cdd8c87-dz5zf\" (UID: \"d4cedb5c-0dc4-4949-b613-b8a93c37731f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf"
Sep 09 04:02:11.658004 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.657973    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf"
Sep 09 04:02:11.742362 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.742328    2580 generic.go:356] "Generic (PLEG): container finished" podID="4192655c-8df5-49a8-8cd0-d7ed94ffb4d2" containerID="cafed77ac07ee334991c44792c896b2cc2aa820da103c8a8bc50a68ef433ff71" exitCode=0
Sep 09 04:02:11.742506 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.742410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75" event={"ID":"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2","Type":"ContainerDied","Data":"cafed77ac07ee334991c44792c896b2cc2aa820da103c8a8bc50a68ef433ff71"}
Sep 09 04:02:11.929128 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:11.929105    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf"]
Sep 09 04:02:11.931495 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:02:11.931454    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4cedb5c_0dc4_4949_b613_b8a93c37731f.slice/crio-131e7bbdee46aec863e0406be1f4087f7a46ace81f855b26ddc13d15156eb1fd WatchSource:0}: Error finding container 131e7bbdee46aec863e0406be1f4087f7a46ace81f855b26ddc13d15156eb1fd: Status 404 returned error can't find the container with id 131e7bbdee46aec863e0406be1f4087f7a46ace81f855b26ddc13d15156eb1fd
Sep 09 04:02:12.747695 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:12.747656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf" event={"ID":"d4cedb5c-0dc4-4949-b613-b8a93c37731f","Type":"ContainerStarted","Data":"131e7bbdee46aec863e0406be1f4087f7a46ace81f855b26ddc13d15156eb1fd"}
Sep 09 04:02:12.749698 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:12.749675    2580 generic.go:356] "Generic (PLEG): container finished" podID="4192655c-8df5-49a8-8cd0-d7ed94ffb4d2" containerID="09ecd3bd2c37fe9e505c7c4900cbb2c62c7d5b07d633d5b2a7e5a46eb54fa89e" exitCode=0
Sep 09 04:02:12.749829 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:12.749744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75" event={"ID":"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2","Type":"ContainerDied","Data":"09ecd3bd2c37fe9e505c7c4900cbb2c62c7d5b07d633d5b2a7e5a46eb54fa89e"}
Sep 09 04:02:13.901862 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:13.901838    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"
Sep 09 04:02:13.970936 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:13.970905    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-bundle\") pod \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\" (UID: \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\") "
Sep 09 04:02:13.971113 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:13.970965    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-util\") pod \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\" (UID: \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\") "
Sep 09 04:02:13.971113 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:13.970995    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6cwb\" (UniqueName: \"kubernetes.io/projected/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-kube-api-access-t6cwb\") pod \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\" (UID: \"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2\") "
Sep 09 04:02:13.971377 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:13.971354    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-bundle" (OuterVolumeSpecName: "bundle") pod "4192655c-8df5-49a8-8cd0-d7ed94ffb4d2" (UID: "4192655c-8df5-49a8-8cd0-d7ed94ffb4d2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:02:13.973474 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:13.973447    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-kube-api-access-t6cwb" (OuterVolumeSpecName: "kube-api-access-t6cwb") pod "4192655c-8df5-49a8-8cd0-d7ed94ffb4d2" (UID: "4192655c-8df5-49a8-8cd0-d7ed94ffb4d2"). InnerVolumeSpecName "kube-api-access-t6cwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:02:13.976728 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:13.976687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-util" (OuterVolumeSpecName: "util") pod "4192655c-8df5-49a8-8cd0-d7ed94ffb4d2" (UID: "4192655c-8df5-49a8-8cd0-d7ed94ffb4d2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:02:14.072131 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:14.072045    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-util\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:02:14.072131 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:14.072076    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6cwb\" (UniqueName: \"kubernetes.io/projected/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-kube-api-access-t6cwb\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:02:14.072131 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:14.072088    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4192655c-8df5-49a8-8cd0-d7ed94ffb4d2-bundle\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:02:14.760818 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:14.760785    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75"
Sep 09 04:02:14.760818 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:14.760802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fffm75" event={"ID":"4192655c-8df5-49a8-8cd0-d7ed94ffb4d2","Type":"ContainerDied","Data":"ec403cfdf592ebd0746981860985ee95a622076392572462d05cf7b71d032b79"}
Sep 09 04:02:14.761063 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:14.760859    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec403cfdf592ebd0746981860985ee95a622076392572462d05cf7b71d032b79"
Sep 09 04:02:15.766422 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:15.766327    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf" event={"ID":"d4cedb5c-0dc4-4949-b613-b8a93c37731f","Type":"ContainerStarted","Data":"f3adffaa23983766ad2902f307dfa989d31aca9ea840dcc1ab80719a9d542ba8"}
Sep 09 04:02:15.783953 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:15.783901    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dz5zf" podStartSLOduration=1.284750148 podStartE2EDuration="4.783879331s" totalImagesPullingTime="3.499129183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:02:15.782288858 +0000 UTC m=+256.731979513" watchObservedRunningTime="2026-09-09 04:02:15.783879331 +0000 UTC m=+256.733569985"
Sep 09 04:02:16.773850 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.773819    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mphrm"]
Sep 09 04:02:16.774215 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.774071    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4192655c-8df5-49a8-8cd0-d7ed94ffb4d2" containerName="util"
Sep 09 04:02:16.774215 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.774081    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4192655c-8df5-49a8-8cd0-d7ed94ffb4d2" containerName="util"
Sep 09 04:02:16.774215 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.774092    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4192655c-8df5-49a8-8cd0-d7ed94ffb4d2" containerName="extract"
Sep 09 04:02:16.774215 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.774097    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4192655c-8df5-49a8-8cd0-d7ed94ffb4d2" containerName="extract"
Sep 09 04:02:16.774215 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.774104    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4192655c-8df5-49a8-8cd0-d7ed94ffb4d2" containerName="pull"
Sep 09 04:02:16.774215 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.774111    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4192655c-8df5-49a8-8cd0-d7ed94ffb4d2" containerName="pull"
Sep 09 04:02:16.774215 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.774152    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4192655c-8df5-49a8-8cd0-d7ed94ffb4d2" containerName="extract"
Sep 09 04:02:16.778266 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.778251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mphrm"
Sep 09 04:02:16.780717 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.780687    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-mwvcz\""
Sep 09 04:02:16.786426 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.786404    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mphrm"]
Sep 09 04:02:16.895445 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.895406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzslc\" (UniqueName: \"kubernetes.io/projected/47a8e1aa-3f31-4821-aaf0-c296f90b9142-kube-api-access-tzslc\") pod \"cert-manager-55b4d954d5-mphrm\" (UID: \"47a8e1aa-3f31-4821-aaf0-c296f90b9142\") " pod="cert-manager/cert-manager-55b4d954d5-mphrm"
Sep 09 04:02:16.895445 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.895445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47a8e1aa-3f31-4821-aaf0-c296f90b9142-bound-sa-token\") pod \"cert-manager-55b4d954d5-mphrm\" (UID: \"47a8e1aa-3f31-4821-aaf0-c296f90b9142\") " pod="cert-manager/cert-manager-55b4d954d5-mphrm"
Sep 09 04:02:16.996506 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.996467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzslc\" (UniqueName: \"kubernetes.io/projected/47a8e1aa-3f31-4821-aaf0-c296f90b9142-kube-api-access-tzslc\") pod \"cert-manager-55b4d954d5-mphrm\" (UID: \"47a8e1aa-3f31-4821-aaf0-c296f90b9142\") " pod="cert-manager/cert-manager-55b4d954d5-mphrm"
Sep 09 04:02:16.996506 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:16.996514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47a8e1aa-3f31-4821-aaf0-c296f90b9142-bound-sa-token\") pod \"cert-manager-55b4d954d5-mphrm\" (UID: \"47a8e1aa-3f31-4821-aaf0-c296f90b9142\") " pod="cert-manager/cert-manager-55b4d954d5-mphrm"
Sep 09 04:02:17.005132 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:17.005105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47a8e1aa-3f31-4821-aaf0-c296f90b9142-bound-sa-token\") pod \"cert-manager-55b4d954d5-mphrm\" (UID: \"47a8e1aa-3f31-4821-aaf0-c296f90b9142\") " pod="cert-manager/cert-manager-55b4d954d5-mphrm"
Sep 09 04:02:17.005386 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:17.005365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzslc\" (UniqueName: \"kubernetes.io/projected/47a8e1aa-3f31-4821-aaf0-c296f90b9142-kube-api-access-tzslc\") pod \"cert-manager-55b4d954d5-mphrm\" (UID: \"47a8e1aa-3f31-4821-aaf0-c296f90b9142\") " pod="cert-manager/cert-manager-55b4d954d5-mphrm"
Sep 09 04:02:17.087212 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:17.087139    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mphrm"
Sep 09 04:02:17.361503 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:17.361339    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mphrm"]
Sep 09 04:02:17.363936 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:02:17.363914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47a8e1aa_3f31_4821_aaf0_c296f90b9142.slice/crio-4308cbd485f4e5dbdb78f689e3594c3fb0e2d0186c18e9a6fb9fb100fbb7ec08 WatchSource:0}: Error finding container 4308cbd485f4e5dbdb78f689e3594c3fb0e2d0186c18e9a6fb9fb100fbb7ec08: Status 404 returned error can't find the container with id 4308cbd485f4e5dbdb78f689e3594c3fb0e2d0186c18e9a6fb9fb100fbb7ec08
Sep 09 04:02:17.774833 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:17.774719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mphrm" event={"ID":"47a8e1aa-3f31-4821-aaf0-c296f90b9142","Type":"ContainerStarted","Data":"84d6c883fb8d607ac543a625b8856c06c974b4150af294da76eb2247609864de"}
Sep 09 04:02:17.774833 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:17.774760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mphrm" event={"ID":"47a8e1aa-3f31-4821-aaf0-c296f90b9142","Type":"ContainerStarted","Data":"4308cbd485f4e5dbdb78f689e3594c3fb0e2d0186c18e9a6fb9fb100fbb7ec08"}
Sep 09 04:02:17.791022 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:17.790975    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mphrm" podStartSLOduration=1.790958993 podStartE2EDuration="1.790958993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:02:17.790857399 +0000 UTC m=+258.740548055" watchObservedRunningTime="2026-09-09 04:02:17.790958993 +0000 UTC m=+258.740649648"
Sep 09 04:02:20.366156 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.366118    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr"]
Sep 09 04:02:20.369233 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.369218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr"
Sep 09 04:02:20.371782 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.371747    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 04:02:20.371914 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.371897    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 04:02:20.373171 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.373156    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-k7tlx\""
Sep 09 04:02:20.377044 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.377023    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr"]
Sep 09 04:02:20.420925 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.420898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mct4m\" (UniqueName: \"kubernetes.io/projected/29e90e45-8728-4aad-9d64-9efbf18ae04f-kube-api-access-mct4m\") pod \"openshift-lws-operator-fd8ccff4c-9dfsr\" (UID: \"29e90e45-8728-4aad-9d64-9efbf18ae04f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr"
Sep 09 04:02:20.421033 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.420934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29e90e45-8728-4aad-9d64-9efbf18ae04f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9dfsr\" (UID: \"29e90e45-8728-4aad-9d64-9efbf18ae04f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr"
Sep 09 04:02:20.521322 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.521257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mct4m\" (UniqueName: \"kubernetes.io/projected/29e90e45-8728-4aad-9d64-9efbf18ae04f-kube-api-access-mct4m\") pod \"openshift-lws-operator-fd8ccff4c-9dfsr\" (UID: \"29e90e45-8728-4aad-9d64-9efbf18ae04f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr"
Sep 09 04:02:20.521322 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.521298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29e90e45-8728-4aad-9d64-9efbf18ae04f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9dfsr\" (UID: \"29e90e45-8728-4aad-9d64-9efbf18ae04f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr"
Sep 09 04:02:20.521649 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.521633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29e90e45-8728-4aad-9d64-9efbf18ae04f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9dfsr\" (UID: \"29e90e45-8728-4aad-9d64-9efbf18ae04f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr"
Sep 09 04:02:20.530181 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.530155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mct4m\" (UniqueName: \"kubernetes.io/projected/29e90e45-8728-4aad-9d64-9efbf18ae04f-kube-api-access-mct4m\") pod \"openshift-lws-operator-fd8ccff4c-9dfsr\" (UID: \"29e90e45-8728-4aad-9d64-9efbf18ae04f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr"
Sep 09 04:02:20.679143 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.679067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr"
Sep 09 04:02:20.953915 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:20.953879    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr"]
Sep 09 04:02:20.956222 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:02:20.956197    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29e90e45_8728_4aad_9d64_9efbf18ae04f.slice/crio-e582a66b53d6ce9ea918809b167564f21ea11e9a434c4a9330f9fd503ce79c6b WatchSource:0}: Error finding container e582a66b53d6ce9ea918809b167564f21ea11e9a434c4a9330f9fd503ce79c6b: Status 404 returned error can't find the container with id e582a66b53d6ce9ea918809b167564f21ea11e9a434c4a9330f9fd503ce79c6b
Sep 09 04:02:21.797484 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:21.797446    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr" event={"ID":"29e90e45-8728-4aad-9d64-9efbf18ae04f","Type":"ContainerStarted","Data":"e582a66b53d6ce9ea918809b167564f21ea11e9a434c4a9330f9fd503ce79c6b"}
Sep 09 04:02:23.809066 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:23.808983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr" event={"ID":"29e90e45-8728-4aad-9d64-9efbf18ae04f","Type":"ContainerStarted","Data":"2ad22fe42b9f07afac251090e2b3e9a8a79eaccfb447db91f1b3b42fddf200ce"}
Sep 09 04:02:23.828621 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:23.828570    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9dfsr" podStartSLOduration=1.349716438 podStartE2EDuration="3.828556866s" totalImagesPullingTime="2.478840428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:02:23.827486598 +0000 UTC m=+264.777177255" watchObservedRunningTime="2026-09-09 04:02:23.828556866 +0000 UTC m=+264.778247520"
Sep 09 04:02:26.545796 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.545748    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"]
Sep 09 04:02:26.568900 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.568872    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"]
Sep 09 04:02:26.569043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.568944    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"
Sep 09 04:02:26.571712 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.571677    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 04:02:26.571843 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.571793    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wqsjv\""
Sep 09 04:02:26.571910 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.571862    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 04:02:26.665485 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.665458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/898943e1-0c72-4949-86f6-79d6a997223e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z\" (UID: \"898943e1-0c72-4949-86f6-79d6a997223e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"
Sep 09 04:02:26.665485 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.665488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/898943e1-0c72-4949-86f6-79d6a997223e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z\" (UID: \"898943e1-0c72-4949-86f6-79d6a997223e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"
Sep 09 04:02:26.665679 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.665545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq6kd\" (UniqueName: \"kubernetes.io/projected/898943e1-0c72-4949-86f6-79d6a997223e-kube-api-access-hq6kd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z\" (UID: \"898943e1-0c72-4949-86f6-79d6a997223e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"
Sep 09 04:02:26.765891 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.765853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/898943e1-0c72-4949-86f6-79d6a997223e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z\" (UID: \"898943e1-0c72-4949-86f6-79d6a997223e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"
Sep 09 04:02:26.766043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.765898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/898943e1-0c72-4949-86f6-79d6a997223e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z\" (UID: \"898943e1-0c72-4949-86f6-79d6a997223e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"
Sep 09 04:02:26.766043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.765985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq6kd\" (UniqueName: \"kubernetes.io/projected/898943e1-0c72-4949-86f6-79d6a997223e-kube-api-access-hq6kd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z\" (UID: \"898943e1-0c72-4949-86f6-79d6a997223e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"
Sep 09 04:02:26.766314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.766289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/898943e1-0c72-4949-86f6-79d6a997223e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z\" (UID: \"898943e1-0c72-4949-86f6-79d6a997223e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"
Sep 09 04:02:26.766407 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.766356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/898943e1-0c72-4949-86f6-79d6a997223e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z\" (UID: \"898943e1-0c72-4949-86f6-79d6a997223e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"
Sep 09 04:02:26.774483 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.774452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq6kd\" (UniqueName: \"kubernetes.io/projected/898943e1-0c72-4949-86f6-79d6a997223e-kube-api-access-hq6kd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z\" (UID: \"898943e1-0c72-4949-86f6-79d6a997223e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"
Sep 09 04:02:26.878357 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:26.878319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"
Sep 09 04:02:27.167928 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:27.167824    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"]
Sep 09 04:02:27.170336 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:02:27.170302    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898943e1_0c72_4949_86f6_79d6a997223e.slice/crio-648e8d658e350d7ab1b0095fa2356bfaf7bf33a4ab29b76a8cfb1f2442735a53 WatchSource:0}: Error finding container 648e8d658e350d7ab1b0095fa2356bfaf7bf33a4ab29b76a8cfb1f2442735a53: Status 404 returned error can't find the container with id 648e8d658e350d7ab1b0095fa2356bfaf7bf33a4ab29b76a8cfb1f2442735a53
Sep 09 04:02:27.826608 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:27.826579    2580 generic.go:356] "Generic (PLEG): container finished" podID="898943e1-0c72-4949-86f6-79d6a997223e" containerID="bb6ec426eccf077cc8d71729fb0f85f0e5aefc9756732fab7aeca3f11504369d" exitCode=0
Sep 09 04:02:27.826977 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:27.826621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z" event={"ID":"898943e1-0c72-4949-86f6-79d6a997223e","Type":"ContainerDied","Data":"bb6ec426eccf077cc8d71729fb0f85f0e5aefc9756732fab7aeca3f11504369d"}
Sep 09 04:02:27.826977 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:27.826656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z" event={"ID":"898943e1-0c72-4949-86f6-79d6a997223e","Type":"ContainerStarted","Data":"648e8d658e350d7ab1b0095fa2356bfaf7bf33a4ab29b76a8cfb1f2442735a53"}
Sep 09 04:02:29.836976 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:29.836951    2580 generic.go:356] "Generic (PLEG): container finished" podID="898943e1-0c72-4949-86f6-79d6a997223e" containerID="e0b19709d936fc92c6a61d81f2ba7cf0b94f646e72ffe244f0e355faea2db56e" exitCode=0
Sep 09 04:02:29.837315 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:29.837010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z" event={"ID":"898943e1-0c72-4949-86f6-79d6a997223e","Type":"ContainerDied","Data":"e0b19709d936fc92c6a61d81f2ba7cf0b94f646e72ffe244f0e355faea2db56e"}
Sep 09 04:02:30.844877 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:30.844841    2580 generic.go:356] "Generic (PLEG): container finished" podID="898943e1-0c72-4949-86f6-79d6a997223e" containerID="70652a0cea10727a576edd075ee389d87214ad6069820df2e1029306e0d20f85" exitCode=0
Sep 09 04:02:30.845230 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:30.844892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z" event={"ID":"898943e1-0c72-4949-86f6-79d6a997223e","Type":"ContainerDied","Data":"70652a0cea10727a576edd075ee389d87214ad6069820df2e1029306e0d20f85"}
Sep 09 04:02:31.972075 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:31.972053    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"
Sep 09 04:02:32.108331 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:32.108222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/898943e1-0c72-4949-86f6-79d6a997223e-util\") pod \"898943e1-0c72-4949-86f6-79d6a997223e\" (UID: \"898943e1-0c72-4949-86f6-79d6a997223e\") "
Sep 09 04:02:32.108331 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:32.108282    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hq6kd\" (UniqueName: \"kubernetes.io/projected/898943e1-0c72-4949-86f6-79d6a997223e-kube-api-access-hq6kd\") pod \"898943e1-0c72-4949-86f6-79d6a997223e\" (UID: \"898943e1-0c72-4949-86f6-79d6a997223e\") "
Sep 09 04:02:32.108331 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:32.108302    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/898943e1-0c72-4949-86f6-79d6a997223e-bundle\") pod \"898943e1-0c72-4949-86f6-79d6a997223e\" (UID: \"898943e1-0c72-4949-86f6-79d6a997223e\") "
Sep 09 04:02:32.109010 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:32.108988    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898943e1-0c72-4949-86f6-79d6a997223e-bundle" (OuterVolumeSpecName: "bundle") pod "898943e1-0c72-4949-86f6-79d6a997223e" (UID: "898943e1-0c72-4949-86f6-79d6a997223e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:02:32.110491 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:32.110466    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898943e1-0c72-4949-86f6-79d6a997223e-kube-api-access-hq6kd" (OuterVolumeSpecName: "kube-api-access-hq6kd") pod "898943e1-0c72-4949-86f6-79d6a997223e" (UID: "898943e1-0c72-4949-86f6-79d6a997223e"). InnerVolumeSpecName "kube-api-access-hq6kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:02:32.113761 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:32.113737    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898943e1-0c72-4949-86f6-79d6a997223e-util" (OuterVolumeSpecName: "util") pod "898943e1-0c72-4949-86f6-79d6a997223e" (UID: "898943e1-0c72-4949-86f6-79d6a997223e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:02:32.209434 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:32.209390    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/898943e1-0c72-4949-86f6-79d6a997223e-util\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:02:32.209434 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:32.209426    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq6kd\" (UniqueName: \"kubernetes.io/projected/898943e1-0c72-4949-86f6-79d6a997223e-kube-api-access-hq6kd\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:02:32.209434 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:32.209437    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/898943e1-0c72-4949-86f6-79d6a997223e-bundle\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:02:32.855955 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:32.855920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z" event={"ID":"898943e1-0c72-4949-86f6-79d6a997223e","Type":"ContainerDied","Data":"648e8d658e350d7ab1b0095fa2356bfaf7bf33a4ab29b76a8cfb1f2442735a53"}
Sep 09 04:02:32.856118 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:32.855964    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="648e8d658e350d7ab1b0095fa2356bfaf7bf33a4ab29b76a8cfb1f2442735a53"
Sep 09 04:02:32.856118 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:32.855967    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlv5z"
Sep 09 04:02:43.557746 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.557708    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"]
Sep 09 04:02:43.558217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.558139    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898943e1-0c72-4949-86f6-79d6a997223e" containerName="util"
Sep 09 04:02:43.558217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.558156    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="898943e1-0c72-4949-86f6-79d6a997223e" containerName="util"
Sep 09 04:02:43.558217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.558184    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898943e1-0c72-4949-86f6-79d6a997223e" containerName="pull"
Sep 09 04:02:43.558217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.558191    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="898943e1-0c72-4949-86f6-79d6a997223e" containerName="pull"
Sep 09 04:02:43.558217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.558208    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898943e1-0c72-4949-86f6-79d6a997223e" containerName="extract"
Sep 09 04:02:43.558217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.558216    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="898943e1-0c72-4949-86f6-79d6a997223e" containerName="extract"
Sep 09 04:02:43.558508 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.558284    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="898943e1-0c72-4949-86f6-79d6a997223e" containerName="extract"
Sep 09 04:02:43.561569 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.561549    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"
Sep 09 04:02:43.564383 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.564361    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 04:02:43.565584 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.565567    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 04:02:43.565655 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.565638    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wqsjv\""
Sep 09 04:02:43.569591 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.569569    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"]
Sep 09 04:02:43.693496 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.693461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf8bl\" (UniqueName: \"kubernetes.io/projected/1ca21d6a-6b70-4618-9019-62f95fa27f3f-kube-api-access-zf8bl\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc\" (UID: \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"
Sep 09 04:02:43.693662 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.693522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca21d6a-6b70-4618-9019-62f95fa27f3f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc\" (UID: \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"
Sep 09 04:02:43.693662 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.693575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca21d6a-6b70-4618-9019-62f95fa27f3f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc\" (UID: \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"
Sep 09 04:02:43.757732 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.757696    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"]
Sep 09 04:02:43.760831 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.760817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.763485 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.763463    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 04:02:43.763605 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.763465    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 04:02:43.764081 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.764064    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 04:02:43.764156 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.764089    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 04:02:43.764156 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.764094    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2jwxl\""
Sep 09 04:02:43.774197 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.774175    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"]
Sep 09 04:02:43.794479 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.794456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca21d6a-6b70-4618-9019-62f95fa27f3f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc\" (UID: \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"
Sep 09 04:02:43.794586 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.794491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca21d6a-6b70-4618-9019-62f95fa27f3f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc\" (UID: \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"
Sep 09 04:02:43.794656 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.794633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8bl\" (UniqueName: \"kubernetes.io/projected/1ca21d6a-6b70-4618-9019-62f95fa27f3f-kube-api-access-zf8bl\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc\" (UID: \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"
Sep 09 04:02:43.794875 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.794857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca21d6a-6b70-4618-9019-62f95fa27f3f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc\" (UID: \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"
Sep 09 04:02:43.794926 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.794870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca21d6a-6b70-4618-9019-62f95fa27f3f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc\" (UID: \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"
Sep 09 04:02:43.810378 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.810322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf8bl\" (UniqueName: \"kubernetes.io/projected/1ca21d6a-6b70-4618-9019-62f95fa27f3f-kube-api-access-zf8bl\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc\" (UID: \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"
Sep 09 04:02:43.871529 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.871500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"
Sep 09 04:02:43.895939 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.895908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-manifests\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.896076 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.895963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-webhook-cert\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.896076 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.895996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-tmp\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.896076 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.896026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcgqt\" (UniqueName: \"kubernetes.io/projected/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-kube-api-access-zcgqt\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.896204 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.896095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.996724 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.996688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-manifests\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.996902 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.996728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-webhook-cert\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.996902 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.996749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-tmp\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.996902 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.996799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgqt\" (UniqueName: \"kubernetes.io/projected/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-kube-api-access-zcgqt\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.996902 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.996830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.997247 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.997223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-manifests\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.999272 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.999248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-tmp\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.999451 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.999426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:43.999509 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:43.999496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-webhook-cert\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:44.004917 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:44.004894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcgqt\" (UniqueName: \"kubernetes.io/projected/2f73fe7e-2f01-4659-8c9d-2e0411b75d66-kube-api-access-zcgqt\") pod \"opendatahub-operator-controller-manager-5996d6cf95-89nxx\" (UID: \"2f73fe7e-2f01-4659-8c9d-2e0411b75d66\") " pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:44.070969 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:44.070891    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:44.146056 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:44.146025    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"]
Sep 09 04:02:44.148881 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:02:44.148852    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca21d6a_6b70_4618_9019_62f95fa27f3f.slice/crio-dfe4e68c8202563205c40a8771c8b8b17e6b56aee8d8fded23c0037b6608986c WatchSource:0}: Error finding container dfe4e68c8202563205c40a8771c8b8b17e6b56aee8d8fded23c0037b6608986c: Status 404 returned error can't find the container with id dfe4e68c8202563205c40a8771c8b8b17e6b56aee8d8fded23c0037b6608986c
Sep 09 04:02:44.349210 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:44.349177    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"]
Sep 09 04:02:44.351614 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:02:44.351582    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f73fe7e_2f01_4659_8c9d_2e0411b75d66.slice/crio-dd23dab7e98e0592955c5d740e875e3aeea20c4a538ef04a785b38895192d4d5 WatchSource:0}: Error finding container dd23dab7e98e0592955c5d740e875e3aeea20c4a538ef04a785b38895192d4d5: Status 404 returned error can't find the container with id dd23dab7e98e0592955c5d740e875e3aeea20c4a538ef04a785b38895192d4d5
Sep 09 04:02:44.907874 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:44.907843    2580 generic.go:356] "Generic (PLEG): container finished" podID="1ca21d6a-6b70-4618-9019-62f95fa27f3f" containerID="288a7f1202575b951d21bf93b56f16ea09648aa79c875a32110ac03cba1866f5" exitCode=0
Sep 09 04:02:44.908313 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:44.907921    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc" event={"ID":"1ca21d6a-6b70-4618-9019-62f95fa27f3f","Type":"ContainerDied","Data":"288a7f1202575b951d21bf93b56f16ea09648aa79c875a32110ac03cba1866f5"}
Sep 09 04:02:44.908313 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:44.907971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc" event={"ID":"1ca21d6a-6b70-4618-9019-62f95fa27f3f","Type":"ContainerStarted","Data":"dfe4e68c8202563205c40a8771c8b8b17e6b56aee8d8fded23c0037b6608986c"}
Sep 09 04:02:44.909397 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:44.909378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx" event={"ID":"2f73fe7e-2f01-4659-8c9d-2e0411b75d66","Type":"ContainerStarted","Data":"dd23dab7e98e0592955c5d740e875e3aeea20c4a538ef04a785b38895192d4d5"}
Sep 09 04:02:45.918888 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:45.918846    2580 generic.go:356] "Generic (PLEG): container finished" podID="1ca21d6a-6b70-4618-9019-62f95fa27f3f" containerID="0bff693af6c52054dbbf73801d1401871ccb4a389b3c77ff76f94c1f6aafac94" exitCode=0
Sep 09 04:02:45.919296 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:45.918906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc" event={"ID":"1ca21d6a-6b70-4618-9019-62f95fa27f3f","Type":"ContainerDied","Data":"0bff693af6c52054dbbf73801d1401871ccb4a389b3c77ff76f94c1f6aafac94"}
Sep 09 04:02:47.930932 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:47.930896    2580 generic.go:356] "Generic (PLEG): container finished" podID="1ca21d6a-6b70-4618-9019-62f95fa27f3f" containerID="4d01c918080c9d59c237c9f5686e627917e3aa840f77d7aa7f27d1bc92cfcaa7" exitCode=0
Sep 09 04:02:47.931325 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:47.930970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc" event={"ID":"1ca21d6a-6b70-4618-9019-62f95fa27f3f","Type":"ContainerDied","Data":"4d01c918080c9d59c237c9f5686e627917e3aa840f77d7aa7f27d1bc92cfcaa7"}
Sep 09 04:02:47.932493 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:47.932471    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx" event={"ID":"2f73fe7e-2f01-4659-8c9d-2e0411b75d66","Type":"ContainerStarted","Data":"5dc12dd0bba0c55ae9da40675a9db0f270be2a6e47e6d8a26f0d605a59669fb6"}
Sep 09 04:02:48.938368 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:48.938332    2580 generic.go:356] "Generic (PLEG): container finished" podID="2f73fe7e-2f01-4659-8c9d-2e0411b75d66" containerID="5dc12dd0bba0c55ae9da40675a9db0f270be2a6e47e6d8a26f0d605a59669fb6" exitCode=0
Sep 09 04:02:48.938758 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:48.938423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx" event={"ID":"2f73fe7e-2f01-4659-8c9d-2e0411b75d66","Type":"ContainerDied","Data":"5dc12dd0bba0c55ae9da40675a9db0f270be2a6e47e6d8a26f0d605a59669fb6"}
Sep 09 04:02:49.065097 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.065074    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"
Sep 09 04:02:49.139123 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.139087    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca21d6a-6b70-4618-9019-62f95fa27f3f-bundle\") pod \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\" (UID: \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\") "
Sep 09 04:02:49.139314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.139136    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca21d6a-6b70-4618-9019-62f95fa27f3f-util\") pod \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\" (UID: \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\") "
Sep 09 04:02:49.139314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.139182    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf8bl\" (UniqueName: \"kubernetes.io/projected/1ca21d6a-6b70-4618-9019-62f95fa27f3f-kube-api-access-zf8bl\") pod \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\" (UID: \"1ca21d6a-6b70-4618-9019-62f95fa27f3f\") "
Sep 09 04:02:49.139813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.139787    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca21d6a-6b70-4618-9019-62f95fa27f3f-bundle" (OuterVolumeSpecName: "bundle") pod "1ca21d6a-6b70-4618-9019-62f95fa27f3f" (UID: "1ca21d6a-6b70-4618-9019-62f95fa27f3f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:02:49.141417 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.141390    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca21d6a-6b70-4618-9019-62f95fa27f3f-kube-api-access-zf8bl" (OuterVolumeSpecName: "kube-api-access-zf8bl") pod "1ca21d6a-6b70-4618-9019-62f95fa27f3f" (UID: "1ca21d6a-6b70-4618-9019-62f95fa27f3f"). InnerVolumeSpecName "kube-api-access-zf8bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:02:49.145120 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.145083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca21d6a-6b70-4618-9019-62f95fa27f3f-util" (OuterVolumeSpecName: "util") pod "1ca21d6a-6b70-4618-9019-62f95fa27f3f" (UID: "1ca21d6a-6b70-4618-9019-62f95fa27f3f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:02:49.240341 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.240293    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ca21d6a-6b70-4618-9019-62f95fa27f3f-util\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:02:49.240341 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.240335    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf8bl\" (UniqueName: \"kubernetes.io/projected/1ca21d6a-6b70-4618-9019-62f95fa27f3f-kube-api-access-zf8bl\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:02:49.240341 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.240346    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ca21d6a-6b70-4618-9019-62f95fa27f3f-bundle\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:02:49.944475 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.944391    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc"
Sep 09 04:02:49.944871 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.944382    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0chsnc" event={"ID":"1ca21d6a-6b70-4618-9019-62f95fa27f3f","Type":"ContainerDied","Data":"dfe4e68c8202563205c40a8771c8b8b17e6b56aee8d8fded23c0037b6608986c"}
Sep 09 04:02:49.944871 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.944514    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfe4e68c8202563205c40a8771c8b8b17e6b56aee8d8fded23c0037b6608986c"
Sep 09 04:02:49.946698 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.946668    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx" event={"ID":"2f73fe7e-2f01-4659-8c9d-2e0411b75d66","Type":"ContainerStarted","Data":"43223ecb0d8d5e5d7a1c5b16bffa90602d1d87e7dc7baee990523110e116c1bf"}
Sep 09 04:02:49.946854 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.946800    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:02:49.967782 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:49.967722    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx" podStartSLOduration=1.921573935 podStartE2EDuration="6.967703912s" totalImagesPullingTime="4.008347611s" totalInitContainerRuntime="1.037782366s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:02:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:02:49.966121983 +0000 UTC m=+290.915812647" watchObservedRunningTime="2026-09-09 04:02:49.967703912 +0000 UTC m=+290.917394567"
Sep 09 04:02:59.483278 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:59.483245    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:02:59.483976 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:59.483952    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:02:59.491085 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:02:59.491069    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 04:03:00.956377 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:00.956350    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5996d6cf95-89nxx"
Sep 09 04:03:03.150447 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.150409    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"]
Sep 09 04:03:03.150851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.150672    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca21d6a-6b70-4618-9019-62f95fa27f3f" containerName="pull"
Sep 09 04:03:03.150851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.150683    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca21d6a-6b70-4618-9019-62f95fa27f3f" containerName="pull"
Sep 09 04:03:03.150851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.150703    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca21d6a-6b70-4618-9019-62f95fa27f3f" containerName="extract"
Sep 09 04:03:03.150851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.150709    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca21d6a-6b70-4618-9019-62f95fa27f3f" containerName="extract"
Sep 09 04:03:03.150851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.150717    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca21d6a-6b70-4618-9019-62f95fa27f3f" containerName="util"
Sep 09 04:03:03.150851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.150723    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca21d6a-6b70-4618-9019-62f95fa27f3f" containerName="util"
Sep 09 04:03:03.150851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.150797    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca21d6a-6b70-4618-9019-62f95fa27f3f" containerName="extract"
Sep 09 04:03:03.153614 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.153598    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"
Sep 09 04:03:03.158596 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.158568    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 04:03:03.158754 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.158616    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 04:03:03.158754 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.158676    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 04:03:03.158754 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.158666    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 04:03:03.159081 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.159061    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gwzpt\""
Sep 09 04:03:03.167229 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.167207    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"]
Sep 09 04:03:03.245901 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.245863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm65z\" (UniqueName: \"kubernetes.io/projected/42362810-b011-4dd6-96d6-d82ff2685386-kube-api-access-wm65z\") pod \"kube-auth-proxy-7449ffbbd4-v2fcj\" (UID: \"42362810-b011-4dd6-96d6-d82ff2685386\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"
Sep 09 04:03:03.246079 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.245927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42362810-b011-4dd6-96d6-d82ff2685386-tmp\") pod \"kube-auth-proxy-7449ffbbd4-v2fcj\" (UID: \"42362810-b011-4dd6-96d6-d82ff2685386\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"
Sep 09 04:03:03.246079 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.246011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42362810-b011-4dd6-96d6-d82ff2685386-tls-certs\") pod \"kube-auth-proxy-7449ffbbd4-v2fcj\" (UID: \"42362810-b011-4dd6-96d6-d82ff2685386\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"
Sep 09 04:03:03.347108 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.347064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42362810-b011-4dd6-96d6-d82ff2685386-tls-certs\") pod \"kube-auth-proxy-7449ffbbd4-v2fcj\" (UID: \"42362810-b011-4dd6-96d6-d82ff2685386\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"
Sep 09 04:03:03.347108 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.347114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm65z\" (UniqueName: \"kubernetes.io/projected/42362810-b011-4dd6-96d6-d82ff2685386-kube-api-access-wm65z\") pod \"kube-auth-proxy-7449ffbbd4-v2fcj\" (UID: \"42362810-b011-4dd6-96d6-d82ff2685386\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"
Sep 09 04:03:03.347328 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.347157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42362810-b011-4dd6-96d6-d82ff2685386-tmp\") pod \"kube-auth-proxy-7449ffbbd4-v2fcj\" (UID: \"42362810-b011-4dd6-96d6-d82ff2685386\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"
Sep 09 04:03:03.349562 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.349531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42362810-b011-4dd6-96d6-d82ff2685386-tmp\") pod \"kube-auth-proxy-7449ffbbd4-v2fcj\" (UID: \"42362810-b011-4dd6-96d6-d82ff2685386\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"
Sep 09 04:03:03.349800 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.349757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42362810-b011-4dd6-96d6-d82ff2685386-tls-certs\") pod \"kube-auth-proxy-7449ffbbd4-v2fcj\" (UID: \"42362810-b011-4dd6-96d6-d82ff2685386\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"
Sep 09 04:03:03.363959 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.363935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm65z\" (UniqueName: \"kubernetes.io/projected/42362810-b011-4dd6-96d6-d82ff2685386-kube-api-access-wm65z\") pod \"kube-auth-proxy-7449ffbbd4-v2fcj\" (UID: \"42362810-b011-4dd6-96d6-d82ff2685386\") " pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"
Sep 09 04:03:03.463285 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.463204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"
Sep 09 04:03:03.751175 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.751149    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj"]
Sep 09 04:03:03.753369 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:03:03.753345    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42362810_b011_4dd6_96d6_d82ff2685386.slice/crio-4f826d0901073bd84c44376a34034400c4b63c2aa1ca45d12acdb85664a2fc80 WatchSource:0}: Error finding container 4f826d0901073bd84c44376a34034400c4b63c2aa1ca45d12acdb85664a2fc80: Status 404 returned error can't find the container with id 4f826d0901073bd84c44376a34034400c4b63c2aa1ca45d12acdb85664a2fc80
Sep 09 04:03:03.755220 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:03.755202    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 04:03:04.012980 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.012893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj" event={"ID":"42362810-b011-4dd6-96d6-d82ff2685386","Type":"ContainerStarted","Data":"4f826d0901073bd84c44376a34034400c4b63c2aa1ca45d12acdb85664a2fc80"}
Sep 09 04:03:04.465848 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.465815    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"]
Sep 09 04:03:04.470544 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.470521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.473193 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.473170    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 04:03:04.473308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.473172    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-89swh\""
Sep 09 04:03:04.473308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.473172    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 04:03:04.473308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.473172    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 04:03:04.473308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.473199    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 04:03:04.481438 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.481412    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"]
Sep 09 04:03:04.555857 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.555829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.556043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.555867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7mdn\" (UniqueName: \"kubernetes.io/projected/9d6c414b-8b72-48e9-974d-a7f7c4839588-kube-api-access-l7mdn\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.556043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.555890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.556043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.555966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.556043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.556036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.556227 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.556067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.556227 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.556106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6c414b-8b72-48e9-974d-a7f7c4839588-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.656942 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.656903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.656942 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.656942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.657182 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.656969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6c414b-8b72-48e9-974d-a7f7c4839588-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.657182 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.657006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.657182 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.657041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7mdn\" (UniqueName: \"kubernetes.io/projected/9d6c414b-8b72-48e9-974d-a7f7c4839588-kube-api-access-l7mdn\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.657182 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.657070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.657182 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.657093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.657596 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.657570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.659533 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.659508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6c414b-8b72-48e9-974d-a7f7c4839588-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.659638 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.659590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.659906 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.659888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.660025 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.660005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.665905 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.665880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.666452 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.666434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7mdn\" (UniqueName: \"kubernetes.io/projected/9d6c414b-8b72-48e9-974d-a7f7c4839588-kube-api-access-l7mdn\") pod \"istiod-openshift-gateway-7b7995d5cd-2r655\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:04.780454 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:04.780360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:05.052378 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:05.052295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"]
Sep 09 04:03:05.055383 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:03:05.055362    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6c414b_8b72_48e9_974d_a7f7c4839588.slice/crio-841be048bf0148504a078bc8f48499d0e9739015196b918ea499daf19290c87a WatchSource:0}: Error finding container 841be048bf0148504a078bc8f48499d0e9739015196b918ea499daf19290c87a: Status 404 returned error can't find the container with id 841be048bf0148504a078bc8f48499d0e9739015196b918ea499daf19290c87a
Sep 09 04:03:06.027251 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:06.027212    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655" event={"ID":"9d6c414b-8b72-48e9-974d-a7f7c4839588","Type":"ContainerStarted","Data":"841be048bf0148504a078bc8f48499d0e9739015196b918ea499daf19290c87a"}
Sep 09 04:03:08.765957 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:08.765907    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:03:08.766335 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:08.765991    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:03:09.042127 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:09.042020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655" event={"ID":"9d6c414b-8b72-48e9-974d-a7f7c4839588","Type":"ContainerStarted","Data":"be26f030a4f22a9b90fb6a859ae6f370a47adff7b8f3fec5a352fcbfb9f59d94"}
Sep 09 04:03:09.042127 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:09.042091    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:09.043640 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:09.043611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj" event={"ID":"42362810-b011-4dd6-96d6-d82ff2685386","Type":"ContainerStarted","Data":"007aab075e9860300d26d06e83c2b6a767888e038ce16063a20a34746b25b715"}
Sep 09 04:03:09.044095 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:09.044074    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-2r655 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 04:03:09.044175 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:09.044118    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655" podUID="9d6c414b-8b72-48e9-974d-a7f7c4839588" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 04:03:09.065856 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:09.065809    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655" podStartSLOduration=1.35773352 podStartE2EDuration="5.065793534s" totalImagesPullingTime="3.708060014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:03:09.064965248 +0000 UTC m=+310.014655904" watchObservedRunningTime="2026-09-09 04:03:09.065793534 +0000 UTC m=+310.015484187"
Sep 09 04:03:09.085053 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:09.085009    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7449ffbbd4-v2fcj" podStartSLOduration=1.122595991 podStartE2EDuration="6.084994869s" totalImagesPullingTime="4.962398878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:03:09.084715791 +0000 UTC m=+310.034406446" watchObservedRunningTime="2026-09-09 04:03:09.084994869 +0000 UTC m=+310.034685523"
Sep 09 04:03:10.048898 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:10.048863    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-2r655 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 04:03:10.049295 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:10.048918    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655" podUID="9d6c414b-8b72-48e9-974d-a7f7c4839588" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 04:03:13.049302 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:13.049270    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:24.384154 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.384116    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"]
Sep 09 04:03:24.393450 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.393408    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"
Sep 09 04:03:24.396118 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.396092    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 09 04:03:24.396481 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.396450    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dpsnf\""
Sep 09 04:03:24.403896 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.403874    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz"]
Sep 09 04:03:24.410533 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.410503    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlxqg\" (UniqueName: \"kubernetes.io/projected/74648403-ac49-4cf3-8930-aa20f79e15b8-kube-api-access-zlxqg\") pod \"kserve-controller-manager-d6f4b65b8-zxrqw\" (UID: \"74648403-ac49-4cf3-8930-aa20f79e15b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"
Sep 09 04:03:24.410651 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.410550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74648403-ac49-4cf3-8930-aa20f79e15b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-zxrqw\" (UID: \"74648403-ac49-4cf3-8930-aa20f79e15b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"
Sep 09 04:03:24.412441 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.412417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"]
Sep 09 04:03:24.412554 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.412526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz"
Sep 09 04:03:24.416539 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.416283    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 09 04:03:24.417155 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.417140    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vc4mn\""
Sep 09 04:03:24.417383 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.417369    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz"]
Sep 09 04:03:24.511494 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.511461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlxqg\" (UniqueName: \"kubernetes.io/projected/74648403-ac49-4cf3-8930-aa20f79e15b8-kube-api-access-zlxqg\") pod \"kserve-controller-manager-d6f4b65b8-zxrqw\" (UID: \"74648403-ac49-4cf3-8930-aa20f79e15b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"
Sep 09 04:03:24.511680 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.511504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74648403-ac49-4cf3-8930-aa20f79e15b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-zxrqw\" (UID: \"74648403-ac49-4cf3-8930-aa20f79e15b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"
Sep 09 04:03:24.511680 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.511538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2zs6\" (UniqueName: \"kubernetes.io/projected/6b7c20ec-f7ca-484b-a33a-e1906aae95f7-kube-api-access-g2zs6\") pod \"llmisvc-controller-manager-777c6bd7d5-7gnhz\" (UID: \"6b7c20ec-f7ca-484b-a33a-e1906aae95f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz"
Sep 09 04:03:24.511680 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.511586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7c20ec-f7ca-484b-a33a-e1906aae95f7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7gnhz\" (UID: \"6b7c20ec-f7ca-484b-a33a-e1906aae95f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz"
Sep 09 04:03:24.511680 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:03:24.511627    2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 09 04:03:24.511992 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:03:24.511696    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74648403-ac49-4cf3-8930-aa20f79e15b8-cert podName:74648403-ac49-4cf3-8930-aa20f79e15b8 nodeName:}" failed. No retries permitted until 2026-09-09 04:03:25.011676112 +0000 UTC m=+325.961366751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/74648403-ac49-4cf3-8930-aa20f79e15b8-cert") pod "kserve-controller-manager-d6f4b65b8-zxrqw" (UID: "74648403-ac49-4cf3-8930-aa20f79e15b8") : secret "kserve-webhook-server-cert" not found
Sep 09 04:03:24.520789 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.520740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlxqg\" (UniqueName: \"kubernetes.io/projected/74648403-ac49-4cf3-8930-aa20f79e15b8-kube-api-access-zlxqg\") pod \"kserve-controller-manager-d6f4b65b8-zxrqw\" (UID: \"74648403-ac49-4cf3-8930-aa20f79e15b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"
Sep 09 04:03:24.612103 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.612061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7c20ec-f7ca-484b-a33a-e1906aae95f7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7gnhz\" (UID: \"6b7c20ec-f7ca-484b-a33a-e1906aae95f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz"
Sep 09 04:03:24.612290 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.612180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2zs6\" (UniqueName: \"kubernetes.io/projected/6b7c20ec-f7ca-484b-a33a-e1906aae95f7-kube-api-access-g2zs6\") pod \"llmisvc-controller-manager-777c6bd7d5-7gnhz\" (UID: \"6b7c20ec-f7ca-484b-a33a-e1906aae95f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz"
Sep 09 04:03:24.614868 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.614842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7c20ec-f7ca-484b-a33a-e1906aae95f7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7gnhz\" (UID: \"6b7c20ec-f7ca-484b-a33a-e1906aae95f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz"
Sep 09 04:03:24.625000 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.624976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2zs6\" (UniqueName: \"kubernetes.io/projected/6b7c20ec-f7ca-484b-a33a-e1906aae95f7-kube-api-access-g2zs6\") pod \"llmisvc-controller-manager-777c6bd7d5-7gnhz\" (UID: \"6b7c20ec-f7ca-484b-a33a-e1906aae95f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz"
Sep 09 04:03:24.726218 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.726133    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz"
Sep 09 04:03:24.997433 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:24.997407    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz"]
Sep 09 04:03:24.999867 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:03:24.999844    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6b7c20ec_f7ca_484b_a33a_e1906aae95f7.slice/crio-851bfb14d503b1315da663dd86018a2dd9332c4ffeed13984c5095a1e0bf212c WatchSource:0}: Error finding container 851bfb14d503b1315da663dd86018a2dd9332c4ffeed13984c5095a1e0bf212c: Status 404 returned error can't find the container with id 851bfb14d503b1315da663dd86018a2dd9332c4ffeed13984c5095a1e0bf212c
Sep 09 04:03:25.014950 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.014929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74648403-ac49-4cf3-8930-aa20f79e15b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-zxrqw\" (UID: \"74648403-ac49-4cf3-8930-aa20f79e15b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"
Sep 09 04:03:25.017303 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.017286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74648403-ac49-4cf3-8930-aa20f79e15b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-zxrqw\" (UID: \"74648403-ac49-4cf3-8930-aa20f79e15b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"
Sep 09 04:03:25.116907 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.116875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz" event={"ID":"6b7c20ec-f7ca-484b-a33a-e1906aae95f7","Type":"ContainerStarted","Data":"851bfb14d503b1315da663dd86018a2dd9332c4ffeed13984c5095a1e0bf212c"}
Sep 09 04:03:25.222784 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.222732    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"]
Sep 09 04:03:25.222967 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.222945    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655" podUID="9d6c414b-8b72-48e9-974d-a7f7c4839588" containerName="discovery" containerID="cri-o://be26f030a4f22a9b90fb6a859ae6f370a47adff7b8f3fec5a352fcbfb9f59d94" gracePeriod=30
Sep 09 04:03:25.307217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.307189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"
Sep 09 04:03:25.456930 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.456901    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:25.519243 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.519173    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-csr-dns-cert\") pod \"9d6c414b-8b72-48e9-974d-a7f7c4839588\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") "
Sep 09 04:03:25.519243 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.519206    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-token\") pod \"9d6c414b-8b72-48e9-974d-a7f7c4839588\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") "
Sep 09 04:03:25.519243 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.519226    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-csr-ca-configmap\") pod \"9d6c414b-8b72-48e9-974d-a7f7c4839588\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") "
Sep 09 04:03:25.519492 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.519245    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6c414b-8b72-48e9-974d-a7f7c4839588-local-certs\") pod \"9d6c414b-8b72-48e9-974d-a7f7c4839588\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") "
Sep 09 04:03:25.519492 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.519270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7mdn\" (UniqueName: \"kubernetes.io/projected/9d6c414b-8b72-48e9-974d-a7f7c4839588-kube-api-access-l7mdn\") pod \"9d6c414b-8b72-48e9-974d-a7f7c4839588\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") "
Sep 09 04:03:25.519492 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.519315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-cacerts\") pod \"9d6c414b-8b72-48e9-974d-a7f7c4839588\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") "
Sep 09 04:03:25.519492 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.519360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-kubeconfig\") pod \"9d6c414b-8b72-48e9-974d-a7f7c4839588\" (UID: \"9d6c414b-8b72-48e9-974d-a7f7c4839588\") "
Sep 09 04:03:25.519701 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.519644    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9d6c414b-8b72-48e9-974d-a7f7c4839588" (UID: "9d6c414b-8b72-48e9-974d-a7f7c4839588"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:03:25.522025 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.521985    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-cacerts" (OuterVolumeSpecName: "cacerts") pod "9d6c414b-8b72-48e9-974d-a7f7c4839588" (UID: "9d6c414b-8b72-48e9-974d-a7f7c4839588"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:03:25.522147 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.522016    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9d6c414b-8b72-48e9-974d-a7f7c4839588" (UID: "9d6c414b-8b72-48e9-974d-a7f7c4839588"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:03:25.522147 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.522098    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6c414b-8b72-48e9-974d-a7f7c4839588-local-certs" (OuterVolumeSpecName: "local-certs") pod "9d6c414b-8b72-48e9-974d-a7f7c4839588" (UID: "9d6c414b-8b72-48e9-974d-a7f7c4839588"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:03:25.522147 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.522120    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6c414b-8b72-48e9-974d-a7f7c4839588-kube-api-access-l7mdn" (OuterVolumeSpecName: "kube-api-access-l7mdn") pod "9d6c414b-8b72-48e9-974d-a7f7c4839588" (UID: "9d6c414b-8b72-48e9-974d-a7f7c4839588"). InnerVolumeSpecName "kube-api-access-l7mdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:03:25.522271 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.522171    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-token" (OuterVolumeSpecName: "istio-token") pod "9d6c414b-8b72-48e9-974d-a7f7c4839588" (UID: "9d6c414b-8b72-48e9-974d-a7f7c4839588"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:03:25.522271 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.522174    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9d6c414b-8b72-48e9-974d-a7f7c4839588" (UID: "9d6c414b-8b72-48e9-974d-a7f7c4839588"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:03:25.583980 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.581855    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"]
Sep 09 04:03:25.591500 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:03:25.591459    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74648403_ac49_4cf3_8930_aa20f79e15b8.slice/crio-1defa22bd5c2cb8119c139119c6d359d8f01f12b3fbf733616a434b8b4560881 WatchSource:0}: Error finding container 1defa22bd5c2cb8119c139119c6d359d8f01f12b3fbf733616a434b8b4560881: Status 404 returned error can't find the container with id 1defa22bd5c2cb8119c139119c6d359d8f01f12b3fbf733616a434b8b4560881
Sep 09 04:03:25.619944 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.619916    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-csr-dns-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:03:25.619944 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.619939    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:03:25.620069 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.619949    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-csr-ca-configmap\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:03:25.620069 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.619959    2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6c414b-8b72-48e9-974d-a7f7c4839588-local-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:03:25.620069 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.619968    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7mdn\" (UniqueName: \"kubernetes.io/projected/9d6c414b-8b72-48e9-974d-a7f7c4839588-kube-api-access-l7mdn\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:03:25.620069 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.619976    2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-cacerts\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:03:25.620069 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:25.619984    2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9d6c414b-8b72-48e9-974d-a7f7c4839588-istio-kubeconfig\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:03:26.123594 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:26.123562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw" event={"ID":"74648403-ac49-4cf3-8930-aa20f79e15b8","Type":"ContainerStarted","Data":"1defa22bd5c2cb8119c139119c6d359d8f01f12b3fbf733616a434b8b4560881"}
Sep 09 04:03:26.124939 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:26.124912    2580 generic.go:356] "Generic (PLEG): container finished" podID="9d6c414b-8b72-48e9-974d-a7f7c4839588" containerID="be26f030a4f22a9b90fb6a859ae6f370a47adff7b8f3fec5a352fcbfb9f59d94" exitCode=0
Sep 09 04:03:26.125057 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:26.124959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655" event={"ID":"9d6c414b-8b72-48e9-974d-a7f7c4839588","Type":"ContainerDied","Data":"be26f030a4f22a9b90fb6a859ae6f370a47adff7b8f3fec5a352fcbfb9f59d94"}
Sep 09 04:03:26.125057 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:26.124964    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"
Sep 09 04:03:26.125057 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:26.125001    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655" event={"ID":"9d6c414b-8b72-48e9-974d-a7f7c4839588","Type":"ContainerDied","Data":"841be048bf0148504a078bc8f48499d0e9739015196b918ea499daf19290c87a"}
Sep 09 04:03:26.125057 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:26.125027    2580 scope.go:117] "RemoveContainer" containerID="be26f030a4f22a9b90fb6a859ae6f370a47adff7b8f3fec5a352fcbfb9f59d94"
Sep 09 04:03:26.145180 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:26.145155    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"]
Sep 09 04:03:26.151477 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:26.151452    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2r655"]
Sep 09 04:03:26.204756 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:26.204733    2580 scope.go:117] "RemoveContainer" containerID="be26f030a4f22a9b90fb6a859ae6f370a47adff7b8f3fec5a352fcbfb9f59d94"
Sep 09 04:03:26.205115 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:03:26.205098    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be26f030a4f22a9b90fb6a859ae6f370a47adff7b8f3fec5a352fcbfb9f59d94\": container with ID starting with be26f030a4f22a9b90fb6a859ae6f370a47adff7b8f3fec5a352fcbfb9f59d94 not found: ID does not exist" containerID="be26f030a4f22a9b90fb6a859ae6f370a47adff7b8f3fec5a352fcbfb9f59d94"
Sep 09 04:03:26.205155 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:26.205124    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be26f030a4f22a9b90fb6a859ae6f370a47adff7b8f3fec5a352fcbfb9f59d94"} err="failed to get container status \"be26f030a4f22a9b90fb6a859ae6f370a47adff7b8f3fec5a352fcbfb9f59d94\": rpc error: code = NotFound desc = could not find container \"be26f030a4f22a9b90fb6a859ae6f370a47adff7b8f3fec5a352fcbfb9f59d94\": container with ID starting with be26f030a4f22a9b90fb6a859ae6f370a47adff7b8f3fec5a352fcbfb9f59d94 not found: ID does not exist"
Sep 09 04:03:27.596477 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:27.596439    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d6c414b-8b72-48e9-974d-a7f7c4839588" path="/var/lib/kubelet/pods/9d6c414b-8b72-48e9-974d-a7f7c4839588/volumes"
Sep 09 04:03:29.147042 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:29.147006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz" event={"ID":"6b7c20ec-f7ca-484b-a33a-e1906aae95f7","Type":"ContainerStarted","Data":"05f6733e5650e7cb41d8e1ebb652254b8968d0e4d5fafdc37c30e7e2d034915d"}
Sep 09 04:03:29.147451 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:29.147104    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz"
Sep 09 04:03:29.148629 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:29.148604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw" event={"ID":"74648403-ac49-4cf3-8930-aa20f79e15b8","Type":"ContainerStarted","Data":"26911d356ecd0f201cd1e8a9b615b9b3eeda9250564f85b978e273fc414d3e79"}
Sep 09 04:03:29.148760 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:29.148647    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"
Sep 09 04:03:29.172953 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:29.172897    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz" podStartSLOduration=1.536357646 podStartE2EDuration="5.172885267s" totalImagesPullingTime="3.636527621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:03:29.17027417 +0000 UTC m=+330.119964824" watchObservedRunningTime="2026-09-09 04:03:29.172885267 +0000 UTC m=+330.122575921"
Sep 09 04:03:29.195479 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:03:29.195438    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw" podStartSLOduration=2.194266614 podStartE2EDuration="5.195426329s" totalImagesPullingTime="3.001159715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:03:29.193783809 +0000 UTC m=+330.143474456" watchObservedRunningTime="2026-09-09 04:03:29.195426329 +0000 UTC m=+330.145116983"
Sep 09 04:04:00.156162 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:00.156127    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gnhz"
Sep 09 04:04:00.159532 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:00.159510    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxrqw"
Sep 09 04:04:18.174368 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.174335    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"]
Sep 09 04:04:18.174844 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.174667    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d6c414b-8b72-48e9-974d-a7f7c4839588" containerName="discovery"
Sep 09 04:04:18.174844 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.174681    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d6c414b-8b72-48e9-974d-a7f7c4839588" containerName="discovery"
Sep 09 04:04:18.174844 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.174737    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d6c414b-8b72-48e9-974d-a7f7c4839588" containerName="discovery"
Sep 09 04:04:18.181312 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.181294    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"
Sep 09 04:04:18.184864 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.184838    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wqsjv\""
Sep 09 04:04:18.184974 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.184841    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 04:04:18.184974 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.184853    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 04:04:18.185993 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.185971    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"]
Sep 09 04:04:18.272057 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.272024    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"]
Sep 09 04:04:18.275391 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.275376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"
Sep 09 04:04:18.283466 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.283442    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"]
Sep 09 04:04:18.302661 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.302639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nqsp\" (UniqueName: \"kubernetes.io/projected/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-kube-api-access-2nqsp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc\" (UID: \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"
Sep 09 04:04:18.302786 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.302673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc\" (UID: \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"
Sep 09 04:04:18.302786 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.302698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc\" (UID: \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"
Sep 09 04:04:18.372507 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.372480    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"]
Sep 09 04:04:18.375874 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.375860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"
Sep 09 04:04:18.384376 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.384355    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"]
Sep 09 04:04:18.403340 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.403320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc\" (UID: \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"
Sep 09 04:04:18.403441 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.403348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc\" (UID: \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"
Sep 09 04:04:18.403441 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.403380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wcb5\" (UniqueName: \"kubernetes.io/projected/9ee2b93f-027f-4edd-a680-a6d512935136-kube-api-access-9wcb5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8\" (UID: \"9ee2b93f-027f-4edd-a680-a6d512935136\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"
Sep 09 04:04:18.403441 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.403399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ee2b93f-027f-4edd-a680-a6d512935136-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8\" (UID: \"9ee2b93f-027f-4edd-a680-a6d512935136\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"
Sep 09 04:04:18.403543 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.403473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ee2b93f-027f-4edd-a680-a6d512935136-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8\" (UID: \"9ee2b93f-027f-4edd-a680-a6d512935136\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"
Sep 09 04:04:18.403543 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.403506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nqsp\" (UniqueName: \"kubernetes.io/projected/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-kube-api-access-2nqsp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc\" (UID: \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"
Sep 09 04:04:18.403702 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.403683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc\" (UID: \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"
Sep 09 04:04:18.403833 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.403810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc\" (UID: \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"
Sep 09 04:04:18.412030 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.412001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nqsp\" (UniqueName: \"kubernetes.io/projected/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-kube-api-access-2nqsp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc\" (UID: \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"
Sep 09 04:04:18.481754 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.481694    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"]
Sep 09 04:04:18.485185 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.485171    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"
Sep 09 04:04:18.491181 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.491159    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"
Sep 09 04:04:18.497274 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.497250    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"]
Sep 09 04:04:18.504103 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.504080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpl6k\" (UniqueName: \"kubernetes.io/projected/50a03cbc-6808-499a-8b14-ac8ebf10a126-kube-api-access-hpl6k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4\" (UID: \"50a03cbc-6808-499a-8b14-ac8ebf10a126\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"
Sep 09 04:04:18.504200 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.504171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ee2b93f-027f-4edd-a680-a6d512935136-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8\" (UID: \"9ee2b93f-027f-4edd-a680-a6d512935136\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"
Sep 09 04:04:18.504254 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.504208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a03cbc-6808-499a-8b14-ac8ebf10a126-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4\" (UID: \"50a03cbc-6808-499a-8b14-ac8ebf10a126\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"
Sep 09 04:04:18.504311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.504254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a03cbc-6808-499a-8b14-ac8ebf10a126-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4\" (UID: \"50a03cbc-6808-499a-8b14-ac8ebf10a126\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"
Sep 09 04:04:18.504311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.504284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wcb5\" (UniqueName: \"kubernetes.io/projected/9ee2b93f-027f-4edd-a680-a6d512935136-kube-api-access-9wcb5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8\" (UID: \"9ee2b93f-027f-4edd-a680-a6d512935136\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"
Sep 09 04:04:18.504416 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.504313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ee2b93f-027f-4edd-a680-a6d512935136-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8\" (UID: \"9ee2b93f-027f-4edd-a680-a6d512935136\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"
Sep 09 04:04:18.504621 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.504575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ee2b93f-027f-4edd-a680-a6d512935136-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8\" (UID: \"9ee2b93f-027f-4edd-a680-a6d512935136\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"
Sep 09 04:04:18.504727 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.504654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ee2b93f-027f-4edd-a680-a6d512935136-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8\" (UID: \"9ee2b93f-027f-4edd-a680-a6d512935136\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"
Sep 09 04:04:18.514462 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.514398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wcb5\" (UniqueName: \"kubernetes.io/projected/9ee2b93f-027f-4edd-a680-a6d512935136-kube-api-access-9wcb5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8\" (UID: \"9ee2b93f-027f-4edd-a680-a6d512935136\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"
Sep 09 04:04:18.584232 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.584194    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"
Sep 09 04:04:18.604699 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.604673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8l6\" (UniqueName: \"kubernetes.io/projected/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-kube-api-access-cb8l6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw\" (UID: \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"
Sep 09 04:04:18.604828 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.604731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a03cbc-6808-499a-8b14-ac8ebf10a126-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4\" (UID: \"50a03cbc-6808-499a-8b14-ac8ebf10a126\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"
Sep 09 04:04:18.604828 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.604801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpl6k\" (UniqueName: \"kubernetes.io/projected/50a03cbc-6808-499a-8b14-ac8ebf10a126-kube-api-access-hpl6k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4\" (UID: \"50a03cbc-6808-499a-8b14-ac8ebf10a126\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"
Sep 09 04:04:18.604909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.604839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw\" (UID: \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"
Sep 09 04:04:18.604909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.604874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw\" (UID: \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"
Sep 09 04:04:18.604971 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.604936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a03cbc-6808-499a-8b14-ac8ebf10a126-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4\" (UID: \"50a03cbc-6808-499a-8b14-ac8ebf10a126\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"
Sep 09 04:04:18.605130 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.605107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a03cbc-6808-499a-8b14-ac8ebf10a126-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4\" (UID: \"50a03cbc-6808-499a-8b14-ac8ebf10a126\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"
Sep 09 04:04:18.605273 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.605244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a03cbc-6808-499a-8b14-ac8ebf10a126-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4\" (UID: \"50a03cbc-6808-499a-8b14-ac8ebf10a126\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"
Sep 09 04:04:18.613106 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.613084    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpl6k\" (UniqueName: \"kubernetes.io/projected/50a03cbc-6808-499a-8b14-ac8ebf10a126-kube-api-access-hpl6k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4\" (UID: \"50a03cbc-6808-499a-8b14-ac8ebf10a126\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"
Sep 09 04:04:18.684891 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.684857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"
Sep 09 04:04:18.705694 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.705666    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw\" (UID: \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"
Sep 09 04:04:18.705885 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.705822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8l6\" (UniqueName: \"kubernetes.io/projected/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-kube-api-access-cb8l6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw\" (UID: \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"
Sep 09 04:04:18.705968 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.705933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw\" (UID: \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"
Sep 09 04:04:18.706111 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.706087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw\" (UID: \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"
Sep 09 04:04:18.706255 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.706236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw\" (UID: \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"
Sep 09 04:04:18.715288 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.715260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8l6\" (UniqueName: \"kubernetes.io/projected/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-kube-api-access-cb8l6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw\" (UID: \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"
Sep 09 04:04:18.766224 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.766195    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"]
Sep 09 04:04:18.768307 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:04:18.768283    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21fe2284_a313_4b86_ac1d_7975a1a0f9b4.slice/crio-8adabdba217aff4c9b07c25f201dc4c2857d73aa181d93fc76940e8dea1b7c18 WatchSource:0}: Error finding container 8adabdba217aff4c9b07c25f201dc4c2857d73aa181d93fc76940e8dea1b7c18: Status 404 returned error can't find the container with id 8adabdba217aff4c9b07c25f201dc4c2857d73aa181d93fc76940e8dea1b7c18
Sep 09 04:04:18.793951 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.793923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"
Sep 09 04:04:18.871488 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.871447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"]
Sep 09 04:04:18.872944 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:04:18.872918    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee2b93f_027f_4edd_a680_a6d512935136.slice/crio-74efb1a55d9b7a9ab17efb0b30dc5770c26c759f7f0825d7056b781a70442cea WatchSource:0}: Error finding container 74efb1a55d9b7a9ab17efb0b30dc5770c26c759f7f0825d7056b781a70442cea: Status 404 returned error can't find the container with id 74efb1a55d9b7a9ab17efb0b30dc5770c26c759f7f0825d7056b781a70442cea
Sep 09 04:04:18.964790 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:18.964740    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"]
Sep 09 04:04:18.966227 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:04:18.966206    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a03cbc_6808_499a_8b14_ac8ebf10a126.slice/crio-b4539161ed379ef912775f384106a657a3b786ff18e0fee64032568fdfcfc1e7 WatchSource:0}: Error finding container b4539161ed379ef912775f384106a657a3b786ff18e0fee64032568fdfcfc1e7: Status 404 returned error can't find the container with id b4539161ed379ef912775f384106a657a3b786ff18e0fee64032568fdfcfc1e7
Sep 09 04:04:19.066964 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:19.066937    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"]
Sep 09 04:04:19.083066 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:04:19.083040    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ec6c54_56b3_4f4b_9c00_5eb6b4fd3c1f.slice/crio-2b78fe076c60a7c212baf39bdf6b72e46bf24855b4fcadfd4a90dc1b4d5a87f5 WatchSource:0}: Error finding container 2b78fe076c60a7c212baf39bdf6b72e46bf24855b4fcadfd4a90dc1b4d5a87f5: Status 404 returned error can't find the container with id 2b78fe076c60a7c212baf39bdf6b72e46bf24855b4fcadfd4a90dc1b4d5a87f5
Sep 09 04:04:19.376675 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:19.376643    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ee2b93f-027f-4edd-a680-a6d512935136" containerID="e870853c93e5fdf9e38d47e7be5d1623d0110eebe9628704ffc546a573e053dd" exitCode=0
Sep 09 04:04:19.377096 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:19.376747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8" event={"ID":"9ee2b93f-027f-4edd-a680-a6d512935136","Type":"ContainerDied","Data":"e870853c93e5fdf9e38d47e7be5d1623d0110eebe9628704ffc546a573e053dd"}
Sep 09 04:04:19.377096 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:19.376817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8" event={"ID":"9ee2b93f-027f-4edd-a680-a6d512935136","Type":"ContainerStarted","Data":"74efb1a55d9b7a9ab17efb0b30dc5770c26c759f7f0825d7056b781a70442cea"}
Sep 09 04:04:19.378374 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:19.378335    2580 generic.go:356] "Generic (PLEG): container finished" podID="24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f" containerID="4fe384d45fe092e045bdc616b480e2065da413f84c139a3c4bb9c86b2682193d" exitCode=0
Sep 09 04:04:19.378481 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:19.378413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw" event={"ID":"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f","Type":"ContainerDied","Data":"4fe384d45fe092e045bdc616b480e2065da413f84c139a3c4bb9c86b2682193d"}
Sep 09 04:04:19.378481 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:19.378447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw" event={"ID":"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f","Type":"ContainerStarted","Data":"2b78fe076c60a7c212baf39bdf6b72e46bf24855b4fcadfd4a90dc1b4d5a87f5"}
Sep 09 04:04:19.380255 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:19.380234    2580 generic.go:356] "Generic (PLEG): container finished" podID="50a03cbc-6808-499a-8b14-ac8ebf10a126" containerID="4ed20c7ec1d23eaea54f061c103a0a75466f502b7a6ba4b4b29fd989754a707d" exitCode=0
Sep 09 04:04:19.380353 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:19.380313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4" event={"ID":"50a03cbc-6808-499a-8b14-ac8ebf10a126","Type":"ContainerDied","Data":"4ed20c7ec1d23eaea54f061c103a0a75466f502b7a6ba4b4b29fd989754a707d"}
Sep 09 04:04:19.380353 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:19.380342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4" event={"ID":"50a03cbc-6808-499a-8b14-ac8ebf10a126","Type":"ContainerStarted","Data":"b4539161ed379ef912775f384106a657a3b786ff18e0fee64032568fdfcfc1e7"}
Sep 09 04:04:19.382046 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:19.382028    2580 generic.go:356] "Generic (PLEG): container finished" podID="21fe2284-a313-4b86-ac1d-7975a1a0f9b4" containerID="26a4e67891eb1af8811b13d83a1af7aeee52631a566548791f087ba432ceb06f" exitCode=0
Sep 09 04:04:19.382130 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:19.382081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc" event={"ID":"21fe2284-a313-4b86-ac1d-7975a1a0f9b4","Type":"ContainerDied","Data":"26a4e67891eb1af8811b13d83a1af7aeee52631a566548791f087ba432ceb06f"}
Sep 09 04:04:19.382130 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:19.382111    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc" event={"ID":"21fe2284-a313-4b86-ac1d-7975a1a0f9b4","Type":"ContainerStarted","Data":"8adabdba217aff4c9b07c25f201dc4c2857d73aa181d93fc76940e8dea1b7c18"}
Sep 09 04:04:20.388470 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:20.388443    2580 generic.go:356] "Generic (PLEG): container finished" podID="50a03cbc-6808-499a-8b14-ac8ebf10a126" containerID="7e2f750a6f38cb3331ee5e93f85cb714aa80ec66484a329b177d3817b5491372" exitCode=0
Sep 09 04:04:20.388861 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:20.388576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4" event={"ID":"50a03cbc-6808-499a-8b14-ac8ebf10a126","Type":"ContainerDied","Data":"7e2f750a6f38cb3331ee5e93f85cb714aa80ec66484a329b177d3817b5491372"}
Sep 09 04:04:20.390706 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:20.390685    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ee2b93f-027f-4edd-a680-a6d512935136" containerID="c02a77de17ba5a4748e2004f55ab2a68b1647ea33ec6518bdb2551748d087f1c" exitCode=0
Sep 09 04:04:20.390811 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:20.390727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8" event={"ID":"9ee2b93f-027f-4edd-a680-a6d512935136","Type":"ContainerDied","Data":"c02a77de17ba5a4748e2004f55ab2a68b1647ea33ec6518bdb2551748d087f1c"}
Sep 09 04:04:21.397648 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:21.397614    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ee2b93f-027f-4edd-a680-a6d512935136" containerID="afd978bcd0448c94603bf7e6d3114e702c90f5a8cd16dee73bac613ced0ddf3e" exitCode=0
Sep 09 04:04:21.398089 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:21.397698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8" event={"ID":"9ee2b93f-027f-4edd-a680-a6d512935136","Type":"ContainerDied","Data":"afd978bcd0448c94603bf7e6d3114e702c90f5a8cd16dee73bac613ced0ddf3e"}
Sep 09 04:04:21.399496 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:21.399476    2580 generic.go:356] "Generic (PLEG): container finished" podID="24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f" containerID="8aaba2c8c653db3bae4d6c32867144a27b99c192f01e7fa00439c2af521f448f" exitCode=0
Sep 09 04:04:21.399588 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:21.399556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw" event={"ID":"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f","Type":"ContainerDied","Data":"8aaba2c8c653db3bae4d6c32867144a27b99c192f01e7fa00439c2af521f448f"}
Sep 09 04:04:21.401685 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:21.401664    2580 generic.go:356] "Generic (PLEG): container finished" podID="50a03cbc-6808-499a-8b14-ac8ebf10a126" containerID="e73e5f972dfc44c540c673a5de59954154cd41df19e4350e75cf913c17505a3e" exitCode=0
Sep 09 04:04:21.401799 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:21.401735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4" event={"ID":"50a03cbc-6808-499a-8b14-ac8ebf10a126","Type":"ContainerDied","Data":"e73e5f972dfc44c540c673a5de59954154cd41df19e4350e75cf913c17505a3e"}
Sep 09 04:04:21.403437 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:21.403421    2580 generic.go:356] "Generic (PLEG): container finished" podID="21fe2284-a313-4b86-ac1d-7975a1a0f9b4" containerID="90050d4a23557297e9d5c74de235c2bfe199a7cd00ccea374ef91567ebfaf64f" exitCode=0
Sep 09 04:04:21.403525 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:21.403460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc" event={"ID":"21fe2284-a313-4b86-ac1d-7975a1a0f9b4","Type":"ContainerDied","Data":"90050d4a23557297e9d5c74de235c2bfe199a7cd00ccea374ef91567ebfaf64f"}
Sep 09 04:04:22.409724 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.409689    2580 generic.go:356] "Generic (PLEG): container finished" podID="24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f" containerID="6597e99365416a8294473cb30562a0a65ae89cf10d430a0bdfb40d77305121f7" exitCode=0
Sep 09 04:04:22.410137 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.409790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw" event={"ID":"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f","Type":"ContainerDied","Data":"6597e99365416a8294473cb30562a0a65ae89cf10d430a0bdfb40d77305121f7"}
Sep 09 04:04:22.411746 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.411719    2580 generic.go:356] "Generic (PLEG): container finished" podID="21fe2284-a313-4b86-ac1d-7975a1a0f9b4" containerID="9910306a2290d36c35e63bbcd0045e98baabd1e131f76b39a9fc97c2d28a24a1" exitCode=0
Sep 09 04:04:22.411864 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.411744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc" event={"ID":"21fe2284-a313-4b86-ac1d-7975a1a0f9b4","Type":"ContainerDied","Data":"9910306a2290d36c35e63bbcd0045e98baabd1e131f76b39a9fc97c2d28a24a1"}
Sep 09 04:04:22.536858 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.536838    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"
Sep 09 04:04:22.574876 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.574851    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"
Sep 09 04:04:22.638042 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.638005    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ee2b93f-027f-4edd-a680-a6d512935136-util\") pod \"9ee2b93f-027f-4edd-a680-a6d512935136\" (UID: \"9ee2b93f-027f-4edd-a680-a6d512935136\") "
Sep 09 04:04:22.638225 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.638072    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a03cbc-6808-499a-8b14-ac8ebf10a126-bundle\") pod \"50a03cbc-6808-499a-8b14-ac8ebf10a126\" (UID: \"50a03cbc-6808-499a-8b14-ac8ebf10a126\") "
Sep 09 04:04:22.638225 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.638110    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a03cbc-6808-499a-8b14-ac8ebf10a126-util\") pod \"50a03cbc-6808-499a-8b14-ac8ebf10a126\" (UID: \"50a03cbc-6808-499a-8b14-ac8ebf10a126\") "
Sep 09 04:04:22.638225 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.638201    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpl6k\" (UniqueName: \"kubernetes.io/projected/50a03cbc-6808-499a-8b14-ac8ebf10a126-kube-api-access-hpl6k\") pod \"50a03cbc-6808-499a-8b14-ac8ebf10a126\" (UID: \"50a03cbc-6808-499a-8b14-ac8ebf10a126\") "
Sep 09 04:04:22.638385 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.638244    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wcb5\" (UniqueName: \"kubernetes.io/projected/9ee2b93f-027f-4edd-a680-a6d512935136-kube-api-access-9wcb5\") pod \"9ee2b93f-027f-4edd-a680-a6d512935136\" (UID: \"9ee2b93f-027f-4edd-a680-a6d512935136\") "
Sep 09 04:04:22.638385 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.638270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ee2b93f-027f-4edd-a680-a6d512935136-bundle\") pod \"9ee2b93f-027f-4edd-a680-a6d512935136\" (UID: \"9ee2b93f-027f-4edd-a680-a6d512935136\") "
Sep 09 04:04:22.639113 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.638937    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a03cbc-6808-499a-8b14-ac8ebf10a126-bundle" (OuterVolumeSpecName: "bundle") pod "50a03cbc-6808-499a-8b14-ac8ebf10a126" (UID: "50a03cbc-6808-499a-8b14-ac8ebf10a126"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:04:22.639113 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.638944    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee2b93f-027f-4edd-a680-a6d512935136-bundle" (OuterVolumeSpecName: "bundle") pod "9ee2b93f-027f-4edd-a680-a6d512935136" (UID: "9ee2b93f-027f-4edd-a680-a6d512935136"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:04:22.640600 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.640571    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee2b93f-027f-4edd-a680-a6d512935136-kube-api-access-9wcb5" (OuterVolumeSpecName: "kube-api-access-9wcb5") pod "9ee2b93f-027f-4edd-a680-a6d512935136" (UID: "9ee2b93f-027f-4edd-a680-a6d512935136"). InnerVolumeSpecName "kube-api-access-9wcb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:04:22.640600 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.640578    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a03cbc-6808-499a-8b14-ac8ebf10a126-kube-api-access-hpl6k" (OuterVolumeSpecName: "kube-api-access-hpl6k") pod "50a03cbc-6808-499a-8b14-ac8ebf10a126" (UID: "50a03cbc-6808-499a-8b14-ac8ebf10a126"). InnerVolumeSpecName "kube-api-access-hpl6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:04:22.644027 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.643996    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee2b93f-027f-4edd-a680-a6d512935136-util" (OuterVolumeSpecName: "util") pod "9ee2b93f-027f-4edd-a680-a6d512935136" (UID: "9ee2b93f-027f-4edd-a680-a6d512935136"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:04:22.645796 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.645752    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a03cbc-6808-499a-8b14-ac8ebf10a126-util" (OuterVolumeSpecName: "util") pod "50a03cbc-6808-499a-8b14-ac8ebf10a126" (UID: "50a03cbc-6808-499a-8b14-ac8ebf10a126"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:04:22.739943 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.739851    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpl6k\" (UniqueName: \"kubernetes.io/projected/50a03cbc-6808-499a-8b14-ac8ebf10a126-kube-api-access-hpl6k\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:22.739943 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.739889    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wcb5\" (UniqueName: \"kubernetes.io/projected/9ee2b93f-027f-4edd-a680-a6d512935136-kube-api-access-9wcb5\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:22.739943 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.739900    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ee2b93f-027f-4edd-a680-a6d512935136-bundle\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:22.739943 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.739913    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ee2b93f-027f-4edd-a680-a6d512935136-util\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:22.739943 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.739924    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a03cbc-6808-499a-8b14-ac8ebf10a126-bundle\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:22.739943 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:22.739935    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a03cbc-6808-499a-8b14-ac8ebf10a126-util\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:23.418182 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.418147    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8"
Sep 09 04:04:23.418182 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.418161    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxjc8" event={"ID":"9ee2b93f-027f-4edd-a680-a6d512935136","Type":"ContainerDied","Data":"74efb1a55d9b7a9ab17efb0b30dc5770c26c759f7f0825d7056b781a70442cea"}
Sep 09 04:04:23.418712 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.418207    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74efb1a55d9b7a9ab17efb0b30dc5770c26c759f7f0825d7056b781a70442cea"
Sep 09 04:04:23.420126 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.420112    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4"
Sep 09 04:04:23.420311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.420141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq9f4" event={"ID":"50a03cbc-6808-499a-8b14-ac8ebf10a126","Type":"ContainerDied","Data":"b4539161ed379ef912775f384106a657a3b786ff18e0fee64032568fdfcfc1e7"}
Sep 09 04:04:23.420311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.420185    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4539161ed379ef912775f384106a657a3b786ff18e0fee64032568fdfcfc1e7"
Sep 09 04:04:23.571419 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.571397    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"
Sep 09 04:04:23.573420 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.573402    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"
Sep 09 04:04:23.647919 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.647885    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-bundle\") pod \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\" (UID: \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\") "
Sep 09 04:04:23.647919 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.647919    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-bundle\") pod \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\" (UID: \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\") "
Sep 09 04:04:23.648112 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.647934    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-util\") pod \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\" (UID: \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\") "
Sep 09 04:04:23.648112 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.647955    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb8l6\" (UniqueName: \"kubernetes.io/projected/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-kube-api-access-cb8l6\") pod \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\" (UID: \"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f\") "
Sep 09 04:04:23.648112 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.648026    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nqsp\" (UniqueName: \"kubernetes.io/projected/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-kube-api-access-2nqsp\") pod \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\" (UID: \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\") "
Sep 09 04:04:23.648112 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.648041    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-util\") pod \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\" (UID: \"21fe2284-a313-4b86-ac1d-7975a1a0f9b4\") "
Sep 09 04:04:23.648976 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.648494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-bundle" (OuterVolumeSpecName: "bundle") pod "21fe2284-a313-4b86-ac1d-7975a1a0f9b4" (UID: "21fe2284-a313-4b86-ac1d-7975a1a0f9b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:04:23.648976 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.648538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-bundle" (OuterVolumeSpecName: "bundle") pod "24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f" (UID: "24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:04:23.650265 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.650239    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-kube-api-access-cb8l6" (OuterVolumeSpecName: "kube-api-access-cb8l6") pod "24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f" (UID: "24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f"). InnerVolumeSpecName "kube-api-access-cb8l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:04:23.650667 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.650639    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-kube-api-access-2nqsp" (OuterVolumeSpecName: "kube-api-access-2nqsp") pod "21fe2284-a313-4b86-ac1d-7975a1a0f9b4" (UID: "21fe2284-a313-4b86-ac1d-7975a1a0f9b4"). InnerVolumeSpecName "kube-api-access-2nqsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:04:23.653788 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.653741    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-util" (OuterVolumeSpecName: "util") pod "24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f" (UID: "24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:04:23.655357 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.655321    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-util" (OuterVolumeSpecName: "util") pod "21fe2284-a313-4b86-ac1d-7975a1a0f9b4" (UID: "21fe2284-a313-4b86-ac1d-7975a1a0f9b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:04:23.749103 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.749032    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nqsp\" (UniqueName: \"kubernetes.io/projected/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-kube-api-access-2nqsp\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:23.749103 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.749060    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-util\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:23.749103 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.749070    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-bundle\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:23.749103 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.749078    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21fe2284-a313-4b86-ac1d-7975a1a0f9b4-bundle\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:23.749103 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.749085    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-util\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:23.749103 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:23.749093    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb8l6\" (UniqueName: \"kubernetes.io/projected/24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f-kube-api-access-cb8l6\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:24.426488 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:24.426453    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw"
Sep 09 04:04:24.426929 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:24.426455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fsbkw" event={"ID":"24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f","Type":"ContainerDied","Data":"2b78fe076c60a7c212baf39bdf6b72e46bf24855b4fcadfd4a90dc1b4d5a87f5"}
Sep 09 04:04:24.426929 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:24.426578    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b78fe076c60a7c212baf39bdf6b72e46bf24855b4fcadfd4a90dc1b4d5a87f5"
Sep 09 04:04:24.428543 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:24.428520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc" event={"ID":"21fe2284-a313-4b86-ac1d-7975a1a0f9b4","Type":"ContainerDied","Data":"8adabdba217aff4c9b07c25f201dc4c2857d73aa181d93fc76940e8dea1b7c18"}
Sep 09 04:04:24.428543 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:24.428534    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bcrdc"
Sep 09 04:04:24.428689 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:24.428555    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8adabdba217aff4c9b07c25f201dc4c2857d73aa181d93fc76940e8dea1b7c18"
Sep 09 04:04:30.078602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078564    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bxj4t"]
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078862    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee2b93f-027f-4edd-a680-a6d512935136" containerName="pull"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078874    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee2b93f-027f-4edd-a680-a6d512935136" containerName="pull"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078884    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee2b93f-027f-4edd-a680-a6d512935136" containerName="extract"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078889    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee2b93f-027f-4edd-a680-a6d512935136" containerName="extract"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078897    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a03cbc-6808-499a-8b14-ac8ebf10a126" containerName="util"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078902    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a03cbc-6808-499a-8b14-ac8ebf10a126" containerName="util"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078910    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21fe2284-a313-4b86-ac1d-7975a1a0f9b4" containerName="util"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078916    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="21fe2284-a313-4b86-ac1d-7975a1a0f9b4" containerName="util"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078922    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f" containerName="pull"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078927    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f" containerName="pull"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078935    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a03cbc-6808-499a-8b14-ac8ebf10a126" containerName="extract"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078939    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a03cbc-6808-499a-8b14-ac8ebf10a126" containerName="extract"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078945    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f" containerName="extract"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078950    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f" containerName="extract"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078957    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21fe2284-a313-4b86-ac1d-7975a1a0f9b4" containerName="extract"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078961    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="21fe2284-a313-4b86-ac1d-7975a1a0f9b4" containerName="extract"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078968    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a03cbc-6808-499a-8b14-ac8ebf10a126" containerName="pull"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078972    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a03cbc-6808-499a-8b14-ac8ebf10a126" containerName="pull"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078978    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f" containerName="util"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078983    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f" containerName="util"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078992    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21fe2284-a313-4b86-ac1d-7975a1a0f9b4" containerName="pull"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.078997    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="21fe2284-a313-4b86-ac1d-7975a1a0f9b4" containerName="pull"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.079002    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee2b93f-027f-4edd-a680-a6d512935136" containerName="util"
Sep 09 04:04:30.079006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.079007    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee2b93f-027f-4edd-a680-a6d512935136" containerName="util"
Sep 09 04:04:30.079718 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.079050    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50a03cbc-6808-499a-8b14-ac8ebf10a126" containerName="extract"
Sep 09 04:04:30.079718 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.079057    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee2b93f-027f-4edd-a680-a6d512935136" containerName="extract"
Sep 09 04:04:30.079718 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.079063    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ec6c54-56b3-4f4b-9c00-5eb6b4fd3c1f" containerName="extract"
Sep 09 04:04:30.079718 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.079068    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="21fe2284-a313-4b86-ac1d-7975a1a0f9b4" containerName="extract"
Sep 09 04:04:30.083067 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.083051    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-bxj4t"
Sep 09 04:04:30.085591 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.085570    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 04:04:30.085716 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.085653    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 04:04:30.085716 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.085665    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jc622\""
Sep 09 04:04:30.092746 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.092719    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bxj4t"]
Sep 09 04:04:30.203990 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.203958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlv6\" (UniqueName: \"kubernetes.io/projected/c7cea49b-35fd-455c-9e83-f133121b85c6-kube-api-access-9xlv6\") pod \"authorino-operator-76b478bd78-bxj4t\" (UID: \"c7cea49b-35fd-455c-9e83-f133121b85c6\") " pod="kuadrant-system/authorino-operator-76b478bd78-bxj4t"
Sep 09 04:04:30.305273 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.305236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlv6\" (UniqueName: \"kubernetes.io/projected/c7cea49b-35fd-455c-9e83-f133121b85c6-kube-api-access-9xlv6\") pod \"authorino-operator-76b478bd78-bxj4t\" (UID: \"c7cea49b-35fd-455c-9e83-f133121b85c6\") " pod="kuadrant-system/authorino-operator-76b478bd78-bxj4t"
Sep 09 04:04:30.313439 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.313417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlv6\" (UniqueName: \"kubernetes.io/projected/c7cea49b-35fd-455c-9e83-f133121b85c6-kube-api-access-9xlv6\") pod \"authorino-operator-76b478bd78-bxj4t\" (UID: \"c7cea49b-35fd-455c-9e83-f133121b85c6\") " pod="kuadrant-system/authorino-operator-76b478bd78-bxj4t"
Sep 09 04:04:30.392915 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.392883    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-bxj4t"
Sep 09 04:04:30.664796 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:30.664758    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bxj4t"]
Sep 09 04:04:30.666502 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:04:30.666471    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cea49b_35fd_455c_9e83_f133121b85c6.slice/crio-786ae6378fcdedfb7ab73b6bfcf8d808d07581dc96b610a0847f42208fbdb15c WatchSource:0}: Error finding container 786ae6378fcdedfb7ab73b6bfcf8d808d07581dc96b610a0847f42208fbdb15c: Status 404 returned error can't find the container with id 786ae6378fcdedfb7ab73b6bfcf8d808d07581dc96b610a0847f42208fbdb15c
Sep 09 04:04:31.466373 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:31.466333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-bxj4t" event={"ID":"c7cea49b-35fd-455c-9e83-f133121b85c6","Type":"ContainerStarted","Data":"786ae6378fcdedfb7ab73b6bfcf8d808d07581dc96b610a0847f42208fbdb15c"}
Sep 09 04:04:31.789301 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:31.789204    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-749ddb48fc-qzxrp"]
Sep 09 04:04:31.792969 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:31.792945    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:31.802728 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:31.801976    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-749ddb48fc-qzxrp"]
Sep 09 04:04:31.918785 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:31.918731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1abfba12-c8de-4e3f-9041-191668c88c7b-trusted-ca-bundle\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:31.918978 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:31.918802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1abfba12-c8de-4e3f-9041-191668c88c7b-console-config\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:31.918978 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:31.918856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1abfba12-c8de-4e3f-9041-191668c88c7b-service-ca\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:31.918978 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:31.918883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1abfba12-c8de-4e3f-9041-191668c88c7b-oauth-serving-cert\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:31.918978 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:31.918940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1abfba12-c8de-4e3f-9041-191668c88c7b-console-serving-cert\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:31.918978 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:31.918970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gckxz\" (UniqueName: \"kubernetes.io/projected/1abfba12-c8de-4e3f-9041-191668c88c7b-kube-api-access-gckxz\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:31.919240 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:31.919051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1abfba12-c8de-4e3f-9041-191668c88c7b-console-oauth-config\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.019544 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.019508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1abfba12-c8de-4e3f-9041-191668c88c7b-trusted-ca-bundle\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.019742 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.019612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1abfba12-c8de-4e3f-9041-191668c88c7b-console-config\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.019742 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.019712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1abfba12-c8de-4e3f-9041-191668c88c7b-service-ca\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.019742 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.019737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1abfba12-c8de-4e3f-9041-191668c88c7b-oauth-serving-cert\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.019945 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.019818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1abfba12-c8de-4e3f-9041-191668c88c7b-console-serving-cert\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.019945 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.019844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gckxz\" (UniqueName: \"kubernetes.io/projected/1abfba12-c8de-4e3f-9041-191668c88c7b-kube-api-access-gckxz\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.019945 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.019887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1abfba12-c8de-4e3f-9041-191668c88c7b-console-oauth-config\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.020568 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.020531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1abfba12-c8de-4e3f-9041-191668c88c7b-console-config\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.020688 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.020531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1abfba12-c8de-4e3f-9041-191668c88c7b-trusted-ca-bundle\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.020688 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.020534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1abfba12-c8de-4e3f-9041-191668c88c7b-service-ca\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.020798 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.020689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1abfba12-c8de-4e3f-9041-191668c88c7b-oauth-serving-cert\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.023191 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.023168    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1abfba12-c8de-4e3f-9041-191668c88c7b-console-serving-cert\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.023302 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.023228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1abfba12-c8de-4e3f-9041-191668c88c7b-console-oauth-config\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.035719 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.035700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gckxz\" (UniqueName: \"kubernetes.io/projected/1abfba12-c8de-4e3f-9041-191668c88c7b-kube-api-access-gckxz\") pod \"console-749ddb48fc-qzxrp\" (UID: \"1abfba12-c8de-4e3f-9041-191668c88c7b\") " pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.106915 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.106875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:32.271737 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.271700    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"]
Sep 09 04:04:32.276507 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.276486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"
Sep 09 04:04:32.279258 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.279235    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fnk6z\""
Sep 09 04:04:32.283501 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.283480    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"]
Sep 09 04:04:32.392571 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.392540    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-749ddb48fc-qzxrp"]
Sep 09 04:04:32.424604 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.424571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96mv\" (UniqueName: \"kubernetes.io/projected/5145bac9-49d8-43bf-bd2b-6a6bf5d51044-kube-api-access-l96mv\") pod \"limitador-operator-controller-manager-74495c69f8-qlwqc\" (UID: \"5145bac9-49d8-43bf-bd2b-6a6bf5d51044\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"
Sep 09 04:04:32.525300 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.525265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96mv\" (UniqueName: \"kubernetes.io/projected/5145bac9-49d8-43bf-bd2b-6a6bf5d51044-kube-api-access-l96mv\") pod \"limitador-operator-controller-manager-74495c69f8-qlwqc\" (UID: \"5145bac9-49d8-43bf-bd2b-6a6bf5d51044\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"
Sep 09 04:04:32.533523 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.533499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96mv\" (UniqueName: \"kubernetes.io/projected/5145bac9-49d8-43bf-bd2b-6a6bf5d51044-kube-api-access-l96mv\") pod \"limitador-operator-controller-manager-74495c69f8-qlwqc\" (UID: \"5145bac9-49d8-43bf-bd2b-6a6bf5d51044\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"
Sep 09 04:04:32.590520 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:32.590493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"
Sep 09 04:04:32.928718 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:04:32.928685    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abfba12_c8de_4e3f_9041_191668c88c7b.slice/crio-8f482d78d0ad012730ea65641b77aad4e4d3967fb7575310cc46ee0a2be636c4 WatchSource:0}: Error finding container 8f482d78d0ad012730ea65641b77aad4e4d3967fb7575310cc46ee0a2be636c4: Status 404 returned error can't find the container with id 8f482d78d0ad012730ea65641b77aad4e4d3967fb7575310cc46ee0a2be636c4
Sep 09 04:04:33.477027 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:33.477004    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"]
Sep 09 04:04:33.478917 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:04:33.478893    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5145bac9_49d8_43bf_bd2b_6a6bf5d51044.slice/crio-341e54000b4d60ddcf2bf329839e358e631328cbd6798793ebdf211b5dbc5295 WatchSource:0}: Error finding container 341e54000b4d60ddcf2bf329839e358e631328cbd6798793ebdf211b5dbc5295: Status 404 returned error can't find the container with id 341e54000b4d60ddcf2bf329839e358e631328cbd6798793ebdf211b5dbc5295
Sep 09 04:04:33.480973 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:33.480862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-bxj4t" event={"ID":"c7cea49b-35fd-455c-9e83-f133121b85c6","Type":"ContainerStarted","Data":"feba56e0bbb305d4c25a8c25fa665c91c17451d3f1e00ce86e51c18c4f3e3159"}
Sep 09 04:04:33.481042 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:33.480975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-bxj4t"
Sep 09 04:04:33.482459 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:33.482437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-749ddb48fc-qzxrp" event={"ID":"1abfba12-c8de-4e3f-9041-191668c88c7b","Type":"ContainerStarted","Data":"76c9069514667a8f01300a337babca376bc55f0e39f88076311d86ac975c90a6"}
Sep 09 04:04:33.482550 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:33.482470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-749ddb48fc-qzxrp" event={"ID":"1abfba12-c8de-4e3f-9041-191668c88c7b","Type":"ContainerStarted","Data":"8f482d78d0ad012730ea65641b77aad4e4d3967fb7575310cc46ee0a2be636c4"}
Sep 09 04:04:33.499284 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:33.499244    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-bxj4t" podStartSLOduration=0.895069998 podStartE2EDuration="3.499231992s" totalImagesPullingTime="2.604161994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:04:33.497505739 +0000 UTC m=+394.447196405" watchObservedRunningTime="2026-09-09 04:04:33.499231992 +0000 UTC m=+394.448922646"
Sep 09 04:04:33.514697 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:33.514658    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-749ddb48fc-qzxrp" podStartSLOduration=2.51464435 podStartE2EDuration="2.51464435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:04:33.514544248 +0000 UTC m=+394.464234906" watchObservedRunningTime="2026-09-09 04:04:33.51464435 +0000 UTC m=+394.464335005"
Sep 09 04:04:34.489009 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:34.488973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc" event={"ID":"5145bac9-49d8-43bf-bd2b-6a6bf5d51044","Type":"ContainerStarted","Data":"341e54000b4d60ddcf2bf329839e358e631328cbd6798793ebdf211b5dbc5295"}
Sep 09 04:04:36.500007 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.499910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc" event={"ID":"5145bac9-49d8-43bf-bd2b-6a6bf5d51044","Type":"ContainerStarted","Data":"55344f984852c4e10b6c7eb77d6cc252ada6a4947be3e620d91cb152ad7d9272"}
Sep 09 04:04:36.500007 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.499989    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"
Sep 09 04:04:36.509028 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.508996    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"]
Sep 09 04:04:36.512231 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.512215    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:36.514524 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.514506    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vg2bx\""
Sep 09 04:04:36.523643 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.523623    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"]
Sep 09 04:04:36.524207 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.524167    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc" podStartSLOduration=1.895020262 podStartE2EDuration="4.524154746s" totalImagesPullingTime="2.629134484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:04:36.520796244 +0000 UTC m=+397.470486898" watchObservedRunningTime="2026-09-09 04:04:36.524154746 +0000 UTC m=+397.473845400"
Sep 09 04:04:36.661244 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.661213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72shq\" (UniqueName: \"kubernetes.io/projected/24437b63-51e5-489b-9e95-a35af4719e21-kube-api-access-72shq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-chq7m\" (UID: \"24437b63-51e5-489b-9e95-a35af4719e21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:36.661423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.661294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24437b63-51e5-489b-9e95-a35af4719e21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-chq7m\" (UID: \"24437b63-51e5-489b-9e95-a35af4719e21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:36.761927 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.761851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72shq\" (UniqueName: \"kubernetes.io/projected/24437b63-51e5-489b-9e95-a35af4719e21-kube-api-access-72shq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-chq7m\" (UID: \"24437b63-51e5-489b-9e95-a35af4719e21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:36.762043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.761969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24437b63-51e5-489b-9e95-a35af4719e21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-chq7m\" (UID: \"24437b63-51e5-489b-9e95-a35af4719e21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:36.762367 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.762348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24437b63-51e5-489b-9e95-a35af4719e21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-chq7m\" (UID: \"24437b63-51e5-489b-9e95-a35af4719e21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:36.770925 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.770900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72shq\" (UniqueName: \"kubernetes.io/projected/24437b63-51e5-489b-9e95-a35af4719e21-kube-api-access-72shq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-chq7m\" (UID: \"24437b63-51e5-489b-9e95-a35af4719e21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:36.822857 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:36.822834    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:37.097872 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:37.097848    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"]
Sep 09 04:04:37.098730 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:04:37.098711    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24437b63_51e5_489b_9e95_a35af4719e21.slice/crio-7cfd82cfb9d06883e738714873675417e0857595e0a2b2cb6d1da18843426714 WatchSource:0}: Error finding container 7cfd82cfb9d06883e738714873675417e0857595e0a2b2cb6d1da18843426714: Status 404 returned error can't find the container with id 7cfd82cfb9d06883e738714873675417e0857595e0a2b2cb6d1da18843426714
Sep 09 04:04:37.505918 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:37.505887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m" event={"ID":"24437b63-51e5-489b-9e95-a35af4719e21","Type":"ContainerStarted","Data":"7cfd82cfb9d06883e738714873675417e0857595e0a2b2cb6d1da18843426714"}
Sep 09 04:04:38.486305 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:38.486272    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wlj2s"]
Sep 09 04:04:38.489888 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:38.489865    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wlj2s"
Sep 09 04:04:38.492404 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:38.492382    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 09 04:04:38.492511 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:38.492452    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-28nf2\""
Sep 09 04:04:38.506514 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:38.506488    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wlj2s"]
Sep 09 04:04:38.582595 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:38.582548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8p6w\" (UniqueName: \"kubernetes.io/projected/a9bb8845-4fff-4727-af27-fd9ffa666500-kube-api-access-d8p6w\") pod \"dns-operator-controller-manager-7b84cd7c9c-wlj2s\" (UID: \"a9bb8845-4fff-4727-af27-fd9ffa666500\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wlj2s"
Sep 09 04:04:38.683370 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:38.683330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8p6w\" (UniqueName: \"kubernetes.io/projected/a9bb8845-4fff-4727-af27-fd9ffa666500-kube-api-access-d8p6w\") pod \"dns-operator-controller-manager-7b84cd7c9c-wlj2s\" (UID: \"a9bb8845-4fff-4727-af27-fd9ffa666500\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wlj2s"
Sep 09 04:04:38.692217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:38.692190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8p6w\" (UniqueName: \"kubernetes.io/projected/a9bb8845-4fff-4727-af27-fd9ffa666500-kube-api-access-d8p6w\") pod \"dns-operator-controller-manager-7b84cd7c9c-wlj2s\" (UID: \"a9bb8845-4fff-4727-af27-fd9ffa666500\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wlj2s"
Sep 09 04:04:38.803550 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:38.803473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wlj2s"
Sep 09 04:04:39.087027 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:39.086991    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wlj2s"]
Sep 09 04:04:39.088715 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:04:39.088692    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bb8845_4fff_4727_af27_fd9ffa666500.slice/crio-ff99bcc6ec0aa6ed7b8330cf032cc025c8ccda831afe5b66be222b23c98da312 WatchSource:0}: Error finding container ff99bcc6ec0aa6ed7b8330cf032cc025c8ccda831afe5b66be222b23c98da312: Status 404 returned error can't find the container with id ff99bcc6ec0aa6ed7b8330cf032cc025c8ccda831afe5b66be222b23c98da312
Sep 09 04:04:39.522477 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:39.522442    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wlj2s" event={"ID":"a9bb8845-4fff-4727-af27-fd9ffa666500","Type":"ContainerStarted","Data":"ff99bcc6ec0aa6ed7b8330cf032cc025c8ccda831afe5b66be222b23c98da312"}
Sep 09 04:04:41.536045 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:41.536002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m" event={"ID":"24437b63-51e5-489b-9e95-a35af4719e21","Type":"ContainerStarted","Data":"d7cef72028f6421995950932aec1fa8b1d35ced3680303c4db71f2f65e101df4"}
Sep 09 04:04:41.536476 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:41.536077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:41.558316 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:41.558262    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m" podStartSLOduration=1.7112543119999999 podStartE2EDuration="5.558247309s" totalImagesPullingTime="3.846992997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:04:41.555325357 +0000 UTC m=+402.505016011" watchObservedRunningTime="2026-09-09 04:04:41.558247309 +0000 UTC m=+402.507937963"
Sep 09 04:04:42.107860 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:42.107799    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:42.108039 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:42.107949    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:42.112840 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:42.112821    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:42.542519 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:42.542430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wlj2s" event={"ID":"a9bb8845-4fff-4727-af27-fd9ffa666500","Type":"ContainerStarted","Data":"759a1e9863720802fdba568ccc0ceb0c3fbc044a814585eeb81e195873a9e0c1"}
Sep 09 04:04:42.546426 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:42.546405    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-749ddb48fc-qzxrp"
Sep 09 04:04:42.564884 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:42.564834    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wlj2s" podStartSLOduration=1.8419164590000001 podStartE2EDuration="4.564813455s" totalImagesPullingTime="2.722896996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:04:42.56337076 +0000 UTC m=+403.513061413" watchObservedRunningTime="2026-09-09 04:04:42.564813455 +0000 UTC m=+403.514504108"
Sep 09 04:04:42.612399 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:42.612370    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68594544b8-kczrm"]
Sep 09 04:04:43.547985 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:43.547956    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wlj2s"
Sep 09 04:04:43.818327 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:43.818244    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"]
Sep 09 04:04:43.821887 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:43.821867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"
Sep 09 04:04:43.824040 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:43.824021    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 04:04:43.824478 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:43.824460    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gnr75\""
Sep 09 04:04:43.824579 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:43.824503    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 04:04:43.830723 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:43.830699    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"]
Sep 09 04:04:43.925516 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:43.925478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn4lk\" (UniqueName: \"kubernetes.io/projected/ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f-kube-api-access-zn4lk\") pod \"kuadrant-console-plugin-fbdd4f445-xgxqg\" (UID: \"ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"
Sep 09 04:04:43.925693 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:43.925523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xgxqg\" (UID: \"ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"
Sep 09 04:04:43.925693 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:43.925559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xgxqg\" (UID: \"ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"
Sep 09 04:04:44.026597 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:44.026559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn4lk\" (UniqueName: \"kubernetes.io/projected/ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f-kube-api-access-zn4lk\") pod \"kuadrant-console-plugin-fbdd4f445-xgxqg\" (UID: \"ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"
Sep 09 04:04:44.026803 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:44.026607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xgxqg\" (UID: \"ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"
Sep 09 04:04:44.026972 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:44.026941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xgxqg\" (UID: \"ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"
Sep 09 04:04:44.027451 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:44.027426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xgxqg\" (UID: \"ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"
Sep 09 04:04:44.029571 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:44.029550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xgxqg\" (UID: \"ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"
Sep 09 04:04:44.035197 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:44.035173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn4lk\" (UniqueName: \"kubernetes.io/projected/ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f-kube-api-access-zn4lk\") pod \"kuadrant-console-plugin-fbdd4f445-xgxqg\" (UID: \"ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"
Sep 09 04:04:44.132957 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:44.132929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"
Sep 09 04:04:44.403489 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:44.403467    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg"]
Sep 09 04:04:44.405250 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:04:44.405214    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff34cbf2_3a19_4fc7_a35f_cd4bee42be4f.slice/crio-8a8351d907580b38c1808986dc237991da73657b4782975e0cbf1b79199727d7 WatchSource:0}: Error finding container 8a8351d907580b38c1808986dc237991da73657b4782975e0cbf1b79199727d7: Status 404 returned error can't find the container with id 8a8351d907580b38c1808986dc237991da73657b4782975e0cbf1b79199727d7
Sep 09 04:04:44.491720 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:44.491690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-bxj4t"
Sep 09 04:04:44.555107 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:44.555074    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg" event={"ID":"ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f","Type":"ContainerStarted","Data":"8a8351d907580b38c1808986dc237991da73657b4782975e0cbf1b79199727d7"}
Sep 09 04:04:47.509318 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:47.509286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"
Sep 09 04:04:50.590448 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:50.590417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg" event={"ID":"ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f","Type":"ContainerStarted","Data":"acb580922b1f92e7ec365aeb205d9f7e4896168dd2978b35b521759681ed26f1"}
Sep 09 04:04:50.609085 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:50.609038    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xgxqg" podStartSLOduration=1.5461207639999999 podStartE2EDuration="7.609024258s" totalImagesPullingTime="6.062903494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:04:50.608018437 +0000 UTC m=+411.557709092" watchObservedRunningTime="2026-09-09 04:04:50.609024258 +0000 UTC m=+411.558714912"
Sep 09 04:04:52.545359 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:52.545326    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:53.580839 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:53.580804    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"]
Sep 09 04:04:53.585847 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:53.585826    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:53.597085 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:53.597061    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"]
Sep 09 04:04:53.608509 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:53.608480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpq29\" (UniqueName: \"kubernetes.io/projected/590ea242-44f9-4791-a0ae-08705ea98486-kube-api-access-bpq29\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" (UID: \"590ea242-44f9-4791-a0ae-08705ea98486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:53.608662 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:53.608602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/590ea242-44f9-4791-a0ae-08705ea98486-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" (UID: \"590ea242-44f9-4791-a0ae-08705ea98486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:53.709838 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:53.709806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpq29\" (UniqueName: \"kubernetes.io/projected/590ea242-44f9-4791-a0ae-08705ea98486-kube-api-access-bpq29\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" (UID: \"590ea242-44f9-4791-a0ae-08705ea98486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:53.709991 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:53.709872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/590ea242-44f9-4791-a0ae-08705ea98486-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" (UID: \"590ea242-44f9-4791-a0ae-08705ea98486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:53.710188 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:53.710172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/590ea242-44f9-4791-a0ae-08705ea98486-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" (UID: \"590ea242-44f9-4791-a0ae-08705ea98486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:53.720521 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:53.720500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpq29\" (UniqueName: \"kubernetes.io/projected/590ea242-44f9-4791-a0ae-08705ea98486-kube-api-access-bpq29\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" (UID: \"590ea242-44f9-4791-a0ae-08705ea98486\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:53.897114 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:53.897079    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:54.185526 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.184728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"]
Sep 09 04:04:54.557950 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.557864    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-wlj2s"
Sep 09 04:04:54.574193 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.574161    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"]
Sep 09 04:04:54.598374 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.598352    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"]
Sep 09 04:04:54.610042 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.610011    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"]
Sep 09 04:04:54.610261 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.610236    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m" podUID="24437b63-51e5-489b-9e95-a35af4719e21" containerName="manager" containerID="cri-o://d7cef72028f6421995950932aec1fa8b1d35ced3680303c4db71f2f65e101df4" gracePeriod=2
Sep 09 04:04:54.614384 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.614359    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr" container="manager"
Sep 09 04:04:54.621318 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.621295    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"]
Sep 09 04:04:54.623050 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.623015    2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"24437b63-51e5-489b-9e95-a35af4719e21\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-chq7m\": pods \"kuadrant-operator-controller-manager-76f8f686bf-chq7m\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:54.625674 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.625018    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="590ea242-44f9-4791-a0ae-08705ea98486" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:54.626989 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.626961    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-chq7m\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="24437b63-51e5-489b-9e95-a35af4719e21" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:54.632907 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.632884    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"]
Sep 09 04:04:54.633231 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.633214    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590ea242-44f9-4791-a0ae-08705ea98486" containerName="manager"
Sep 09 04:04:54.633231 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.633233    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="590ea242-44f9-4791-a0ae-08705ea98486" containerName="manager"
Sep 09 04:04:54.633363 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.633259    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24437b63-51e5-489b-9e95-a35af4719e21" containerName="manager"
Sep 09 04:04:54.633363 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.633264    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24437b63-51e5-489b-9e95-a35af4719e21" containerName="manager"
Sep 09 04:04:54.633363 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.633326    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="590ea242-44f9-4791-a0ae-08705ea98486" containerName="manager"
Sep 09 04:04:54.633363 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.633338    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="24437b63-51e5-489b-9e95-a35af4719e21" containerName="manager"
Sep 09 04:04:54.636329 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.636312    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"
Sep 09 04:04:54.637386 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.637363    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"]
Sep 09 04:04:54.637603 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.637562    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc" podUID="5145bac9-49d8-43bf-bd2b-6a6bf5d51044" containerName="manager" containerID="cri-o://55344f984852c4e10b6c7eb77d6cc252ada6a4947be3e620d91cb152ad7d9272" gracePeriod=2
Sep 09 04:04:54.649789 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.649748    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"]
Sep 09 04:04:54.652382 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.652358    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"]
Sep 09 04:04:54.657284 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.657262    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-czlk7"]
Sep 09 04:04:54.657617 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.657604    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5145bac9-49d8-43bf-bd2b-6a6bf5d51044" containerName="manager"
Sep 09 04:04:54.657659 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.657619    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5145bac9-49d8-43bf-bd2b-6a6bf5d51044" containerName="manager"
Sep 09 04:04:54.657701 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.657692    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5145bac9-49d8-43bf-bd2b-6a6bf5d51044" containerName="manager"
Sep 09 04:04:54.660680 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.660663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-czlk7"
Sep 09 04:04:54.670201 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.670181    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-czlk7"]
Sep 09 04:04:54.675194 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.675161    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="590ea242-44f9-4791-a0ae-08705ea98486" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:54.677154 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.677127    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-chq7m\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="24437b63-51e5-489b-9e95-a35af4719e21" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:54.678951 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.678925    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qlwqc\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="5145bac9-49d8-43bf-bd2b-6a6bf5d51044" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"
Sep 09 04:04:54.680704 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.680683    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="590ea242-44f9-4791-a0ae-08705ea98486" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:54.682406 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.682374    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-chq7m\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="24437b63-51e5-489b-9e95-a35af4719e21" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:54.715894 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.715863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f460a6c-b1a9-48c6-838a-25e55ede8370-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5vxbk\" (UID: \"7f460a6c-b1a9-48c6-838a-25e55ede8370\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"
Sep 09 04:04:54.716026 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.715929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcsqs\" (UniqueName: \"kubernetes.io/projected/7f460a6c-b1a9-48c6-838a-25e55ede8370-kube-api-access-zcsqs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5vxbk\" (UID: \"7f460a6c-b1a9-48c6-838a-25e55ede8370\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"
Sep 09 04:04:54.716026 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.716000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wl4v\" (UniqueName: \"kubernetes.io/projected/2c51c71e-8575-4a24-8ad9-985e11d84942-kube-api-access-6wl4v\") pod \"limitador-operator-controller-manager-74495c69f8-czlk7\" (UID: \"2c51c71e-8575-4a24-8ad9-985e11d84942\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-czlk7"
Sep 09 04:04:54.816688 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.816615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcsqs\" (UniqueName: \"kubernetes.io/projected/7f460a6c-b1a9-48c6-838a-25e55ede8370-kube-api-access-zcsqs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5vxbk\" (UID: \"7f460a6c-b1a9-48c6-838a-25e55ede8370\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"
Sep 09 04:04:54.816688 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.816660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wl4v\" (UniqueName: \"kubernetes.io/projected/2c51c71e-8575-4a24-8ad9-985e11d84942-kube-api-access-6wl4v\") pod \"limitador-operator-controller-manager-74495c69f8-czlk7\" (UID: \"2c51c71e-8575-4a24-8ad9-985e11d84942\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-czlk7"
Sep 09 04:04:54.816888 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.816705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f460a6c-b1a9-48c6-838a-25e55ede8370-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5vxbk\" (UID: \"7f460a6c-b1a9-48c6-838a-25e55ede8370\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"
Sep 09 04:04:54.817143 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.817121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f460a6c-b1a9-48c6-838a-25e55ede8370-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5vxbk\" (UID: \"7f460a6c-b1a9-48c6-838a-25e55ede8370\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"
Sep 09 04:04:54.833002 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.832948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcsqs\" (UniqueName: \"kubernetes.io/projected/7f460a6c-b1a9-48c6-838a-25e55ede8370-kube-api-access-zcsqs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5vxbk\" (UID: \"7f460a6c-b1a9-48c6-838a-25e55ede8370\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"
Sep 09 04:04:54.834266 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.834228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wl4v\" (UniqueName: \"kubernetes.io/projected/2c51c71e-8575-4a24-8ad9-985e11d84942-kube-api-access-6wl4v\") pod \"limitador-operator-controller-manager-74495c69f8-czlk7\" (UID: \"2c51c71e-8575-4a24-8ad9-985e11d84942\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-czlk7"
Sep 09 04:04:54.875280 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.875259    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:54.877614 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.877591    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qlwqc\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="5145bac9-49d8-43bf-bd2b-6a6bf5d51044" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"
Sep 09 04:04:54.877848 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.877832    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"
Sep 09 04:04:54.879644 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.879625    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="590ea242-44f9-4791-a0ae-08705ea98486" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:54.881571 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.881550    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-chq7m\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="24437b63-51e5-489b-9e95-a35af4719e21" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:54.883479 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.883459    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qlwqc\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="5145bac9-49d8-43bf-bd2b-6a6bf5d51044" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"
Sep 09 04:04:54.885346 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.885327    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="590ea242-44f9-4791-a0ae-08705ea98486" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:54.887363 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:54.887344    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-chq7m\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="24437b63-51e5-489b-9e95-a35af4719e21" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:54.917995 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.917964    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72shq\" (UniqueName: \"kubernetes.io/projected/24437b63-51e5-489b-9e95-a35af4719e21-kube-api-access-72shq\") pod \"24437b63-51e5-489b-9e95-a35af4719e21\" (UID: \"24437b63-51e5-489b-9e95-a35af4719e21\") "
Sep 09 04:04:54.918117 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.918027    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l96mv\" (UniqueName: \"kubernetes.io/projected/5145bac9-49d8-43bf-bd2b-6a6bf5d51044-kube-api-access-l96mv\") pod \"5145bac9-49d8-43bf-bd2b-6a6bf5d51044\" (UID: \"5145bac9-49d8-43bf-bd2b-6a6bf5d51044\") "
Sep 09 04:04:54.918117 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.918087    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24437b63-51e5-489b-9e95-a35af4719e21-extensions-socket-volume\") pod \"24437b63-51e5-489b-9e95-a35af4719e21\" (UID: \"24437b63-51e5-489b-9e95-a35af4719e21\") "
Sep 09 04:04:54.918570 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.918543    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24437b63-51e5-489b-9e95-a35af4719e21-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "24437b63-51e5-489b-9e95-a35af4719e21" (UID: "24437b63-51e5-489b-9e95-a35af4719e21"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:04:54.920384 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.920358    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5145bac9-49d8-43bf-bd2b-6a6bf5d51044-kube-api-access-l96mv" (OuterVolumeSpecName: "kube-api-access-l96mv") pod "5145bac9-49d8-43bf-bd2b-6a6bf5d51044" (UID: "5145bac9-49d8-43bf-bd2b-6a6bf5d51044"). InnerVolumeSpecName "kube-api-access-l96mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:04:54.920384 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:54.920369    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24437b63-51e5-489b-9e95-a35af4719e21-kube-api-access-72shq" (OuterVolumeSpecName: "kube-api-access-72shq") pod "24437b63-51e5-489b-9e95-a35af4719e21" (UID: "24437b63-51e5-489b-9e95-a35af4719e21"). InnerVolumeSpecName "kube-api-access-72shq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:04:55.019113 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.019077    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24437b63-51e5-489b-9e95-a35af4719e21-extensions-socket-volume\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:55.019113 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.019105    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72shq\" (UniqueName: \"kubernetes.io/projected/24437b63-51e5-489b-9e95-a35af4719e21-kube-api-access-72shq\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:55.019113 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.019114    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l96mv\" (UniqueName: \"kubernetes.io/projected/5145bac9-49d8-43bf-bd2b-6a6bf5d51044-kube-api-access-l96mv\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:55.043717 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.043688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"
Sep 09 04:04:55.052797 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.052739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-czlk7"
Sep 09 04:04:55.325724 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.325694    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-czlk7"]
Sep 09 04:04:55.327932 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:04:55.327902    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c51c71e_8575_4a24_8ad9_985e11d84942.slice/crio-ce8d4201fde6cecfb600ed1995b53d492c0a97a9d306c074b6cd46c69f91a93e WatchSource:0}: Error finding container ce8d4201fde6cecfb600ed1995b53d492c0a97a9d306c074b6cd46c69f91a93e: Status 404 returned error can't find the container with id ce8d4201fde6cecfb600ed1995b53d492c0a97a9d306c074b6cd46c69f91a93e
Sep 09 04:04:55.328987 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.328965    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"]
Sep 09 04:04:55.330371 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:04:55.330351    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f460a6c_b1a9_48c6_838a_25e55ede8370.slice/crio-ecc822bc8b8994025d3f3fe9ee9bbde09653a4169ef91d143c75a0fc18ed1d8f WatchSource:0}: Error finding container ecc822bc8b8994025d3f3fe9ee9bbde09653a4169ef91d143c75a0fc18ed1d8f: Status 404 returned error can't find the container with id ecc822bc8b8994025d3f3fe9ee9bbde09653a4169ef91d143c75a0fc18ed1d8f
Sep 09 04:04:55.595910 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.595871    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24437b63-51e5-489b-9e95-a35af4719e21" path="/var/lib/kubelet/pods/24437b63-51e5-489b-9e95-a35af4719e21/volumes"
Sep 09 04:04:55.596282 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.596263    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5145bac9-49d8-43bf-bd2b-6a6bf5d51044" path="/var/lib/kubelet/pods/5145bac9-49d8-43bf-bd2b-6a6bf5d51044/volumes"
Sep 09 04:04:55.621727 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.621689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk" event={"ID":"7f460a6c-b1a9-48c6-838a-25e55ede8370","Type":"ContainerStarted","Data":"11e51b40309f2153859b95960dcebddeb83b1cd0bac83c4721a2d77b5d14113a"}
Sep 09 04:04:55.622113 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.621754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk" event={"ID":"7f460a6c-b1a9-48c6-838a-25e55ede8370","Type":"ContainerStarted","Data":"ecc822bc8b8994025d3f3fe9ee9bbde09653a4169ef91d143c75a0fc18ed1d8f"}
Sep 09 04:04:55.622113 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.621876    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"
Sep 09 04:04:55.623118 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.623084    2580 generic.go:356] "Generic (PLEG): container finished" podID="5145bac9-49d8-43bf-bd2b-6a6bf5d51044" containerID="55344f984852c4e10b6c7eb77d6cc252ada6a4947be3e620d91cb152ad7d9272" exitCode=0
Sep 09 04:04:55.623230 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.623130    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qlwqc"
Sep 09 04:04:55.623230 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.623172    2580 scope.go:117] "RemoveContainer" containerID="55344f984852c4e10b6c7eb77d6cc252ada6a4947be3e620d91cb152ad7d9272"
Sep 09 04:04:55.624676 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.624636    2580 generic.go:356] "Generic (PLEG): container finished" podID="24437b63-51e5-489b-9e95-a35af4719e21" containerID="d7cef72028f6421995950932aec1fa8b1d35ced3680303c4db71f2f65e101df4" exitCode=0
Sep 09 04:04:55.624676 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.624672    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-chq7m"
Sep 09 04:04:55.626896 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.626715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-czlk7" event={"ID":"2c51c71e-8575-4a24-8ad9-985e11d84942","Type":"ContainerStarted","Data":"ff25ba4b901266f3bb039cf1915544d9985c586705c882935bfc2f6912608037"}
Sep 09 04:04:55.626896 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.626791    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-czlk7" event={"ID":"2c51c71e-8575-4a24-8ad9-985e11d84942","Type":"ContainerStarted","Data":"ce8d4201fde6cecfb600ed1995b53d492c0a97a9d306c074b6cd46c69f91a93e"}
Sep 09 04:04:55.626896 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.626824    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-czlk7"
Sep 09 04:04:55.628778 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.628715    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr" podUID="590ea242-44f9-4791-a0ae-08705ea98486" containerName="manager" containerID="cri-o://750bda7209561ceec968ca6568aaf5e376bdfd2259995f1f3d437aa00da2b2ba" gracePeriod=2
Sep 09 04:04:55.632427 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.632412    2580 scope.go:117] "RemoveContainer" containerID="55344f984852c4e10b6c7eb77d6cc252ada6a4947be3e620d91cb152ad7d9272"
Sep 09 04:04:55.632897 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:55.632876    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55344f984852c4e10b6c7eb77d6cc252ada6a4947be3e620d91cb152ad7d9272\": container with ID starting with 55344f984852c4e10b6c7eb77d6cc252ada6a4947be3e620d91cb152ad7d9272 not found: ID does not exist" containerID="55344f984852c4e10b6c7eb77d6cc252ada6a4947be3e620d91cb152ad7d9272"
Sep 09 04:04:55.632984 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.632907    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55344f984852c4e10b6c7eb77d6cc252ada6a4947be3e620d91cb152ad7d9272"} err="failed to get container status \"55344f984852c4e10b6c7eb77d6cc252ada6a4947be3e620d91cb152ad7d9272\": rpc error: code = NotFound desc = could not find container \"55344f984852c4e10b6c7eb77d6cc252ada6a4947be3e620d91cb152ad7d9272\": container with ID starting with 55344f984852c4e10b6c7eb77d6cc252ada6a4947be3e620d91cb152ad7d9272 not found: ID does not exist"
Sep 09 04:04:55.632984 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.632923    2580 scope.go:117] "RemoveContainer" containerID="d7cef72028f6421995950932aec1fa8b1d35ced3680303c4db71f2f65e101df4"
Sep 09 04:04:55.641290 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.641274    2580 scope.go:117] "RemoveContainer" containerID="d7cef72028f6421995950932aec1fa8b1d35ced3680303c4db71f2f65e101df4"
Sep 09 04:04:55.641522 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:55.641507    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7cef72028f6421995950932aec1fa8b1d35ced3680303c4db71f2f65e101df4\": container with ID starting with d7cef72028f6421995950932aec1fa8b1d35ced3680303c4db71f2f65e101df4 not found: ID does not exist" containerID="d7cef72028f6421995950932aec1fa8b1d35ced3680303c4db71f2f65e101df4"
Sep 09 04:04:55.641579 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.641530    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7cef72028f6421995950932aec1fa8b1d35ced3680303c4db71f2f65e101df4"} err="failed to get container status \"d7cef72028f6421995950932aec1fa8b1d35ced3680303c4db71f2f65e101df4\": rpc error: code = NotFound desc = could not find container \"d7cef72028f6421995950932aec1fa8b1d35ced3680303c4db71f2f65e101df4\": container with ID starting with d7cef72028f6421995950932aec1fa8b1d35ced3680303c4db71f2f65e101df4 not found: ID does not exist"
Sep 09 04:04:55.647514 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.647478    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk" podStartSLOduration=1.647466399 podStartE2EDuration="1.647466399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:04:55.646159165 +0000 UTC m=+416.595849821" watchObservedRunningTime="2026-09-09 04:04:55.647466399 +0000 UTC m=+416.597157052"
Sep 09 04:04:55.666009 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.665967    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-czlk7" podStartSLOduration=1.6659551110000002 podStartE2EDuration="1.665955111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:04:55.664234138 +0000 UTC m=+416.613924791" watchObservedRunningTime="2026-09-09 04:04:55.665955111 +0000 UTC m=+416.615645764"
Sep 09 04:04:55.879623 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.879597    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:55.881724 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:55.881703    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="590ea242-44f9-4791-a0ae-08705ea98486" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:55.926023 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.925997    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpq29\" (UniqueName: \"kubernetes.io/projected/590ea242-44f9-4791-a0ae-08705ea98486-kube-api-access-bpq29\") pod \"590ea242-44f9-4791-a0ae-08705ea98486\" (UID: \"590ea242-44f9-4791-a0ae-08705ea98486\") "
Sep 09 04:04:55.926158 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.926064    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/590ea242-44f9-4791-a0ae-08705ea98486-extensions-socket-volume\") pod \"590ea242-44f9-4791-a0ae-08705ea98486\" (UID: \"590ea242-44f9-4791-a0ae-08705ea98486\") "
Sep 09 04:04:55.926443 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.926411    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590ea242-44f9-4791-a0ae-08705ea98486-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "590ea242-44f9-4791-a0ae-08705ea98486" (UID: "590ea242-44f9-4791-a0ae-08705ea98486"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:04:55.928219 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:55.928199    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590ea242-44f9-4791-a0ae-08705ea98486-kube-api-access-bpq29" (OuterVolumeSpecName: "kube-api-access-bpq29") pod "590ea242-44f9-4791-a0ae-08705ea98486" (UID: "590ea242-44f9-4791-a0ae-08705ea98486"). InnerVolumeSpecName "kube-api-access-bpq29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:04:56.026909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:56.026878    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/590ea242-44f9-4791-a0ae-08705ea98486-extensions-socket-volume\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:56.026909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:56.026902    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpq29\" (UniqueName: \"kubernetes.io/projected/590ea242-44f9-4791-a0ae-08705ea98486-kube-api-access-bpq29\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:04:56.642308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:56.642275    2580 generic.go:356] "Generic (PLEG): container finished" podID="590ea242-44f9-4791-a0ae-08705ea98486" containerID="750bda7209561ceec968ca6568aaf5e376bdfd2259995f1f3d437aa00da2b2ba" exitCode=0
Sep 09 04:04:56.642789 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:56.642317    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:56.642789 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:56.642387    2580 scope.go:117] "RemoveContainer" containerID="750bda7209561ceec968ca6568aaf5e376bdfd2259995f1f3d437aa00da2b2ba"
Sep 09 04:04:56.644659 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:56.644637    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="590ea242-44f9-4791-a0ae-08705ea98486" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:56.652009 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:56.651979    2580 scope.go:117] "RemoveContainer" containerID="750bda7209561ceec968ca6568aaf5e376bdfd2259995f1f3d437aa00da2b2ba"
Sep 09 04:04:56.652261 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:56.652243    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750bda7209561ceec968ca6568aaf5e376bdfd2259995f1f3d437aa00da2b2ba\": container with ID starting with 750bda7209561ceec968ca6568aaf5e376bdfd2259995f1f3d437aa00da2b2ba not found: ID does not exist" containerID="750bda7209561ceec968ca6568aaf5e376bdfd2259995f1f3d437aa00da2b2ba"
Sep 09 04:04:56.652307 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:56.652268    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750bda7209561ceec968ca6568aaf5e376bdfd2259995f1f3d437aa00da2b2ba"} err="failed to get container status \"750bda7209561ceec968ca6568aaf5e376bdfd2259995f1f3d437aa00da2b2ba\": rpc error: code = NotFound desc = could not find container \"750bda7209561ceec968ca6568aaf5e376bdfd2259995f1f3d437aa00da2b2ba\": container with ID starting with 750bda7209561ceec968ca6568aaf5e376bdfd2259995f1f3d437aa00da2b2ba not found: ID does not exist"
Sep 09 04:04:56.652896 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:04:56.652877    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-vcnzr\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="590ea242-44f9-4791-a0ae-08705ea98486" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-vcnzr"
Sep 09 04:04:57.595017 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:04:57.594989    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="590ea242-44f9-4791-a0ae-08705ea98486" path="/var/lib/kubelet/pods/590ea242-44f9-4791-a0ae-08705ea98486/volumes"
Sep 09 04:05:06.644673 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:06.644639    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"
Sep 09 04:05:06.645249 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:06.645232    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-czlk7"
Sep 09 04:05:07.634992 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:07.634953    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68594544b8-kczrm" podUID="53fa2568-55bb-4c15-aa30-2cd8527e43cb" containerName="console" containerID="cri-o://1ddb28ba1f256af57f6ea522c58a50a3ce6abe4320095fb19bd6eb1f8f757f88" gracePeriod=15
Sep 09 04:05:07.925413 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:07.925390    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:05:08.026701 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.026666    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-oauth-config\") pod \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") "
Sep 09 04:05:08.026701 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.026707    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lc8h\" (UniqueName: \"kubernetes.io/projected/53fa2568-55bb-4c15-aa30-2cd8527e43cb-kube-api-access-4lc8h\") pod \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") "
Sep 09 04:05:08.026971 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.026728    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-serving-cert\") pod \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") "
Sep 09 04:05:08.026971 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.026750    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-oauth-serving-cert\") pod \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") "
Sep 09 04:05:08.026971 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.026804    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-service-ca\") pod \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") "
Sep 09 04:05:08.026971 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.026866    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-trusted-ca-bundle\") pod \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") "
Sep 09 04:05:08.026971 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.026883    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-config\") pod \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\" (UID: \"53fa2568-55bb-4c15-aa30-2cd8527e43cb\") "
Sep 09 04:05:08.027293 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.027266    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "53fa2568-55bb-4c15-aa30-2cd8527e43cb" (UID: "53fa2568-55bb-4c15-aa30-2cd8527e43cb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:05:08.027353 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.027289    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "53fa2568-55bb-4c15-aa30-2cd8527e43cb" (UID: "53fa2568-55bb-4c15-aa30-2cd8527e43cb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:05:08.027353 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.027302    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-service-ca" (OuterVolumeSpecName: "service-ca") pod "53fa2568-55bb-4c15-aa30-2cd8527e43cb" (UID: "53fa2568-55bb-4c15-aa30-2cd8527e43cb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:05:08.027434 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.027397    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-config" (OuterVolumeSpecName: "console-config") pod "53fa2568-55bb-4c15-aa30-2cd8527e43cb" (UID: "53fa2568-55bb-4c15-aa30-2cd8527e43cb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:05:08.027490 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.027409    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-oauth-serving-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:05:08.027633 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.027617    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:05:08.027694 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.027639    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-trusted-ca-bundle\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:05:08.029193 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.029165    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fa2568-55bb-4c15-aa30-2cd8527e43cb-kube-api-access-4lc8h" (OuterVolumeSpecName: "kube-api-access-4lc8h") pod "53fa2568-55bb-4c15-aa30-2cd8527e43cb" (UID: "53fa2568-55bb-4c15-aa30-2cd8527e43cb"). InnerVolumeSpecName "kube-api-access-4lc8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:05:08.029293 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.029235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "53fa2568-55bb-4c15-aa30-2cd8527e43cb" (UID: "53fa2568-55bb-4c15-aa30-2cd8527e43cb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:05:08.029359 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.029343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "53fa2568-55bb-4c15-aa30-2cd8527e43cb" (UID: "53fa2568-55bb-4c15-aa30-2cd8527e43cb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:05:08.128207 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.128178    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-config\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:05:08.128207 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.128202    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-oauth-config\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:05:08.128207 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.128212    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lc8h\" (UniqueName: \"kubernetes.io/projected/53fa2568-55bb-4c15-aa30-2cd8527e43cb-kube-api-access-4lc8h\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:05:08.128407 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.128220    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53fa2568-55bb-4c15-aa30-2cd8527e43cb-console-serving-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:05:08.709266 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.709235    2580 generic.go:356] "Generic (PLEG): container finished" podID="53fa2568-55bb-4c15-aa30-2cd8527e43cb" containerID="1ddb28ba1f256af57f6ea522c58a50a3ce6abe4320095fb19bd6eb1f8f757f88" exitCode=0
Sep 09 04:05:08.709453 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.709291    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68594544b8-kczrm" event={"ID":"53fa2568-55bb-4c15-aa30-2cd8527e43cb","Type":"ContainerDied","Data":"1ddb28ba1f256af57f6ea522c58a50a3ce6abe4320095fb19bd6eb1f8f757f88"}
Sep 09 04:05:08.709453 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.709293    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68594544b8-kczrm"
Sep 09 04:05:08.709453 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.709330    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68594544b8-kczrm" event={"ID":"53fa2568-55bb-4c15-aa30-2cd8527e43cb","Type":"ContainerDied","Data":"b652f4fc968e6051755cec0ded13acf9c5f59f0125559a93421054279fcb58d5"}
Sep 09 04:05:08.709453 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.709351    2580 scope.go:117] "RemoveContainer" containerID="1ddb28ba1f256af57f6ea522c58a50a3ce6abe4320095fb19bd6eb1f8f757f88"
Sep 09 04:05:08.718858 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.718838    2580 scope.go:117] "RemoveContainer" containerID="1ddb28ba1f256af57f6ea522c58a50a3ce6abe4320095fb19bd6eb1f8f757f88"
Sep 09 04:05:08.719136 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:05:08.719115    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ddb28ba1f256af57f6ea522c58a50a3ce6abe4320095fb19bd6eb1f8f757f88\": container with ID starting with 1ddb28ba1f256af57f6ea522c58a50a3ce6abe4320095fb19bd6eb1f8f757f88 not found: ID does not exist" containerID="1ddb28ba1f256af57f6ea522c58a50a3ce6abe4320095fb19bd6eb1f8f757f88"
Sep 09 04:05:08.719213 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.719148    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ddb28ba1f256af57f6ea522c58a50a3ce6abe4320095fb19bd6eb1f8f757f88"} err="failed to get container status \"1ddb28ba1f256af57f6ea522c58a50a3ce6abe4320095fb19bd6eb1f8f757f88\": rpc error: code = NotFound desc = could not find container \"1ddb28ba1f256af57f6ea522c58a50a3ce6abe4320095fb19bd6eb1f8f757f88\": container with ID starting with 1ddb28ba1f256af57f6ea522c58a50a3ce6abe4320095fb19bd6eb1f8f757f88 not found: ID does not exist"
Sep 09 04:05:08.730228 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.730203    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68594544b8-kczrm"]
Sep 09 04:05:08.734658 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:08.734638    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68594544b8-kczrm"]
Sep 09 04:05:09.594949 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:09.594915    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53fa2568-55bb-4c15-aa30-2cd8527e43cb" path="/var/lib/kubelet/pods/53fa2568-55bb-4c15-aa30-2cd8527e43cb/volumes"
Sep 09 04:05:10.963684 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:10.963648    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"]
Sep 09 04:05:10.964062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:10.963852    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk" podUID="7f460a6c-b1a9-48c6-838a-25e55ede8370" containerName="manager" containerID="cri-o://11e51b40309f2153859b95960dcebddeb83b1cd0bac83c4721a2d77b5d14113a" gracePeriod=10
Sep 09 04:05:11.198587 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.198555    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"]
Sep 09 04:05:11.198910 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.198897    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53fa2568-55bb-4c15-aa30-2cd8527e43cb" containerName="console"
Sep 09 04:05:11.198956 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.198911    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53fa2568-55bb-4c15-aa30-2cd8527e43cb" containerName="console"
Sep 09 04:05:11.198992 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.198963    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53fa2568-55bb-4c15-aa30-2cd8527e43cb" containerName="console"
Sep 09 04:05:11.216361 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.216291    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"]
Sep 09 04:05:11.216489 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.216413    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"
Sep 09 04:05:11.352057 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.352026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4svh\" (UniqueName: \"kubernetes.io/projected/e495ced2-49ba-427f-b0db-30a2127ce723-kube-api-access-f4svh\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rs62s\" (UID: \"e495ced2-49ba-427f-b0db-30a2127ce723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"
Sep 09 04:05:11.352217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.352076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e495ced2-49ba-427f-b0db-30a2127ce723-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rs62s\" (UID: \"e495ced2-49ba-427f-b0db-30a2127ce723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"
Sep 09 04:05:11.452747 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.452704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4svh\" (UniqueName: \"kubernetes.io/projected/e495ced2-49ba-427f-b0db-30a2127ce723-kube-api-access-f4svh\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rs62s\" (UID: \"e495ced2-49ba-427f-b0db-30a2127ce723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"
Sep 09 04:05:11.452919 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.452810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e495ced2-49ba-427f-b0db-30a2127ce723-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rs62s\" (UID: \"e495ced2-49ba-427f-b0db-30a2127ce723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"
Sep 09 04:05:11.453128 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.453113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e495ced2-49ba-427f-b0db-30a2127ce723-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rs62s\" (UID: \"e495ced2-49ba-427f-b0db-30a2127ce723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"
Sep 09 04:05:11.461589 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.461559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4svh\" (UniqueName: \"kubernetes.io/projected/e495ced2-49ba-427f-b0db-30a2127ce723-kube-api-access-f4svh\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rs62s\" (UID: \"e495ced2-49ba-427f-b0db-30a2127ce723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"
Sep 09 04:05:11.527088 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.527010    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"
Sep 09 04:05:11.728127 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.728100    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f460a6c-b1a9-48c6-838a-25e55ede8370" containerID="11e51b40309f2153859b95960dcebddeb83b1cd0bac83c4721a2d77b5d14113a" exitCode=0
Sep 09 04:05:11.728234 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.728130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk" event={"ID":"7f460a6c-b1a9-48c6-838a-25e55ede8370","Type":"ContainerDied","Data":"11e51b40309f2153859b95960dcebddeb83b1cd0bac83c4721a2d77b5d14113a"}
Sep 09 04:05:11.728234 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.728201    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk" event={"ID":"7f460a6c-b1a9-48c6-838a-25e55ede8370","Type":"ContainerDied","Data":"ecc822bc8b8994025d3f3fe9ee9bbde09653a4169ef91d143c75a0fc18ed1d8f"}
Sep 09 04:05:11.728234 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.728229    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecc822bc8b8994025d3f3fe9ee9bbde09653a4169ef91d143c75a0fc18ed1d8f"
Sep 09 04:05:11.754348 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.754320    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"
Sep 09 04:05:11.825572 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.825546    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"]
Sep 09 04:05:11.827114 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:05:11.827094    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode495ced2_49ba_427f_b0db_30a2127ce723.slice/crio-670f68c8c4494aea462db4e5afc9b98d797467f098246c995611b43db1269c7b WatchSource:0}: Error finding container 670f68c8c4494aea462db4e5afc9b98d797467f098246c995611b43db1269c7b: Status 404 returned error can't find the container with id 670f68c8c4494aea462db4e5afc9b98d797467f098246c995611b43db1269c7b
Sep 09 04:05:11.855814 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.855781    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f460a6c-b1a9-48c6-838a-25e55ede8370-extensions-socket-volume\") pod \"7f460a6c-b1a9-48c6-838a-25e55ede8370\" (UID: \"7f460a6c-b1a9-48c6-838a-25e55ede8370\") "
Sep 09 04:05:11.855934 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.855832    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcsqs\" (UniqueName: \"kubernetes.io/projected/7f460a6c-b1a9-48c6-838a-25e55ede8370-kube-api-access-zcsqs\") pod \"7f460a6c-b1a9-48c6-838a-25e55ede8370\" (UID: \"7f460a6c-b1a9-48c6-838a-25e55ede8370\") "
Sep 09 04:05:11.856111 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.856090    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f460a6c-b1a9-48c6-838a-25e55ede8370-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7f460a6c-b1a9-48c6-838a-25e55ede8370" (UID: "7f460a6c-b1a9-48c6-838a-25e55ede8370"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:05:11.857980 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.857958    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f460a6c-b1a9-48c6-838a-25e55ede8370-kube-api-access-zcsqs" (OuterVolumeSpecName: "kube-api-access-zcsqs") pod "7f460a6c-b1a9-48c6-838a-25e55ede8370" (UID: "7f460a6c-b1a9-48c6-838a-25e55ede8370"). InnerVolumeSpecName "kube-api-access-zcsqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:05:11.957159 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.957132    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f460a6c-b1a9-48c6-838a-25e55ede8370-extensions-socket-volume\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:05:11.957159 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:11.957157    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcsqs\" (UniqueName: \"kubernetes.io/projected/7f460a6c-b1a9-48c6-838a-25e55ede8370-kube-api-access-zcsqs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:05:12.735869 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:12.735838    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"
Sep 09 04:05:12.736424 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:12.735835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s" event={"ID":"e495ced2-49ba-427f-b0db-30a2127ce723","Type":"ContainerStarted","Data":"23fa99eaa42f18ba8608c90c2335bea3c2552438efb6565f8e00e3b8da1cbdde"}
Sep 09 04:05:12.736424 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:12.735988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s" event={"ID":"e495ced2-49ba-427f-b0db-30a2127ce723","Type":"ContainerStarted","Data":"670f68c8c4494aea462db4e5afc9b98d797467f098246c995611b43db1269c7b"}
Sep 09 04:05:12.736424 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:12.736129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"
Sep 09 04:05:12.760942 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:12.760896    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s" podStartSLOduration=1.7608824570000001 podStartE2EDuration="1.760882457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:05:12.758434918 +0000 UTC m=+433.708125574" watchObservedRunningTime="2026-09-09 04:05:12.760882457 +0000 UTC m=+433.710573110"
Sep 09 04:05:12.776011 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:12.775983    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"]
Sep 09 04:05:12.779667 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:12.779644    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5vxbk"]
Sep 09 04:05:13.595726 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:13.595691    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f460a6c-b1a9-48c6-838a-25e55ede8370" path="/var/lib/kubelet/pods/7f460a6c-b1a9-48c6-838a-25e55ede8370/volumes"
Sep 09 04:05:24.749028 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:24.748950    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"
Sep 09 04:05:40.947274 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:40.947233    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-s5bcg"]
Sep 09 04:05:40.947682 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:40.947577    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f460a6c-b1a9-48c6-838a-25e55ede8370" containerName="manager"
Sep 09 04:05:40.947682 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:40.947587    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f460a6c-b1a9-48c6-838a-25e55ede8370" containerName="manager"
Sep 09 04:05:40.947682 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:40.947644    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f460a6c-b1a9-48c6-838a-25e55ede8370" containerName="manager"
Sep 09 04:05:40.950777 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:40.950742    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-s5bcg"
Sep 09 04:05:40.953078 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:40.953057    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 04:05:40.963389 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:40.963359    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-s5bcg"]
Sep 09 04:05:41.047458 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.047426    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-s5bcg"]
Sep 09 04:05:41.090235 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.090198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f5hq\" (UniqueName: \"kubernetes.io/projected/b101965e-c762-402e-af4d-83ecca8306eb-kube-api-access-4f5hq\") pod \"limitador-limitador-8588749977-s5bcg\" (UID: \"b101965e-c762-402e-af4d-83ecca8306eb\") " pod="kuadrant-system/limitador-limitador-8588749977-s5bcg"
Sep 09 04:05:41.090398 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.090376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b101965e-c762-402e-af4d-83ecca8306eb-config-file\") pod \"limitador-limitador-8588749977-s5bcg\" (UID: \"b101965e-c762-402e-af4d-83ecca8306eb\") " pod="kuadrant-system/limitador-limitador-8588749977-s5bcg"
Sep 09 04:05:41.191345 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.191308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b101965e-c762-402e-af4d-83ecca8306eb-config-file\") pod \"limitador-limitador-8588749977-s5bcg\" (UID: \"b101965e-c762-402e-af4d-83ecca8306eb\") " pod="kuadrant-system/limitador-limitador-8588749977-s5bcg"
Sep 09 04:05:41.191510 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.191360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f5hq\" (UniqueName: \"kubernetes.io/projected/b101965e-c762-402e-af4d-83ecca8306eb-kube-api-access-4f5hq\") pod \"limitador-limitador-8588749977-s5bcg\" (UID: \"b101965e-c762-402e-af4d-83ecca8306eb\") " pod="kuadrant-system/limitador-limitador-8588749977-s5bcg"
Sep 09 04:05:41.191999 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.191980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b101965e-c762-402e-af4d-83ecca8306eb-config-file\") pod \"limitador-limitador-8588749977-s5bcg\" (UID: \"b101965e-c762-402e-af4d-83ecca8306eb\") " pod="kuadrant-system/limitador-limitador-8588749977-s5bcg"
Sep 09 04:05:41.199806 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.199721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f5hq\" (UniqueName: \"kubernetes.io/projected/b101965e-c762-402e-af4d-83ecca8306eb-kube-api-access-4f5hq\") pod \"limitador-limitador-8588749977-s5bcg\" (UID: \"b101965e-c762-402e-af4d-83ecca8306eb\") " pod="kuadrant-system/limitador-limitador-8588749977-s5bcg"
Sep 09 04:05:41.263812 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.263679    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-s5bcg"
Sep 09 04:05:41.536789 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.536745    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-s5bcg"]
Sep 09 04:05:41.538926 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:05:41.538897    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb101965e_c762_402e_af4d_83ecca8306eb.slice/crio-fe8fd396dc44af6510d29b44426efdcbd5fc564c34857fb93a6e85a8f04127a7 WatchSource:0}: Error finding container fe8fd396dc44af6510d29b44426efdcbd5fc564c34857fb93a6e85a8f04127a7: Status 404 returned error can't find the container with id fe8fd396dc44af6510d29b44426efdcbd5fc564c34857fb93a6e85a8f04127a7
Sep 09 04:05:41.759162 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.759129    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h5rsh"]
Sep 09 04:05:41.763577 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.763558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h5rsh"
Sep 09 04:05:41.766198 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.766005    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-l56ml\""
Sep 09 04:05:41.768998 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.768974    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h5rsh"]
Sep 09 04:05:41.886678 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.886647    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-nxgmw"]
Sep 09 04:05:41.890111 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.890084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nxgmw"
Sep 09 04:05:41.895298 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.895272    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nxgmw"]
Sep 09 04:05:41.895963 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.895936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-s5bcg" event={"ID":"b101965e-c762-402e-af4d-83ecca8306eb","Type":"ContainerStarted","Data":"fe8fd396dc44af6510d29b44426efdcbd5fc564c34857fb93a6e85a8f04127a7"}
Sep 09 04:05:41.897669 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.897648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg52n\" (UniqueName: \"kubernetes.io/projected/d0815835-63be-4112-81ad-8f281568c44b-kube-api-access-kg52n\") pod \"authorino-7bcfd8f9cd-h5rsh\" (UID: \"d0815835-63be-4112-81ad-8f281568c44b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-h5rsh"
Sep 09 04:05:41.998238 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.998202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsvj6\" (UniqueName: \"kubernetes.io/projected/7194923e-8637-44a5-93e7-9a87f3bac2f2-kube-api-access-hsvj6\") pod \"authorino-59c974bd48-nxgmw\" (UID: \"7194923e-8637-44a5-93e7-9a87f3bac2f2\") " pod="kuadrant-system/authorino-59c974bd48-nxgmw"
Sep 09 04:05:41.998600 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:41.998258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg52n\" (UniqueName: \"kubernetes.io/projected/d0815835-63be-4112-81ad-8f281568c44b-kube-api-access-kg52n\") pod \"authorino-7bcfd8f9cd-h5rsh\" (UID: \"d0815835-63be-4112-81ad-8f281568c44b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-h5rsh"
Sep 09 04:05:42.007022 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:42.006993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg52n\" (UniqueName: \"kubernetes.io/projected/d0815835-63be-4112-81ad-8f281568c44b-kube-api-access-kg52n\") pod \"authorino-7bcfd8f9cd-h5rsh\" (UID: \"d0815835-63be-4112-81ad-8f281568c44b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-h5rsh"
Sep 09 04:05:42.075000 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:42.074968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h5rsh"
Sep 09 04:05:42.099144 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:42.099120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsvj6\" (UniqueName: \"kubernetes.io/projected/7194923e-8637-44a5-93e7-9a87f3bac2f2-kube-api-access-hsvj6\") pod \"authorino-59c974bd48-nxgmw\" (UID: \"7194923e-8637-44a5-93e7-9a87f3bac2f2\") " pod="kuadrant-system/authorino-59c974bd48-nxgmw"
Sep 09 04:05:42.108464 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:42.108433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsvj6\" (UniqueName: \"kubernetes.io/projected/7194923e-8637-44a5-93e7-9a87f3bac2f2-kube-api-access-hsvj6\") pod \"authorino-59c974bd48-nxgmw\" (UID: \"7194923e-8637-44a5-93e7-9a87f3bac2f2\") " pod="kuadrant-system/authorino-59c974bd48-nxgmw"
Sep 09 04:05:42.200884 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:42.200805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nxgmw"
Sep 09 04:05:42.344746 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:42.344720    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h5rsh"]
Sep 09 04:05:42.346752 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:05:42.346727    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0815835_63be_4112_81ad_8f281568c44b.slice/crio-aa6d8d4de758f929e791e3da9b79ceb6353c3275aacf06f6cfa28129d0c1cc76 WatchSource:0}: Error finding container aa6d8d4de758f929e791e3da9b79ceb6353c3275aacf06f6cfa28129d0c1cc76: Status 404 returned error can't find the container with id aa6d8d4de758f929e791e3da9b79ceb6353c3275aacf06f6cfa28129d0c1cc76
Sep 09 04:05:42.478581 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:42.478551    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nxgmw"]
Sep 09 04:05:42.480129 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:05:42.480104    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7194923e_8637_44a5_93e7_9a87f3bac2f2.slice/crio-d4d780d363101bbad2d059947e14cff3f6630c9a8fd1f1eda095e6dc73ae3e0d WatchSource:0}: Error finding container d4d780d363101bbad2d059947e14cff3f6630c9a8fd1f1eda095e6dc73ae3e0d: Status 404 returned error can't find the container with id d4d780d363101bbad2d059947e14cff3f6630c9a8fd1f1eda095e6dc73ae3e0d
Sep 09 04:05:42.902664 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:42.902625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h5rsh" event={"ID":"d0815835-63be-4112-81ad-8f281568c44b","Type":"ContainerStarted","Data":"aa6d8d4de758f929e791e3da9b79ceb6353c3275aacf06f6cfa28129d0c1cc76"}
Sep 09 04:05:42.903804 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:42.903778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nxgmw" event={"ID":"7194923e-8637-44a5-93e7-9a87f3bac2f2","Type":"ContainerStarted","Data":"d4d780d363101bbad2d059947e14cff3f6630c9a8fd1f1eda095e6dc73ae3e0d"}
Sep 09 04:05:42.905318 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:42.905297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-s5bcg" event={"ID":"b101965e-c762-402e-af4d-83ecca8306eb","Type":"ContainerStarted","Data":"92baa390785a4258f11a80093eb336ef41dbc7e286f367978bb66531c0cd3adb"}
Sep 09 04:05:42.905414 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:42.905345    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-s5bcg"
Sep 09 04:05:42.923093 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:42.923049    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-s5bcg" podStartSLOduration=1.652349676 podStartE2EDuration="2.923036153s" totalImagesPullingTime="1.270686477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:05:42.921821175 +0000 UTC m=+463.871511828" watchObservedRunningTime="2026-09-09 04:05:42.923036153 +0000 UTC m=+463.872726806"
Sep 09 04:05:44.923548 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:44.923511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h5rsh" event={"ID":"d0815835-63be-4112-81ad-8f281568c44b","Type":"ContainerStarted","Data":"f55cf4222bd4d903bc1a7ce1b5a472902c92f2034828954ff54e31f84563ff11"}
Sep 09 04:05:44.924949 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:44.924928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nxgmw" event={"ID":"7194923e-8637-44a5-93e7-9a87f3bac2f2","Type":"ContainerStarted","Data":"82761392954e8c54ed76c42ccc8db91f8fbd440d59f7dd305032f62346c9e0d9"}
Sep 09 04:05:44.938797 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:44.938733    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-h5rsh" podStartSLOduration=1.9829252130000001 podStartE2EDuration="3.93871808s" totalImagesPullingTime="1.955792867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:05:44.937570011 +0000 UTC m=+465.887260665" watchObservedRunningTime="2026-09-09 04:05:44.93871808 +0000 UTC m=+465.888408736"
Sep 09 04:05:44.951827 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:44.951785    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-nxgmw" podStartSLOduration=2.125202834 podStartE2EDuration="3.951749584s" totalImagesPullingTime="1.82654675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:05:44.95088645 +0000 UTC m=+465.900577103" watchObservedRunningTime="2026-09-09 04:05:44.951749584 +0000 UTC m=+465.901440237"
Sep 09 04:05:44.979463 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:44.979434    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h5rsh"]
Sep 09 04:05:46.936592 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:46.936534    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-h5rsh" podUID="d0815835-63be-4112-81ad-8f281568c44b" containerName="authorino" containerID="cri-o://f55cf4222bd4d903bc1a7ce1b5a472902c92f2034828954ff54e31f84563ff11" gracePeriod=30
Sep 09 04:05:47.185881 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:47.185857    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h5rsh"
Sep 09 04:05:47.343199 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:47.343172    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg52n\" (UniqueName: \"kubernetes.io/projected/d0815835-63be-4112-81ad-8f281568c44b-kube-api-access-kg52n\") pod \"d0815835-63be-4112-81ad-8f281568c44b\" (UID: \"d0815835-63be-4112-81ad-8f281568c44b\") "
Sep 09 04:05:47.345306 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:47.345277    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0815835-63be-4112-81ad-8f281568c44b-kube-api-access-kg52n" (OuterVolumeSpecName: "kube-api-access-kg52n") pod "d0815835-63be-4112-81ad-8f281568c44b" (UID: "d0815835-63be-4112-81ad-8f281568c44b"). InnerVolumeSpecName "kube-api-access-kg52n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:05:47.443743 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:47.443706    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg52n\" (UniqueName: \"kubernetes.io/projected/d0815835-63be-4112-81ad-8f281568c44b-kube-api-access-kg52n\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:05:47.942905 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:47.942812    2580 generic.go:356] "Generic (PLEG): container finished" podID="d0815835-63be-4112-81ad-8f281568c44b" containerID="f55cf4222bd4d903bc1a7ce1b5a472902c92f2034828954ff54e31f84563ff11" exitCode=0
Sep 09 04:05:47.942905 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:47.942857    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h5rsh"
Sep 09 04:05:47.942905 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:47.942893    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h5rsh" event={"ID":"d0815835-63be-4112-81ad-8f281568c44b","Type":"ContainerDied","Data":"f55cf4222bd4d903bc1a7ce1b5a472902c92f2034828954ff54e31f84563ff11"}
Sep 09 04:05:47.943412 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:47.942939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h5rsh" event={"ID":"d0815835-63be-4112-81ad-8f281568c44b","Type":"ContainerDied","Data":"aa6d8d4de758f929e791e3da9b79ceb6353c3275aacf06f6cfa28129d0c1cc76"}
Sep 09 04:05:47.943412 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:47.942963    2580 scope.go:117] "RemoveContainer" containerID="f55cf4222bd4d903bc1a7ce1b5a472902c92f2034828954ff54e31f84563ff11"
Sep 09 04:05:47.951830 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:47.951811    2580 scope.go:117] "RemoveContainer" containerID="f55cf4222bd4d903bc1a7ce1b5a472902c92f2034828954ff54e31f84563ff11"
Sep 09 04:05:47.952145 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:05:47.952125    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55cf4222bd4d903bc1a7ce1b5a472902c92f2034828954ff54e31f84563ff11\": container with ID starting with f55cf4222bd4d903bc1a7ce1b5a472902c92f2034828954ff54e31f84563ff11 not found: ID does not exist" containerID="f55cf4222bd4d903bc1a7ce1b5a472902c92f2034828954ff54e31f84563ff11"
Sep 09 04:05:47.952210 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:47.952154    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55cf4222bd4d903bc1a7ce1b5a472902c92f2034828954ff54e31f84563ff11"} err="failed to get container status \"f55cf4222bd4d903bc1a7ce1b5a472902c92f2034828954ff54e31f84563ff11\": rpc error: code = NotFound desc = could not find container \"f55cf4222bd4d903bc1a7ce1b5a472902c92f2034828954ff54e31f84563ff11\": container with ID starting with f55cf4222bd4d903bc1a7ce1b5a472902c92f2034828954ff54e31f84563ff11 not found: ID does not exist"
Sep 09 04:05:47.959801 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:47.959757    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h5rsh"]
Sep 09 04:05:47.965377 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:47.965357    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h5rsh"]
Sep 09 04:05:49.595285 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:49.595253    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0815835-63be-4112-81ad-8f281568c44b" path="/var/lib/kubelet/pods/d0815835-63be-4112-81ad-8f281568c44b/volumes"
Sep 09 04:05:53.913288 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:53.913260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-s5bcg"
Sep 09 04:05:55.166090 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:55.166059    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-s5bcg"]
Sep 09 04:05:55.166527 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:55.166239    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-s5bcg" podUID="b101965e-c762-402e-af4d-83ecca8306eb" containerName="limitador" containerID="cri-o://92baa390785a4258f11a80093eb336ef41dbc7e286f367978bb66531c0cd3adb" gracePeriod=30
Sep 09 04:05:55.987607 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:55.987580    2580 generic.go:356] "Generic (PLEG): container finished" podID="b101965e-c762-402e-af4d-83ecca8306eb" containerID="92baa390785a4258f11a80093eb336ef41dbc7e286f367978bb66531c0cd3adb" exitCode=0
Sep 09 04:05:55.987782 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:55.987657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-s5bcg" event={"ID":"b101965e-c762-402e-af4d-83ecca8306eb","Type":"ContainerDied","Data":"92baa390785a4258f11a80093eb336ef41dbc7e286f367978bb66531c0cd3adb"}
Sep 09 04:05:56.118711 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:56.118690    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-s5bcg"
Sep 09 04:05:56.220448 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:56.220416    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f5hq\" (UniqueName: \"kubernetes.io/projected/b101965e-c762-402e-af4d-83ecca8306eb-kube-api-access-4f5hq\") pod \"b101965e-c762-402e-af4d-83ecca8306eb\" (UID: \"b101965e-c762-402e-af4d-83ecca8306eb\") "
Sep 09 04:05:56.220448 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:56.220452    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b101965e-c762-402e-af4d-83ecca8306eb-config-file\") pod \"b101965e-c762-402e-af4d-83ecca8306eb\" (UID: \"b101965e-c762-402e-af4d-83ecca8306eb\") "
Sep 09 04:05:56.220890 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:56.220869    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b101965e-c762-402e-af4d-83ecca8306eb-config-file" (OuterVolumeSpecName: "config-file") pod "b101965e-c762-402e-af4d-83ecca8306eb" (UID: "b101965e-c762-402e-af4d-83ecca8306eb"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:05:56.222652 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:56.222632    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b101965e-c762-402e-af4d-83ecca8306eb-kube-api-access-4f5hq" (OuterVolumeSpecName: "kube-api-access-4f5hq") pod "b101965e-c762-402e-af4d-83ecca8306eb" (UID: "b101965e-c762-402e-af4d-83ecca8306eb"). InnerVolumeSpecName "kube-api-access-4f5hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:05:56.321875 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:56.321796    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f5hq\" (UniqueName: \"kubernetes.io/projected/b101965e-c762-402e-af4d-83ecca8306eb-kube-api-access-4f5hq\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:05:56.321875 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:56.321826    2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b101965e-c762-402e-af4d-83ecca8306eb-config-file\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:05:56.995344 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:56.995314    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-s5bcg"
Sep 09 04:05:56.995519 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:56.995357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-s5bcg" event={"ID":"b101965e-c762-402e-af4d-83ecca8306eb","Type":"ContainerDied","Data":"fe8fd396dc44af6510d29b44426efdcbd5fc564c34857fb93a6e85a8f04127a7"}
Sep 09 04:05:56.995519 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:56.995424    2580 scope.go:117] "RemoveContainer" containerID="92baa390785a4258f11a80093eb336ef41dbc7e286f367978bb66531c0cd3adb"
Sep 09 04:05:57.017400 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:57.017378    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-s5bcg"]
Sep 09 04:05:57.021418 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:57.021395    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-s5bcg"]
Sep 09 04:05:57.595482 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:05:57.595453    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b101965e-c762-402e-af4d-83ecca8306eb" path="/var/lib/kubelet/pods/b101965e-c762-402e-af4d-83ecca8306eb/volumes"
Sep 09 04:06:00.999015 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:00.998984    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl"]
Sep 09 04:06:00.999375 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:00.999358    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b101965e-c762-402e-af4d-83ecca8306eb" containerName="limitador"
Sep 09 04:06:00.999375 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:00.999371    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b101965e-c762-402e-af4d-83ecca8306eb" containerName="limitador"
Sep 09 04:06:00.999450 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:00.999387    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0815835-63be-4112-81ad-8f281568c44b" containerName="authorino"
Sep 09 04:06:00.999450 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:00.999397    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0815835-63be-4112-81ad-8f281568c44b" containerName="authorino"
Sep 09 04:06:00.999513 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:00.999471    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b101965e-c762-402e-af4d-83ecca8306eb" containerName="limitador"
Sep 09 04:06:00.999513 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:00.999482    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0815835-63be-4112-81ad-8f281568c44b" containerName="authorino"
Sep 09 04:06:01.003830 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.003812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl"
Sep 09 04:06:01.006432 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.006404    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 04:06:01.007599 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.007577    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 04:06:01.007705 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.007597    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 09 04:06:01.007705 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.007597    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-q76p9\""
Sep 09 04:06:01.012988 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.012969    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl"]
Sep 09 04:06:01.054354 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.054324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzcsq\" (UniqueName: \"kubernetes.io/projected/6395fdb6-dda5-432d-834b-a581d4ef7596-kube-api-access-fzcsq\") pod \"postgres-69fb4594cf-ztlvl\" (UID: \"6395fdb6-dda5-432d-834b-a581d4ef7596\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl"
Sep 09 04:06:01.054499 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.054360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6395fdb6-dda5-432d-834b-a581d4ef7596-data\") pod \"postgres-69fb4594cf-ztlvl\" (UID: \"6395fdb6-dda5-432d-834b-a581d4ef7596\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl"
Sep 09 04:06:01.155615 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.155583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcsq\" (UniqueName: \"kubernetes.io/projected/6395fdb6-dda5-432d-834b-a581d4ef7596-kube-api-access-fzcsq\") pod \"postgres-69fb4594cf-ztlvl\" (UID: \"6395fdb6-dda5-432d-834b-a581d4ef7596\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl"
Sep 09 04:06:01.155806 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.155626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6395fdb6-dda5-432d-834b-a581d4ef7596-data\") pod \"postgres-69fb4594cf-ztlvl\" (UID: \"6395fdb6-dda5-432d-834b-a581d4ef7596\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl"
Sep 09 04:06:01.156021 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.156003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6395fdb6-dda5-432d-834b-a581d4ef7596-data\") pod \"postgres-69fb4594cf-ztlvl\" (UID: \"6395fdb6-dda5-432d-834b-a581d4ef7596\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl"
Sep 09 04:06:01.163786 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.163748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcsq\" (UniqueName: \"kubernetes.io/projected/6395fdb6-dda5-432d-834b-a581d4ef7596-kube-api-access-fzcsq\") pod \"postgres-69fb4594cf-ztlvl\" (UID: \"6395fdb6-dda5-432d-834b-a581d4ef7596\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl"
Sep 09 04:06:01.316013 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.315926    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl"
Sep 09 04:06:01.587333 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:01.587302    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl"]
Sep 09 04:06:01.588899 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:06:01.588873    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6395fdb6_dda5_432d_834b_a581d4ef7596.slice/crio-a94cfed388aeadbaa76cf18eaf473ea2336b39e21d5221b8424c884c84c6ef19 WatchSource:0}: Error finding container a94cfed388aeadbaa76cf18eaf473ea2336b39e21d5221b8424c884c84c6ef19: Status 404 returned error can't find the container with id a94cfed388aeadbaa76cf18eaf473ea2336b39e21d5221b8424c884c84c6ef19
Sep 09 04:06:02.025504 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:02.025470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl" event={"ID":"6395fdb6-dda5-432d-834b-a581d4ef7596","Type":"ContainerStarted","Data":"a94cfed388aeadbaa76cf18eaf473ea2336b39e21d5221b8424c884c84c6ef19"}
Sep 09 04:06:05.046822 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:05.046786    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl" event={"ID":"6395fdb6-dda5-432d-834b-a581d4ef7596","Type":"ContainerStarted","Data":"34c251d6d8f70adcfee336fabaa803c6a5f05aecefe4651737175ea4071cfaa2"}
Sep 09 04:06:05.047245 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:05.046859    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl"
Sep 09 04:06:05.062870 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:05.062826    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl" podStartSLOduration=2.534515526 podStartE2EDuration="5.062810982s" totalImagesPullingTime="2.528295456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:06:05.061626079 +0000 UTC m=+486.011316744" watchObservedRunningTime="2026-09-09 04:06:05.062810982 +0000 UTC m=+486.012501636"
Sep 09 04:06:11.082329 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.082298    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ztlvl"
Sep 09 04:06:11.596176 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.596147    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-2mbxd"]
Sep 09 04:06:11.602685 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.602670    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2mbxd"
Sep 09 04:06:11.608042 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.608016    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-2mbxd"]
Sep 09 04:06:11.636115 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.636086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zdj9\" (UniqueName: \"kubernetes.io/projected/7a378b52-7cb2-45e8-907a-6fdbadc3abb2-kube-api-access-5zdj9\") pod \"authorino-557f876f69-2mbxd\" (UID: \"7a378b52-7cb2-45e8-907a-6fdbadc3abb2\") " pod="kuadrant-system/authorino-557f876f69-2mbxd"
Sep 09 04:06:11.653874 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.653843    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-2mbxd"]
Sep 09 04:06:11.654074 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:06:11.654058    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5zdj9], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-2mbxd" podUID="7a378b52-7cb2-45e8-907a-6fdbadc3abb2"
Sep 09 04:06:11.682750 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.682723    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mxdvf"]
Sep 09 04:06:11.687255 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.687240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mxdvf"
Sep 09 04:06:11.690042 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.690022    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 04:06:11.694501 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.694416    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mxdvf"]
Sep 09 04:06:11.736585 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.736553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zdj9\" (UniqueName: \"kubernetes.io/projected/7a378b52-7cb2-45e8-907a-6fdbadc3abb2-kube-api-access-5zdj9\") pod \"authorino-557f876f69-2mbxd\" (UID: \"7a378b52-7cb2-45e8-907a-6fdbadc3abb2\") " pod="kuadrant-system/authorino-557f876f69-2mbxd"
Sep 09 04:06:11.736719 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.736590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjns\" (UniqueName: \"kubernetes.io/projected/39c31fe5-9aa4-4422-aec9-ef711243bcab-kube-api-access-9gjns\") pod \"authorino-6d4dfc649f-mxdvf\" (UID: \"39c31fe5-9aa4-4422-aec9-ef711243bcab\") " pod="kuadrant-system/authorino-6d4dfc649f-mxdvf"
Sep 09 04:06:11.736719 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.736632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39c31fe5-9aa4-4422-aec9-ef711243bcab-tls-cert\") pod \"authorino-6d4dfc649f-mxdvf\" (UID: \"39c31fe5-9aa4-4422-aec9-ef711243bcab\") " pod="kuadrant-system/authorino-6d4dfc649f-mxdvf"
Sep 09 04:06:11.749242 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.749216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zdj9\" (UniqueName: \"kubernetes.io/projected/7a378b52-7cb2-45e8-907a-6fdbadc3abb2-kube-api-access-5zdj9\") pod \"authorino-557f876f69-2mbxd\" (UID: \"7a378b52-7cb2-45e8-907a-6fdbadc3abb2\") " pod="kuadrant-system/authorino-557f876f69-2mbxd"
Sep 09 04:06:11.837854 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.837827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39c31fe5-9aa4-4422-aec9-ef711243bcab-tls-cert\") pod \"authorino-6d4dfc649f-mxdvf\" (UID: \"39c31fe5-9aa4-4422-aec9-ef711243bcab\") " pod="kuadrant-system/authorino-6d4dfc649f-mxdvf"
Sep 09 04:06:11.837989 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.837922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjns\" (UniqueName: \"kubernetes.io/projected/39c31fe5-9aa4-4422-aec9-ef711243bcab-kube-api-access-9gjns\") pod \"authorino-6d4dfc649f-mxdvf\" (UID: \"39c31fe5-9aa4-4422-aec9-ef711243bcab\") " pod="kuadrant-system/authorino-6d4dfc649f-mxdvf"
Sep 09 04:06:11.840391 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.840360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39c31fe5-9aa4-4422-aec9-ef711243bcab-tls-cert\") pod \"authorino-6d4dfc649f-mxdvf\" (UID: \"39c31fe5-9aa4-4422-aec9-ef711243bcab\") " pod="kuadrant-system/authorino-6d4dfc649f-mxdvf"
Sep 09 04:06:11.851203 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.851152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjns\" (UniqueName: \"kubernetes.io/projected/39c31fe5-9aa4-4422-aec9-ef711243bcab-kube-api-access-9gjns\") pod \"authorino-6d4dfc649f-mxdvf\" (UID: \"39c31fe5-9aa4-4422-aec9-ef711243bcab\") " pod="kuadrant-system/authorino-6d4dfc649f-mxdvf"
Sep 09 04:06:11.856759 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.856737    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mxdvf"]
Sep 09 04:06:11.856998 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.856984    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mxdvf"
Sep 09 04:06:11.883679 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.883653    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6cb59c5b-vk6t6"]
Sep 09 04:06:11.889591 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.889563    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6"
Sep 09 04:06:11.894777 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.894740    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6cb59c5b-vk6t6"]
Sep 09 04:06:11.939412 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.939373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f094812-8368-4712-9c51-2cbcebd2917c-tls-cert\") pod \"authorino-6f6cb59c5b-vk6t6\" (UID: \"8f094812-8368-4712-9c51-2cbcebd2917c\") " pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6"
Sep 09 04:06:11.939601 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:11.939427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4db8\" (UniqueName: \"kubernetes.io/projected/8f094812-8368-4712-9c51-2cbcebd2917c-kube-api-access-r4db8\") pod \"authorino-6f6cb59c5b-vk6t6\" (UID: \"8f094812-8368-4712-9c51-2cbcebd2917c\") " pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6"
Sep 09 04:06:12.040813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:12.040752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f094812-8368-4712-9c51-2cbcebd2917c-tls-cert\") pod \"authorino-6f6cb59c5b-vk6t6\" (UID: \"8f094812-8368-4712-9c51-2cbcebd2917c\") " pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6"
Sep 09 04:06:12.040996 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:12.040824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4db8\" (UniqueName: \"kubernetes.io/projected/8f094812-8368-4712-9c51-2cbcebd2917c-kube-api-access-r4db8\") pod \"authorino-6f6cb59c5b-vk6t6\" (UID: \"8f094812-8368-4712-9c51-2cbcebd2917c\") " pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6"
Sep 09 04:06:12.043340 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:12.043319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f094812-8368-4712-9c51-2cbcebd2917c-tls-cert\") pod \"authorino-6f6cb59c5b-vk6t6\" (UID: \"8f094812-8368-4712-9c51-2cbcebd2917c\") " pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6"
Sep 09 04:06:12.049279 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:12.049257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4db8\" (UniqueName: \"kubernetes.io/projected/8f094812-8368-4712-9c51-2cbcebd2917c-kube-api-access-r4db8\") pod \"authorino-6f6cb59c5b-vk6t6\" (UID: \"8f094812-8368-4712-9c51-2cbcebd2917c\") " pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6"
Sep 09 04:06:12.083866 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:12.083845    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2mbxd"
Sep 09 04:06:12.091046 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:12.091022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2mbxd"
Sep 09 04:06:12.133418 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:12.133397    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mxdvf"]
Sep 09 04:06:12.135248 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:06:12.135227    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39c31fe5_9aa4_4422_aec9_ef711243bcab.slice/crio-a1b6aec31551453bcc763dcdd3d21ba0303799dc47b0d2319ba76a12b48bc4be WatchSource:0}: Error finding container a1b6aec31551453bcc763dcdd3d21ba0303799dc47b0d2319ba76a12b48bc4be: Status 404 returned error can't find the container with id a1b6aec31551453bcc763dcdd3d21ba0303799dc47b0d2319ba76a12b48bc4be
Sep 09 04:06:12.141195 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:12.141175    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zdj9\" (UniqueName: \"kubernetes.io/projected/7a378b52-7cb2-45e8-907a-6fdbadc3abb2-kube-api-access-5zdj9\") pod \"7a378b52-7cb2-45e8-907a-6fdbadc3abb2\" (UID: \"7a378b52-7cb2-45e8-907a-6fdbadc3abb2\") "
Sep 09 04:06:12.143340 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:12.143319    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a378b52-7cb2-45e8-907a-6fdbadc3abb2-kube-api-access-5zdj9" (OuterVolumeSpecName: "kube-api-access-5zdj9") pod "7a378b52-7cb2-45e8-907a-6fdbadc3abb2" (UID: "7a378b52-7cb2-45e8-907a-6fdbadc3abb2"). InnerVolumeSpecName "kube-api-access-5zdj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:06:12.202159 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:12.202132    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6"
Sep 09 04:06:12.242593 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:12.242567    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zdj9\" (UniqueName: \"kubernetes.io/projected/7a378b52-7cb2-45e8-907a-6fdbadc3abb2-kube-api-access-5zdj9\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:06:12.474105 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:12.474077    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6cb59c5b-vk6t6"]
Sep 09 04:06:12.478272 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:06:12.475435    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f094812_8368_4712_9c51_2cbcebd2917c.slice/crio-b2a8f8ec9c840339b6f9e8f95eb9cc533a1122e98db65386a04567c7bc498f8f WatchSource:0}: Error finding container b2a8f8ec9c840339b6f9e8f95eb9cc533a1122e98db65386a04567c7bc498f8f: Status 404 returned error can't find the container with id b2a8f8ec9c840339b6f9e8f95eb9cc533a1122e98db65386a04567c7bc498f8f
Sep 09 04:06:13.090568 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.090527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-mxdvf" event={"ID":"39c31fe5-9aa4-4422-aec9-ef711243bcab","Type":"ContainerStarted","Data":"8202fc4be740ff4cc85ef0be22447b30d0db9b03d4f1b804c1a25d076e06e8d0"}
Sep 09 04:06:13.090568 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.090555    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d4dfc649f-mxdvf" podUID="39c31fe5-9aa4-4422-aec9-ef711243bcab" containerName="authorino" containerID="cri-o://8202fc4be740ff4cc85ef0be22447b30d0db9b03d4f1b804c1a25d076e06e8d0" gracePeriod=30
Sep 09 04:06:13.091215 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.090579    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-mxdvf" event={"ID":"39c31fe5-9aa4-4422-aec9-ef711243bcab","Type":"ContainerStarted","Data":"a1b6aec31551453bcc763dcdd3d21ba0303799dc47b0d2319ba76a12b48bc4be"}
Sep 09 04:06:13.091928 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.091901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2mbxd"
Sep 09 04:06:13.091928 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.091920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6" event={"ID":"8f094812-8368-4712-9c51-2cbcebd2917c","Type":"ContainerStarted","Data":"b2a8f8ec9c840339b6f9e8f95eb9cc533a1122e98db65386a04567c7bc498f8f"}
Sep 09 04:06:13.107013 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.106975    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d4dfc649f-mxdvf" podStartSLOduration=1.380911658 podStartE2EDuration="2.106961487s" totalImagesPullingTime="726.049829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:06:13.105179928 +0000 UTC m=+494.054870585" watchObservedRunningTime="2026-09-09 04:06:13.106961487 +0000 UTC m=+494.056652140"
Sep 09 04:06:13.126807 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.126782    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-2mbxd"]
Sep 09 04:06:13.128941 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.128917    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-2mbxd"]
Sep 09 04:06:13.329689 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.329668    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mxdvf"
Sep 09 04:06:13.454257 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.454233    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gjns\" (UniqueName: \"kubernetes.io/projected/39c31fe5-9aa4-4422-aec9-ef711243bcab-kube-api-access-9gjns\") pod \"39c31fe5-9aa4-4422-aec9-ef711243bcab\" (UID: \"39c31fe5-9aa4-4422-aec9-ef711243bcab\") "
Sep 09 04:06:13.454372 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.454269    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39c31fe5-9aa4-4422-aec9-ef711243bcab-tls-cert\") pod \"39c31fe5-9aa4-4422-aec9-ef711243bcab\" (UID: \"39c31fe5-9aa4-4422-aec9-ef711243bcab\") "
Sep 09 04:06:13.456383 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.456357    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c31fe5-9aa4-4422-aec9-ef711243bcab-kube-api-access-9gjns" (OuterVolumeSpecName: "kube-api-access-9gjns") pod "39c31fe5-9aa4-4422-aec9-ef711243bcab" (UID: "39c31fe5-9aa4-4422-aec9-ef711243bcab"). InnerVolumeSpecName "kube-api-access-9gjns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:06:13.464568 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.464540    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c31fe5-9aa4-4422-aec9-ef711243bcab-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "39c31fe5-9aa4-4422-aec9-ef711243bcab" (UID: "39c31fe5-9aa4-4422-aec9-ef711243bcab"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:06:13.555118 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.555082    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gjns\" (UniqueName: \"kubernetes.io/projected/39c31fe5-9aa4-4422-aec9-ef711243bcab-kube-api-access-9gjns\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:06:13.555118 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.555111    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39c31fe5-9aa4-4422-aec9-ef711243bcab-tls-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:06:13.600304 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:13.600277    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a378b52-7cb2-45e8-907a-6fdbadc3abb2" path="/var/lib/kubelet/pods/7a378b52-7cb2-45e8-907a-6fdbadc3abb2/volumes"
Sep 09 04:06:14.098119 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.098085    2580 generic.go:356] "Generic (PLEG): container finished" podID="39c31fe5-9aa4-4422-aec9-ef711243bcab" containerID="8202fc4be740ff4cc85ef0be22447b30d0db9b03d4f1b804c1a25d076e06e8d0" exitCode=0
Sep 09 04:06:14.098552 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.098144    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mxdvf"
Sep 09 04:06:14.098552 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.098169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-mxdvf" event={"ID":"39c31fe5-9aa4-4422-aec9-ef711243bcab","Type":"ContainerDied","Data":"8202fc4be740ff4cc85ef0be22447b30d0db9b03d4f1b804c1a25d076e06e8d0"}
Sep 09 04:06:14.098552 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.098214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-mxdvf" event={"ID":"39c31fe5-9aa4-4422-aec9-ef711243bcab","Type":"ContainerDied","Data":"a1b6aec31551453bcc763dcdd3d21ba0303799dc47b0d2319ba76a12b48bc4be"}
Sep 09 04:06:14.098552 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.098233    2580 scope.go:117] "RemoveContainer" containerID="8202fc4be740ff4cc85ef0be22447b30d0db9b03d4f1b804c1a25d076e06e8d0"
Sep 09 04:06:14.099945 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.099920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6" event={"ID":"8f094812-8368-4712-9c51-2cbcebd2917c","Type":"ContainerStarted","Data":"62914ffa8e4421a1ed48d0e2be4b5222265581d5b17f9b76344539103b023e07"}
Sep 09 04:06:14.107099 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.107084    2580 scope.go:117] "RemoveContainer" containerID="8202fc4be740ff4cc85ef0be22447b30d0db9b03d4f1b804c1a25d076e06e8d0"
Sep 09 04:06:14.107348 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:06:14.107330    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8202fc4be740ff4cc85ef0be22447b30d0db9b03d4f1b804c1a25d076e06e8d0\": container with ID starting with 8202fc4be740ff4cc85ef0be22447b30d0db9b03d4f1b804c1a25d076e06e8d0 not found: ID does not exist" containerID="8202fc4be740ff4cc85ef0be22447b30d0db9b03d4f1b804c1a25d076e06e8d0"
Sep 09 04:06:14.107402 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.107357    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8202fc4be740ff4cc85ef0be22447b30d0db9b03d4f1b804c1a25d076e06e8d0"} err="failed to get container status \"8202fc4be740ff4cc85ef0be22447b30d0db9b03d4f1b804c1a25d076e06e8d0\": rpc error: code = NotFound desc = could not find container \"8202fc4be740ff4cc85ef0be22447b30d0db9b03d4f1b804c1a25d076e06e8d0\": container with ID starting with 8202fc4be740ff4cc85ef0be22447b30d0db9b03d4f1b804c1a25d076e06e8d0 not found: ID does not exist"
Sep 09 04:06:14.116705 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.116681    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mxdvf"]
Sep 09 04:06:14.120541 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.120522    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mxdvf"]
Sep 09 04:06:14.132554 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.132514    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6" podStartSLOduration=2.243222213 podStartE2EDuration="3.132501494s" totalImagesPullingTime="889.279281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:06:14.132280523 +0000 UTC m=+495.081971180" watchObservedRunningTime="2026-09-09 04:06:14.132501494 +0000 UTC m=+495.082192148"
Sep 09 04:06:14.160036 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.160001    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nxgmw"]
Sep 09 04:06:14.160224 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.160195    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-nxgmw" podUID="7194923e-8637-44a5-93e7-9a87f3bac2f2" containerName="authorino" containerID="cri-o://82761392954e8c54ed76c42ccc8db91f8fbd440d59f7dd305032f62346c9e0d9" gracePeriod=30
Sep 09 04:06:14.407463 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.407436    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nxgmw"
Sep 09 04:06:14.461887 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.461857    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsvj6\" (UniqueName: \"kubernetes.io/projected/7194923e-8637-44a5-93e7-9a87f3bac2f2-kube-api-access-hsvj6\") pod \"7194923e-8637-44a5-93e7-9a87f3bac2f2\" (UID: \"7194923e-8637-44a5-93e7-9a87f3bac2f2\") "
Sep 09 04:06:14.463988 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.463952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7194923e-8637-44a5-93e7-9a87f3bac2f2-kube-api-access-hsvj6" (OuterVolumeSpecName: "kube-api-access-hsvj6") pod "7194923e-8637-44a5-93e7-9a87f3bac2f2" (UID: "7194923e-8637-44a5-93e7-9a87f3bac2f2"). InnerVolumeSpecName "kube-api-access-hsvj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:06:14.562779 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:14.562746    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsvj6\" (UniqueName: \"kubernetes.io/projected/7194923e-8637-44a5-93e7-9a87f3bac2f2-kube-api-access-hsvj6\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:06:15.105844 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:15.105813    2580 generic.go:356] "Generic (PLEG): container finished" podID="7194923e-8637-44a5-93e7-9a87f3bac2f2" containerID="82761392954e8c54ed76c42ccc8db91f8fbd440d59f7dd305032f62346c9e0d9" exitCode=0
Sep 09 04:06:15.106362 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:15.105854    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nxgmw"
Sep 09 04:06:15.106362 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:15.105910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nxgmw" event={"ID":"7194923e-8637-44a5-93e7-9a87f3bac2f2","Type":"ContainerDied","Data":"82761392954e8c54ed76c42ccc8db91f8fbd440d59f7dd305032f62346c9e0d9"}
Sep 09 04:06:15.106362 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:15.105959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nxgmw" event={"ID":"7194923e-8637-44a5-93e7-9a87f3bac2f2","Type":"ContainerDied","Data":"d4d780d363101bbad2d059947e14cff3f6630c9a8fd1f1eda095e6dc73ae3e0d"}
Sep 09 04:06:15.106362 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:15.105985    2580 scope.go:117] "RemoveContainer" containerID="82761392954e8c54ed76c42ccc8db91f8fbd440d59f7dd305032f62346c9e0d9"
Sep 09 04:06:15.115525 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:15.115509    2580 scope.go:117] "RemoveContainer" containerID="82761392954e8c54ed76c42ccc8db91f8fbd440d59f7dd305032f62346c9e0d9"
Sep 09 04:06:15.115815 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:06:15.115794    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82761392954e8c54ed76c42ccc8db91f8fbd440d59f7dd305032f62346c9e0d9\": container with ID starting with 82761392954e8c54ed76c42ccc8db91f8fbd440d59f7dd305032f62346c9e0d9 not found: ID does not exist" containerID="82761392954e8c54ed76c42ccc8db91f8fbd440d59f7dd305032f62346c9e0d9"
Sep 09 04:06:15.115875 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:15.115825    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82761392954e8c54ed76c42ccc8db91f8fbd440d59f7dd305032f62346c9e0d9"} err="failed to get container status \"82761392954e8c54ed76c42ccc8db91f8fbd440d59f7dd305032f62346c9e0d9\": rpc error: code = NotFound desc = could not find container \"82761392954e8c54ed76c42ccc8db91f8fbd440d59f7dd305032f62346c9e0d9\": container with ID starting with 82761392954e8c54ed76c42ccc8db91f8fbd440d59f7dd305032f62346c9e0d9 not found: ID does not exist"
Sep 09 04:06:15.127958 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:15.127939    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nxgmw"]
Sep 09 04:06:15.129921 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:15.129904    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nxgmw"]
Sep 09 04:06:15.595382 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:15.595352    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39c31fe5-9aa4-4422-aec9-ef711243bcab" path="/var/lib/kubelet/pods/39c31fe5-9aa4-4422-aec9-ef711243bcab/volumes"
Sep 09 04:06:15.595672 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:15.595659    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7194923e-8637-44a5-93e7-9a87f3bac2f2" path="/var/lib/kubelet/pods/7194923e-8637-44a5-93e7-9a87f3bac2f2/volumes"
Sep 09 04:06:18.878476 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.878441    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f58485c69-grxjz"]
Sep 09 04:06:18.878950 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.878824    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c31fe5-9aa4-4422-aec9-ef711243bcab" containerName="authorino"
Sep 09 04:06:18.878950 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.878836    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c31fe5-9aa4-4422-aec9-ef711243bcab" containerName="authorino"
Sep 09 04:06:18.878950 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.878848    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7194923e-8637-44a5-93e7-9a87f3bac2f2" containerName="authorino"
Sep 09 04:06:18.878950 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.878853    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7194923e-8637-44a5-93e7-9a87f3bac2f2" containerName="authorino"
Sep 09 04:06:18.878950 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.878907    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7194923e-8637-44a5-93e7-9a87f3bac2f2" containerName="authorino"
Sep 09 04:06:18.878950 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.878916    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c31fe5-9aa4-4422-aec9-ef711243bcab" containerName="authorino"
Sep 09 04:06:18.887124 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.887087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:18.889753 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.889731    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kdm6s\""
Sep 09 04:06:18.890270 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.890094    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 04:06:18.890270 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.890195    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f58485c69-grxjz"]
Sep 09 04:06:18.890270 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.890258    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 04:06:18.890855 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.890834    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 04:06:18.996871 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.996834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-metrics-certs\") pod \"maas-controller-f58485c69-grxjz\" (UID: \"68ee8855-3c6e-410e-a9d9-826657c2838b\") " pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:18.997071 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.996884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q96ss\" (UniqueName: \"kubernetes.io/projected/68ee8855-3c6e-410e-a9d9-826657c2838b-kube-api-access-q96ss\") pod \"maas-controller-f58485c69-grxjz\" (UID: \"68ee8855-3c6e-410e-a9d9-826657c2838b\") " pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:18.997071 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:18.996911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-webhook-cert\") pod \"maas-controller-f58485c69-grxjz\" (UID: \"68ee8855-3c6e-410e-a9d9-826657c2838b\") " pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:19.045892 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.045856    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-z82dz"]
Sep 09 04:06:19.049314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.049300    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:06:19.058538 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.058512    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-z82dz"]
Sep 09 04:06:19.098223 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.098189    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fznh\" (UniqueName: \"kubernetes.io/projected/669da265-2e72-437a-b0c5-19aa2d470b69-kube-api-access-6fznh\") pod \"maas-controller-7d8bb9d654-z82dz\" (UID: \"669da265-2e72-437a-b0c5-19aa2d470b69\") " pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:06:19.098364 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.098229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-metrics-certs\") pod \"maas-controller-f58485c69-grxjz\" (UID: \"68ee8855-3c6e-410e-a9d9-826657c2838b\") " pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:19.098364 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.098277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q96ss\" (UniqueName: \"kubernetes.io/projected/68ee8855-3c6e-410e-a9d9-826657c2838b-kube-api-access-q96ss\") pod \"maas-controller-f58485c69-grxjz\" (UID: \"68ee8855-3c6e-410e-a9d9-826657c2838b\") " pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:19.098364 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.098296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/669da265-2e72-437a-b0c5-19aa2d470b69-webhook-cert\") pod \"maas-controller-7d8bb9d654-z82dz\" (UID: \"669da265-2e72-437a-b0c5-19aa2d470b69\") " pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:06:19.098364 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.098310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669da265-2e72-437a-b0c5-19aa2d470b69-metrics-certs\") pod \"maas-controller-7d8bb9d654-z82dz\" (UID: \"669da265-2e72-437a-b0c5-19aa2d470b69\") " pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:06:19.098364 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.098334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-webhook-cert\") pod \"maas-controller-f58485c69-grxjz\" (UID: \"68ee8855-3c6e-410e-a9d9-826657c2838b\") " pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:19.098572 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:06:19.098459    2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 09 04:06:19.098572 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:06:19.098561    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-metrics-certs podName:68ee8855-3c6e-410e-a9d9-826657c2838b nodeName:}" failed. No retries permitted until 2026-09-09 04:06:19.598542267 +0000 UTC m=+500.548232912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-metrics-certs") pod "maas-controller-f58485c69-grxjz" (UID: "68ee8855-3c6e-410e-a9d9-826657c2838b") : secret "maas-controller-metrics-tls" not found
Sep 09 04:06:19.100917 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.100890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-webhook-cert\") pod \"maas-controller-f58485c69-grxjz\" (UID: \"68ee8855-3c6e-410e-a9d9-826657c2838b\") " pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:19.107385 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.107364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q96ss\" (UniqueName: \"kubernetes.io/projected/68ee8855-3c6e-410e-a9d9-826657c2838b-kube-api-access-q96ss\") pod \"maas-controller-f58485c69-grxjz\" (UID: \"68ee8855-3c6e-410e-a9d9-826657c2838b\") " pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:19.199600 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.199510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/669da265-2e72-437a-b0c5-19aa2d470b69-webhook-cert\") pod \"maas-controller-7d8bb9d654-z82dz\" (UID: \"669da265-2e72-437a-b0c5-19aa2d470b69\") " pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:06:19.199600 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.199548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669da265-2e72-437a-b0c5-19aa2d470b69-metrics-certs\") pod \"maas-controller-7d8bb9d654-z82dz\" (UID: \"669da265-2e72-437a-b0c5-19aa2d470b69\") " pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:06:19.199797 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.199599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fznh\" (UniqueName: \"kubernetes.io/projected/669da265-2e72-437a-b0c5-19aa2d470b69-kube-api-access-6fznh\") pod \"maas-controller-7d8bb9d654-z82dz\" (UID: \"669da265-2e72-437a-b0c5-19aa2d470b69\") " pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:06:19.202171 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.202144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/669da265-2e72-437a-b0c5-19aa2d470b69-webhook-cert\") pod \"maas-controller-7d8bb9d654-z82dz\" (UID: \"669da265-2e72-437a-b0c5-19aa2d470b69\") " pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:06:19.202283 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.202268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669da265-2e72-437a-b0c5-19aa2d470b69-metrics-certs\") pod \"maas-controller-7d8bb9d654-z82dz\" (UID: \"669da265-2e72-437a-b0c5-19aa2d470b69\") " pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:06:19.207673 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.207652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fznh\" (UniqueName: \"kubernetes.io/projected/669da265-2e72-437a-b0c5-19aa2d470b69-kube-api-access-6fznh\") pod \"maas-controller-7d8bb9d654-z82dz\" (UID: \"669da265-2e72-437a-b0c5-19aa2d470b69\") " pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:06:19.360836 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.360757    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:06:19.603194 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.603161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-metrics-certs\") pod \"maas-controller-f58485c69-grxjz\" (UID: \"68ee8855-3c6e-410e-a9d9-826657c2838b\") " pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:19.605599 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.605580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-metrics-certs\") pod \"maas-controller-f58485c69-grxjz\" (UID: \"68ee8855-3c6e-410e-a9d9-826657c2838b\") " pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:19.632805 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.632783    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-z82dz"]
Sep 09 04:06:19.633885 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:06:19.633862    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669da265_2e72_437a_b0c5_19aa2d470b69.slice/crio-8786ed8ee10274d79a5230581d3ccf334ed04f12d85bbaea03b7a7a0a1d5e705 WatchSource:0}: Error finding container 8786ed8ee10274d79a5230581d3ccf334ed04f12d85bbaea03b7a7a0a1d5e705: Status 404 returned error can't find the container with id 8786ed8ee10274d79a5230581d3ccf334ed04f12d85bbaea03b7a7a0a1d5e705
Sep 09 04:06:19.801956 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:19.801924    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:20.082721 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:20.082697    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f58485c69-grxjz"]
Sep 09 04:06:20.084034 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:06:20.084002    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ee8855_3c6e_410e_a9d9_826657c2838b.slice/crio-5918e7aeac495dce213d4d838a9b32b61fb2823afda5614605396a3ea03c7c45 WatchSource:0}: Error finding container 5918e7aeac495dce213d4d838a9b32b61fb2823afda5614605396a3ea03c7c45: Status 404 returned error can't find the container with id 5918e7aeac495dce213d4d838a9b32b61fb2823afda5614605396a3ea03c7c45
Sep 09 04:06:20.141610 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:20.141527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f58485c69-grxjz" event={"ID":"68ee8855-3c6e-410e-a9d9-826657c2838b","Type":"ContainerStarted","Data":"5918e7aeac495dce213d4d838a9b32b61fb2823afda5614605396a3ea03c7c45"}
Sep 09 04:06:20.142786 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:20.142736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bb9d654-z82dz" event={"ID":"669da265-2e72-437a-b0c5-19aa2d470b69","Type":"ContainerStarted","Data":"8786ed8ee10274d79a5230581d3ccf334ed04f12d85bbaea03b7a7a0a1d5e705"}
Sep 09 04:06:22.155847 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:22.155812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f58485c69-grxjz" event={"ID":"68ee8855-3c6e-410e-a9d9-826657c2838b","Type":"ContainerStarted","Data":"847292229d27cb3ccee38f550d25afd13cbe3495eb0d3cd76b490fd7afd05928"}
Sep 09 04:06:22.156212 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:22.155929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:22.172699 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:22.172654    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f58485c69-grxjz" podStartSLOduration=2.59291304 podStartE2EDuration="4.17264058s" totalImagesPullingTime="1.57972754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:06:22.171973684 +0000 UTC m=+503.121664337" watchObservedRunningTime="2026-09-09 04:06:22.17264058 +0000 UTC m=+503.122331234"
Sep 09 04:06:23.163927 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:23.163885    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bb9d654-z82dz" event={"ID":"669da265-2e72-437a-b0c5-19aa2d470b69","Type":"ContainerStarted","Data":"c586d9f972ae7a30589082f2e92eef858826325676bb4fc0525548ffb5a52355"}
Sep 09 04:06:23.164347 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:23.164118    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:06:23.181628 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:23.181586    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d8bb9d654-z82dz" podStartSLOduration=1.49945424 podStartE2EDuration="4.181573712s" totalImagesPullingTime="2.682119472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:06:23.180461531 +0000 UTC m=+504.130152184" watchObservedRunningTime="2026-09-09 04:06:23.181573712 +0000 UTC m=+504.131264367"
Sep 09 04:06:33.166153 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:33.166123    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:34.171700 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:34.171670    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:06:34.214307 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:34.214273    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f58485c69-grxjz"]
Sep 09 04:06:34.215062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:34.215029    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f58485c69-grxjz" podUID="68ee8855-3c6e-410e-a9d9-826657c2838b" containerName="manager" containerID="cri-o://847292229d27cb3ccee38f550d25afd13cbe3495eb0d3cd76b490fd7afd05928" gracePeriod=30
Sep 09 04:06:34.456743 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:34.456722    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:34.526554 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:34.526529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-webhook-cert\") pod \"68ee8855-3c6e-410e-a9d9-826657c2838b\" (UID: \"68ee8855-3c6e-410e-a9d9-826657c2838b\") "
Sep 09 04:06:34.526690 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:34.526569    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-metrics-certs\") pod \"68ee8855-3c6e-410e-a9d9-826657c2838b\" (UID: \"68ee8855-3c6e-410e-a9d9-826657c2838b\") "
Sep 09 04:06:34.526690 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:34.526592    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q96ss\" (UniqueName: \"kubernetes.io/projected/68ee8855-3c6e-410e-a9d9-826657c2838b-kube-api-access-q96ss\") pod \"68ee8855-3c6e-410e-a9d9-826657c2838b\" (UID: \"68ee8855-3c6e-410e-a9d9-826657c2838b\") "
Sep 09 04:06:34.528842 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:34.528809    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "68ee8855-3c6e-410e-a9d9-826657c2838b" (UID: "68ee8855-3c6e-410e-a9d9-826657c2838b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:06:34.528947 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:34.528872    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "68ee8855-3c6e-410e-a9d9-826657c2838b" (UID: "68ee8855-3c6e-410e-a9d9-826657c2838b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:06:34.528947 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:34.528891    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ee8855-3c6e-410e-a9d9-826657c2838b-kube-api-access-q96ss" (OuterVolumeSpecName: "kube-api-access-q96ss") pod "68ee8855-3c6e-410e-a9d9-826657c2838b" (UID: "68ee8855-3c6e-410e-a9d9-826657c2838b"). InnerVolumeSpecName "kube-api-access-q96ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:06:34.628173 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:34.628146    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-webhook-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:06:34.628173 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:34.628173    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ee8855-3c6e-410e-a9d9-826657c2838b-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:06:34.628394 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:34.628185    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q96ss\" (UniqueName: \"kubernetes.io/projected/68ee8855-3c6e-410e-a9d9-826657c2838b-kube-api-access-q96ss\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:06:35.233409 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:35.233376    2580 generic.go:356] "Generic (PLEG): container finished" podID="68ee8855-3c6e-410e-a9d9-826657c2838b" containerID="847292229d27cb3ccee38f550d25afd13cbe3495eb0d3cd76b490fd7afd05928" exitCode=0
Sep 09 04:06:35.233909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:35.233449    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f58485c69-grxjz"
Sep 09 04:06:35.233909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:35.233466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f58485c69-grxjz" event={"ID":"68ee8855-3c6e-410e-a9d9-826657c2838b","Type":"ContainerDied","Data":"847292229d27cb3ccee38f550d25afd13cbe3495eb0d3cd76b490fd7afd05928"}
Sep 09 04:06:35.233909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:35.233548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f58485c69-grxjz" event={"ID":"68ee8855-3c6e-410e-a9d9-826657c2838b","Type":"ContainerDied","Data":"5918e7aeac495dce213d4d838a9b32b61fb2823afda5614605396a3ea03c7c45"}
Sep 09 04:06:35.233909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:35.233571    2580 scope.go:117] "RemoveContainer" containerID="847292229d27cb3ccee38f550d25afd13cbe3495eb0d3cd76b490fd7afd05928"
Sep 09 04:06:35.245659 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:35.245621    2580 scope.go:117] "RemoveContainer" containerID="847292229d27cb3ccee38f550d25afd13cbe3495eb0d3cd76b490fd7afd05928"
Sep 09 04:06:35.246285 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:06:35.246264    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847292229d27cb3ccee38f550d25afd13cbe3495eb0d3cd76b490fd7afd05928\": container with ID starting with 847292229d27cb3ccee38f550d25afd13cbe3495eb0d3cd76b490fd7afd05928 not found: ID does not exist" containerID="847292229d27cb3ccee38f550d25afd13cbe3495eb0d3cd76b490fd7afd05928"
Sep 09 04:06:35.246384 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:35.246295    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847292229d27cb3ccee38f550d25afd13cbe3495eb0d3cd76b490fd7afd05928"} err="failed to get container status \"847292229d27cb3ccee38f550d25afd13cbe3495eb0d3cd76b490fd7afd05928\": rpc error: code = NotFound desc = could not find container \"847292229d27cb3ccee38f550d25afd13cbe3495eb0d3cd76b490fd7afd05928\": container with ID starting with 847292229d27cb3ccee38f550d25afd13cbe3495eb0d3cd76b490fd7afd05928 not found: ID does not exist"
Sep 09 04:06:35.256370 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:35.256345    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f58485c69-grxjz"]
Sep 09 04:06:35.259737 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:35.259718    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f58485c69-grxjz"]
Sep 09 04:06:35.594995 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:35.594964    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68ee8855-3c6e-410e-a9d9-826657c2838b" path="/var/lib/kubelet/pods/68ee8855-3c6e-410e-a9d9-826657c2838b/volumes"
Sep 09 04:06:58.737363 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:58.737267    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-69989b7c55-tks28"]
Sep 09 04:06:58.737939 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:58.737748    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68ee8855-3c6e-410e-a9d9-826657c2838b" containerName="manager"
Sep 09 04:06:58.737939 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:58.737786    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="68ee8855-3c6e-410e-a9d9-826657c2838b" containerName="manager"
Sep 09 04:06:58.737939 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:58.737913    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="68ee8855-3c6e-410e-a9d9-826657c2838b" containerName="manager"
Sep 09 04:06:58.741223 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:58.741203    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:58.743680 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:58.743655    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 04:06:58.743680 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:58.743675    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 04:06:58.743954 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:58.743940    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-thdks\""
Sep 09 04:06:58.751349 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:58.751328    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-69989b7c55-tks28"]
Sep 09 04:06:58.931418 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:58.931378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-openshift-service-ca\") pod \"maas-api-69989b7c55-tks28\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") " pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:58.931595 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:58.931435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-maas-api-tls\") pod \"maas-api-69989b7c55-tks28\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") " pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:58.931595 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:58.931564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m47w\" (UniqueName: \"kubernetes.io/projected/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-kube-api-access-4m47w\") pod \"maas-api-69989b7c55-tks28\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") " pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:58.931711 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:58.931664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-metrics-certs\") pod \"maas-api-69989b7c55-tks28\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") " pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:59.033119 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.033026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-maas-api-tls\") pod \"maas-api-69989b7c55-tks28\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") " pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:59.033119 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.033089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m47w\" (UniqueName: \"kubernetes.io/projected/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-kube-api-access-4m47w\") pod \"maas-api-69989b7c55-tks28\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") " pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:59.033358 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.033149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-metrics-certs\") pod \"maas-api-69989b7c55-tks28\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") " pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:59.033358 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.033193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-openshift-service-ca\") pod \"maas-api-69989b7c55-tks28\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") " pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:59.033358 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:06:59.033195    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 09 04:06:59.033358 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:06:59.033300    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-maas-api-tls podName:436875c8-aed4-4d1d-bcf8-57eaa9a84f88 nodeName:}" failed. No retries permitted until 2026-09-09 04:06:59.533279433 +0000 UTC m=+540.482970071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-maas-api-tls") pod "maas-api-69989b7c55-tks28" (UID: "436875c8-aed4-4d1d-bcf8-57eaa9a84f88") : secret "maas-api-serving-cert" not found
Sep 09 04:06:59.033867 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.033847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-openshift-service-ca\") pod \"maas-api-69989b7c55-tks28\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") " pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:59.035950 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.035921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-metrics-certs\") pod \"maas-api-69989b7c55-tks28\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") " pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:59.042746 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.042720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m47w\" (UniqueName: \"kubernetes.io/projected/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-kube-api-access-4m47w\") pod \"maas-api-69989b7c55-tks28\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") " pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:59.262786 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.262735    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs"]
Sep 09 04:06:59.266283 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.266250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs"
Sep 09 04:06:59.268587 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.268566    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 04:06:59.268715 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.268568    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2cfl6\""
Sep 09 04:06:59.276558 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.276534    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs"]
Sep 09 04:06:59.436690 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.436652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150e74af-5522-4537-ad9b-e4dbae244cc6-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fkvbs\" (UID: \"150e74af-5522-4537-ad9b-e4dbae244cc6\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs"
Sep 09 04:06:59.436899 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.436711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhq7x\" (UniqueName: \"kubernetes.io/projected/150e74af-5522-4537-ad9b-e4dbae244cc6-kube-api-access-hhq7x\") pod \"payload-pre-processing-59fbc47bf8-fkvbs\" (UID: \"150e74af-5522-4537-ad9b-e4dbae244cc6\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs"
Sep 09 04:06:59.538023 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.537979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-maas-api-tls\") pod \"maas-api-69989b7c55-tks28\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") " pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:59.538218 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.538039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150e74af-5522-4537-ad9b-e4dbae244cc6-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fkvbs\" (UID: \"150e74af-5522-4537-ad9b-e4dbae244cc6\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs"
Sep 09 04:06:59.538218 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.538070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhq7x\" (UniqueName: \"kubernetes.io/projected/150e74af-5522-4537-ad9b-e4dbae244cc6-kube-api-access-hhq7x\") pod \"payload-pre-processing-59fbc47bf8-fkvbs\" (UID: \"150e74af-5522-4537-ad9b-e4dbae244cc6\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs"
Sep 09 04:06:59.540571 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.540534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-maas-api-tls\") pod \"maas-api-69989b7c55-tks28\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") " pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:59.540848 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.540831    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 04:06:59.548006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.547974    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhq7x\" (UniqueName: \"kubernetes.io/projected/150e74af-5522-4537-ad9b-e4dbae244cc6-kube-api-access-hhq7x\") pod \"payload-pre-processing-59fbc47bf8-fkvbs\" (UID: \"150e74af-5522-4537-ad9b-e4dbae244cc6\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs"
Sep 09 04:06:59.548988 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.548963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150e74af-5522-4537-ad9b-e4dbae244cc6-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fkvbs\" (UID: \"150e74af-5522-4537-ad9b-e4dbae244cc6\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs"
Sep 09 04:06:59.582592 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.582566    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2cfl6\""
Sep 09 04:06:59.590096 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.590070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs"
Sep 09 04:06:59.656675 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.656649    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-thdks\""
Sep 09 04:06:59.664245 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.664218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:06:59.867279 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.867253    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs"]
Sep 09 04:06:59.868130 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:06:59.868112    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150e74af_5522_4537_ad9b_e4dbae244cc6.slice/crio-5d4ac3673129b19f6289258b2176873c2a02f8776b19aae91a6cd5bf0d195b68 WatchSource:0}: Error finding container 5d4ac3673129b19f6289258b2176873c2a02f8776b19aae91a6cd5bf0d195b68: Status 404 returned error can't find the container with id 5d4ac3673129b19f6289258b2176873c2a02f8776b19aae91a6cd5bf0d195b68
Sep 09 04:06:59.940667 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:06:59.940636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-69989b7c55-tks28"]
Sep 09 04:06:59.942046 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:06:59.942021    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436875c8_aed4_4d1d_bcf8_57eaa9a84f88.slice/crio-f6e9a61632c666ae27e317bb1345c79fa18999ed84cc8c361f5d7bba9708f114 WatchSource:0}: Error finding container f6e9a61632c666ae27e317bb1345c79fa18999ed84cc8c361f5d7bba9708f114: Status 404 returned error can't find the container with id f6e9a61632c666ae27e317bb1345c79fa18999ed84cc8c361f5d7bba9708f114
Sep 09 04:07:00.369217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:00.369182    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28" event={"ID":"436875c8-aed4-4d1d-bcf8-57eaa9a84f88","Type":"ContainerStarted","Data":"f6e9a61632c666ae27e317bb1345c79fa18999ed84cc8c361f5d7bba9708f114"}
Sep 09 04:07:00.370401 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:00.370380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs" event={"ID":"150e74af-5522-4537-ad9b-e4dbae244cc6","Type":"ContainerStarted","Data":"5d4ac3673129b19f6289258b2176873c2a02f8776b19aae91a6cd5bf0d195b68"}
Sep 09 04:07:02.385701 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:02.385665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs" event={"ID":"150e74af-5522-4537-ad9b-e4dbae244cc6","Type":"ContainerStarted","Data":"1873075e632d1854852b05287c689dcc4e70fb62c7608792975b1d568bd63f02"}
Sep 09 04:07:02.386146 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:02.385750    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs"
Sep 09 04:07:02.404782 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:02.404722    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs" podStartSLOduration=1.7176449900000001 podStartE2EDuration="3.404698385s" totalImagesPullingTime="1.687053395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:07:02.403410509 +0000 UTC m=+543.353101163" watchObservedRunningTime="2026-09-09 04:07:02.404698385 +0000 UTC m=+543.354389039"
Sep 09 04:07:04.401237 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:04.401204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28" event={"ID":"436875c8-aed4-4d1d-bcf8-57eaa9a84f88","Type":"ContainerStarted","Data":"9de7f349f07c39cb32989003cf66defa4d0bd5d39da9b1ce10562da94d801dc4"}
Sep 09 04:07:04.401731 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:04.401295    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:07:04.418334 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:04.418281    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28" podStartSLOduration=2.292120447 podStartE2EDuration="6.418264793s" totalImagesPullingTime="4.126144346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:07:04.417423175 +0000 UTC m=+545.367113830" watchObservedRunningTime="2026-09-09 04:07:04.418264793 +0000 UTC m=+545.367955447"
Sep 09 04:07:10.412739 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:10.412710    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:07:13.394904 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:13.394875    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkvbs"
Sep 09 04:07:28.273889 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.273846    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"]
Sep 09 04:07:28.277477 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.277456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.280882 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.280861    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 04:07:28.280988 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.280860    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 04:07:28.280988 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.280862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6qgzz\""
Sep 09 04:07:28.280988 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.280862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 09 04:07:28.285538 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.285514    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"]
Sep 09 04:07:28.376334 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.376294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.376334 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.376334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.376569 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.376354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.376569 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.376372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.376569 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.376461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7szz\" (UniqueName: \"kubernetes.io/projected/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-kube-api-access-f7szz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.376569 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.376549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.477521 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.477480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7szz\" (UniqueName: \"kubernetes.io/projected/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-kube-api-access-f7szz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.477724 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.477565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.477724 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.477600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.477724 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.477626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.477724 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.477649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.477724 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.477676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.478079 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.478054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.478150 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.478094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.478193 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.478160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.480155 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.480135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.480281 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.480264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.485181 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.485160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7szz\" (UniqueName: \"kubernetes.io/projected/5ea9cce7-b914-409b-a2f4-d9a2ef99ac93-kube-api-access-f7szz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg\" (UID: \"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.587873 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.587842    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:28.862882 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:28.862858    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"]
Sep 09 04:07:28.864958 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:07:28.864929    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea9cce7_b914_409b_a2f4_d9a2ef99ac93.slice/crio-99bdd701cc6ee5eebb4242025a02bb8efc2d1a02563bbeec2c184282c0b9dd7e WatchSource:0}: Error finding container 99bdd701cc6ee5eebb4242025a02bb8efc2d1a02563bbeec2c184282c0b9dd7e: Status 404 returned error can't find the container with id 99bdd701cc6ee5eebb4242025a02bb8efc2d1a02563bbeec2c184282c0b9dd7e
Sep 09 04:07:29.548193 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:29.548154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg" event={"ID":"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93","Type":"ContainerStarted","Data":"99bdd701cc6ee5eebb4242025a02bb8efc2d1a02563bbeec2c184282c0b9dd7e"}
Sep 09 04:07:30.555700 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:30.555667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg" event={"ID":"5ea9cce7-b914-409b-a2f4-d9a2ef99ac93","Type":"ContainerStarted","Data":"3bdbc0078597e890cd5c755759786566da301b7f0dd58f57283473719547c72d"}
Sep 09 04:07:30.574544 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:30.574496    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg" podStartSLOduration=1.428941403 podStartE2EDuration="2.574480379s" totalImagesPullingTime="1.145538976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:07:30.571906619 +0000 UTC m=+571.521597276" watchObservedRunningTime="2026-09-09 04:07:30.574480379 +0000 UTC m=+571.524171033"
Sep 09 04:07:38.588675 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:38.588624    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:38.589219 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:38.588723    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:38.602062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:38.602039    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:38.613388 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:38.613367    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg"
Sep 09 04:07:59.521019 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:59.520990    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:07:59.521510 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:07:59.520994    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:08:21.153314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.153234    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"]
Sep 09 04:08:21.157587 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.157571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.160126 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.160103    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 04:08:21.166119 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.166098    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"]
Sep 09 04:08:21.240206 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.240157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.240380 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.240220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.240380 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.240252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j8h6\" (UniqueName: \"kubernetes.io/projected/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-kube-api-access-2j8h6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.240380 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.240357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.240503 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.240396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.240544 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.240503    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.341184 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.341143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.341184 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.341186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.341415 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.341252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.341415 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.341292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.341415 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.341318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.341564 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.341404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j8h6\" (UniqueName: \"kubernetes.io/projected/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-kube-api-access-2j8h6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.341933 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.341911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.342065 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.342040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.342368 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.342345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.349792 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.349736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.350013 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.349993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.352112 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.352090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j8h6\" (UniqueName: \"kubernetes.io/projected/f7433f3e-fee6-4f46-be9c-519d7f3bafc1-kube-api-access-2j8h6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-pjw44\" (UID: \"f7433f3e-fee6-4f46-be9c-519d7f3bafc1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.469207 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.469118    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:21.745160 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.745137    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"]
Sep 09 04:08:21.747521 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:08:21.747496    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7433f3e_fee6_4f46_be9c_519d7f3bafc1.slice/crio-434dbad9300acd91d7bca5c3884fa987d077a2918c2fa709bd71db3307800239 WatchSource:0}: Error finding container 434dbad9300acd91d7bca5c3884fa987d077a2918c2fa709bd71db3307800239: Status 404 returned error can't find the container with id 434dbad9300acd91d7bca5c3884fa987d077a2918c2fa709bd71db3307800239
Sep 09 04:08:21.748935 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.748918    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 04:08:21.838352 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:21.838312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44" event={"ID":"f7433f3e-fee6-4f46-be9c-519d7f3bafc1","Type":"ContainerStarted","Data":"434dbad9300acd91d7bca5c3884fa987d077a2918c2fa709bd71db3307800239"}
Sep 09 04:08:22.848057 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:22.848019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44" event={"ID":"f7433f3e-fee6-4f46-be9c-519d7f3bafc1","Type":"ContainerStarted","Data":"4a581339a564d47370efcee8cf2e702cb22c2bd8b7e6bcaf41f22a430ad1218d"}
Sep 09 04:08:22.865053 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:22.864985    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44" podStartSLOduration=1.686092903 podStartE2EDuration="1.864945175s" totalImagesPullingTime="178.852272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:08:22.864213545 +0000 UTC m=+623.813904201" watchObservedRunningTime="2026-09-09 04:08:22.864945175 +0000 UTC m=+623.814635832"
Sep 09 04:08:31.469257 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:31.469220    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:31.469861 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:31.469311    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:31.481874 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:31.481854    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:08:31.910461 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:08:31.910433    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-pjw44"
Sep 09 04:10:16.028907 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:16.028871    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77448c7dc7-whv4f"]
Sep 09 04:10:16.032391 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:16.032373    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77448c7dc7-whv4f"
Sep 09 04:10:16.040046 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:16.039759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77448c7dc7-whv4f"]
Sep 09 04:10:16.130048 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:16.130005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d07af248-a22d-4433-878d-594d325a1e6c-tls-cert\") pod \"authorino-77448c7dc7-whv4f\" (UID: \"d07af248-a22d-4433-878d-594d325a1e6c\") " pod="kuadrant-system/authorino-77448c7dc7-whv4f"
Sep 09 04:10:16.130216 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:16.130078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wptcv\" (UniqueName: \"kubernetes.io/projected/d07af248-a22d-4433-878d-594d325a1e6c-kube-api-access-wptcv\") pod \"authorino-77448c7dc7-whv4f\" (UID: \"d07af248-a22d-4433-878d-594d325a1e6c\") " pod="kuadrant-system/authorino-77448c7dc7-whv4f"
Sep 09 04:10:16.231542 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:16.231511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wptcv\" (UniqueName: \"kubernetes.io/projected/d07af248-a22d-4433-878d-594d325a1e6c-kube-api-access-wptcv\") pod \"authorino-77448c7dc7-whv4f\" (UID: \"d07af248-a22d-4433-878d-594d325a1e6c\") " pod="kuadrant-system/authorino-77448c7dc7-whv4f"
Sep 09 04:10:16.231720 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:16.231571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d07af248-a22d-4433-878d-594d325a1e6c-tls-cert\") pod \"authorino-77448c7dc7-whv4f\" (UID: \"d07af248-a22d-4433-878d-594d325a1e6c\") " pod="kuadrant-system/authorino-77448c7dc7-whv4f"
Sep 09 04:10:16.234172 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:16.234154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d07af248-a22d-4433-878d-594d325a1e6c-tls-cert\") pod \"authorino-77448c7dc7-whv4f\" (UID: \"d07af248-a22d-4433-878d-594d325a1e6c\") " pod="kuadrant-system/authorino-77448c7dc7-whv4f"
Sep 09 04:10:16.239898 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:16.239881    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wptcv\" (UniqueName: \"kubernetes.io/projected/d07af248-a22d-4433-878d-594d325a1e6c-kube-api-access-wptcv\") pod \"authorino-77448c7dc7-whv4f\" (UID: \"d07af248-a22d-4433-878d-594d325a1e6c\") " pod="kuadrant-system/authorino-77448c7dc7-whv4f"
Sep 09 04:10:16.343518 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:16.343493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77448c7dc7-whv4f"
Sep 09 04:10:16.622148 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:16.622117    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77448c7dc7-whv4f"]
Sep 09 04:10:16.623231 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:10:16.623193    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd07af248_a22d_4433_878d_594d325a1e6c.slice/crio-c461b5dfb537b83051d16fe3cc34ade7ad6c471af9b30dd55a9c2d8494eb35d7 WatchSource:0}: Error finding container c461b5dfb537b83051d16fe3cc34ade7ad6c471af9b30dd55a9c2d8494eb35d7: Status 404 returned error can't find the container with id c461b5dfb537b83051d16fe3cc34ade7ad6c471af9b30dd55a9c2d8494eb35d7
Sep 09 04:10:17.488868 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:17.488831    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77448c7dc7-whv4f" event={"ID":"d07af248-a22d-4433-878d-594d325a1e6c","Type":"ContainerStarted","Data":"b040be7866e13991fdbf8c5e885ae6ca7718460657ae952d1827ea896630c984"}
Sep 09 04:10:17.489207 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:17.488887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77448c7dc7-whv4f" event={"ID":"d07af248-a22d-4433-878d-594d325a1e6c","Type":"ContainerStarted","Data":"c461b5dfb537b83051d16fe3cc34ade7ad6c471af9b30dd55a9c2d8494eb35d7"}
Sep 09 04:10:17.506968 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:17.506926    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-77448c7dc7-whv4f" podStartSLOduration=0.735450691 podStartE2EDuration="1.506909176s" totalImagesPullingTime="771.458485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:10:17.504641543 +0000 UTC m=+738.454332208" watchObservedRunningTime="2026-09-09 04:10:17.506909176 +0000 UTC m=+738.456599830"
Sep 09 04:10:17.534575 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:17.534540    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6cb59c5b-vk6t6"]
Sep 09 04:10:17.534781 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:17.534745    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6" podUID="8f094812-8368-4712-9c51-2cbcebd2917c" containerName="authorino" containerID="cri-o://62914ffa8e4421a1ed48d0e2be4b5222265581d5b17f9b76344539103b023e07" gracePeriod=30
Sep 09 04:10:17.804304 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:17.804274    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6"
Sep 09 04:10:17.948570 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:17.948470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4db8\" (UniqueName: \"kubernetes.io/projected/8f094812-8368-4712-9c51-2cbcebd2917c-kube-api-access-r4db8\") pod \"8f094812-8368-4712-9c51-2cbcebd2917c\" (UID: \"8f094812-8368-4712-9c51-2cbcebd2917c\") "
Sep 09 04:10:17.948570 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:17.948530    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f094812-8368-4712-9c51-2cbcebd2917c-tls-cert\") pod \"8f094812-8368-4712-9c51-2cbcebd2917c\" (UID: \"8f094812-8368-4712-9c51-2cbcebd2917c\") "
Sep 09 04:10:17.952930 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:17.952887    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f094812-8368-4712-9c51-2cbcebd2917c-kube-api-access-r4db8" (OuterVolumeSpecName: "kube-api-access-r4db8") pod "8f094812-8368-4712-9c51-2cbcebd2917c" (UID: "8f094812-8368-4712-9c51-2cbcebd2917c"). InnerVolumeSpecName "kube-api-access-r4db8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:10:17.962721 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:17.962680    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f094812-8368-4712-9c51-2cbcebd2917c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8f094812-8368-4712-9c51-2cbcebd2917c" (UID: "8f094812-8368-4712-9c51-2cbcebd2917c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:10:18.050034 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:18.049996    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4db8\" (UniqueName: \"kubernetes.io/projected/8f094812-8368-4712-9c51-2cbcebd2917c-kube-api-access-r4db8\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:10:18.050034 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:18.050029    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f094812-8368-4712-9c51-2cbcebd2917c-tls-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:10:18.496066 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:18.496035    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f094812-8368-4712-9c51-2cbcebd2917c" containerID="62914ffa8e4421a1ed48d0e2be4b5222265581d5b17f9b76344539103b023e07" exitCode=0
Sep 09 04:10:18.496457 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:18.496081    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6"
Sep 09 04:10:18.496457 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:18.496122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6" event={"ID":"8f094812-8368-4712-9c51-2cbcebd2917c","Type":"ContainerDied","Data":"62914ffa8e4421a1ed48d0e2be4b5222265581d5b17f9b76344539103b023e07"}
Sep 09 04:10:18.496457 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:18.496168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6cb59c5b-vk6t6" event={"ID":"8f094812-8368-4712-9c51-2cbcebd2917c","Type":"ContainerDied","Data":"b2a8f8ec9c840339b6f9e8f95eb9cc533a1122e98db65386a04567c7bc498f8f"}
Sep 09 04:10:18.496457 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:18.496189    2580 scope.go:117] "RemoveContainer" containerID="62914ffa8e4421a1ed48d0e2be4b5222265581d5b17f9b76344539103b023e07"
Sep 09 04:10:18.507430 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:18.507410    2580 scope.go:117] "RemoveContainer" containerID="62914ffa8e4421a1ed48d0e2be4b5222265581d5b17f9b76344539103b023e07"
Sep 09 04:10:18.507800 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:10:18.507743    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62914ffa8e4421a1ed48d0e2be4b5222265581d5b17f9b76344539103b023e07\": container with ID starting with 62914ffa8e4421a1ed48d0e2be4b5222265581d5b17f9b76344539103b023e07 not found: ID does not exist" containerID="62914ffa8e4421a1ed48d0e2be4b5222265581d5b17f9b76344539103b023e07"
Sep 09 04:10:18.507933 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:18.507811    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62914ffa8e4421a1ed48d0e2be4b5222265581d5b17f9b76344539103b023e07"} err="failed to get container status \"62914ffa8e4421a1ed48d0e2be4b5222265581d5b17f9b76344539103b023e07\": rpc error: code = NotFound desc = could not find container \"62914ffa8e4421a1ed48d0e2be4b5222265581d5b17f9b76344539103b023e07\": container with ID starting with 62914ffa8e4421a1ed48d0e2be4b5222265581d5b17f9b76344539103b023e07 not found: ID does not exist"
Sep 09 04:10:18.519398 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:18.519373    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6cb59c5b-vk6t6"]
Sep 09 04:10:18.528704 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:18.528670    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6cb59c5b-vk6t6"]
Sep 09 04:10:19.596582 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:19.596551    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f094812-8368-4712-9c51-2cbcebd2917c" path="/var/lib/kubelet/pods/8f094812-8368-4712-9c51-2cbcebd2917c/volumes"
Sep 09 04:10:35.864310 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.864271    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"]
Sep 09 04:10:35.864847 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.864831    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f094812-8368-4712-9c51-2cbcebd2917c" containerName="authorino"
Sep 09 04:10:35.864902 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.864851    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f094812-8368-4712-9c51-2cbcebd2917c" containerName="authorino"
Sep 09 04:10:35.864960 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.864950    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f094812-8368-4712-9c51-2cbcebd2917c" containerName="authorino"
Sep 09 04:10:35.868445 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.868424    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:35.871318 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.871054    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-85732c-openshift-default-dockercfg-t6z85\""
Sep 09 04:10:35.871318 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.871285    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 04:10:35.880350 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.880329    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"]
Sep 09 04:10:35.900154 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.900126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istiod-ca-cert\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:35.900285 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.900168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istio-token\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:35.900285 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.900194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-credential-socket\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:35.900285 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.900216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istio-envoy\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:35.900285 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.900241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istio-podinfo\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:35.900285 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.900268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istio-data\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:35.900547 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.900309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvnw4\" (UniqueName: \"kubernetes.io/projected/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-kube-api-access-hvnw4\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:35.900547 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.900377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-workload-certs\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:35.900547 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:35.900463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-workload-socket\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.001348 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.001315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-workload-socket\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.001348 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.001354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istiod-ca-cert\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.001600 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.001373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istio-token\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.001600 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.001400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-credential-socket\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.001600 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.001425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istio-envoy\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.001600 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.001450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istio-podinfo\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.001600 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.001478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istio-data\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.001901 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.001700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvnw4\" (UniqueName: \"kubernetes.io/projected/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-kube-api-access-hvnw4\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.001901 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.001817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-workload-certs\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.001901 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.001867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istio-data\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.001901 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.001887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-credential-socket\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.002143 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.002119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-workload-socket\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.002211 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.002137    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-workload-certs\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.002211 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.002155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istiod-ca-cert\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.004625 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.004592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istio-podinfo\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.005276 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.005214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istio-envoy\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.013085 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.013036    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-istio-token\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.013802 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.013748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvnw4\" (UniqueName: \"kubernetes.io/projected/f5ec7505-fc62-4ce0-9548-28b278a1c2cc-kube-api-access-hvnw4\") pod \"e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp\" (UID: \"f5ec7505-fc62-4ce0-9548-28b278a1c2cc\") " pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.183971 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.183902    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:36.467851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.467823    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"]
Sep 09 04:10:36.469080 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:10:36.469049    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ec7505_fc62_4ce0_9548_28b278a1c2cc.slice/crio-e5230d3ec1410b5974d7d86846a7c9925841c285f4293cec6a01102d7ee5fcf0 WatchSource:0}: Error finding container e5230d3ec1410b5974d7d86846a7c9925841c285f4293cec6a01102d7ee5fcf0: Status 404 returned error can't find the container with id e5230d3ec1410b5974d7d86846a7c9925841c285f4293cec6a01102d7ee5fcf0
Sep 09 04:10:36.597869 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:36.597826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp" event={"ID":"f5ec7505-fc62-4ce0-9548-28b278a1c2cc","Type":"ContainerStarted","Data":"e5230d3ec1410b5974d7d86846a7c9925841c285f4293cec6a01102d7ee5fcf0"}
Sep 09 04:10:37.470604 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:37.470569    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7826-139895587133104"]
Sep 09 04:10:37.474239 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:37.474222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:37.476624 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:37.476602    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pcbxq\""
Sep 09 04:10:37.481151 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:37.481105    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7826-139895587133104"]
Sep 09 04:10:37.514871 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:37.514832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kwcb\" (UniqueName: \"kubernetes.io/projected/b9065650-2dfb-4f3e-8345-df5108c8b445-kube-api-access-8kwcb\") pod \"test-curl-7826-139895587133104\" (UID: \"b9065650-2dfb-4f3e-8345-df5108c8b445\") " pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:37.616234 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:37.616195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kwcb\" (UniqueName: \"kubernetes.io/projected/b9065650-2dfb-4f3e-8345-df5108c8b445-kube-api-access-8kwcb\") pod \"test-curl-7826-139895587133104\" (UID: \"b9065650-2dfb-4f3e-8345-df5108c8b445\") " pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:37.626836 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:37.626809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kwcb\" (UniqueName: \"kubernetes.io/projected/b9065650-2dfb-4f3e-8345-df5108c8b445-kube-api-access-8kwcb\") pod \"test-curl-7826-139895587133104\" (UID: \"b9065650-2dfb-4f3e-8345-df5108c8b445\") " pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:37.787876 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:37.787794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:38.071410 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:38.071333    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7826-139895587133104"]
Sep 09 04:10:38.074739 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:10:38.074711    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9065650_2dfb_4f3e_8345_df5108c8b445.slice/crio-b1af9b4fe24f8a88967890f576257cf74500a9b91313c8821f31c1cc87c1dcf0 WatchSource:0}: Error finding container b1af9b4fe24f8a88967890f576257cf74500a9b91313c8821f31c1cc87c1dcf0: Status 404 returned error can't find the container with id b1af9b4fe24f8a88967890f576257cf74500a9b91313c8821f31c1cc87c1dcf0
Sep 09 04:10:38.614228 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:38.614192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7826-139895587133104" event={"ID":"b9065650-2dfb-4f3e-8345-df5108c8b445","Type":"ContainerStarted","Data":"b1af9b4fe24f8a88967890f576257cf74500a9b91313c8821f31c1cc87c1dcf0"}
Sep 09 04:10:39.319874 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:39.319834    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:10:39.319968 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:39.319922    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:10:39.320036 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:39.319964    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:10:39.623616 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:39.623576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp" event={"ID":"f5ec7505-fc62-4ce0-9548-28b278a1c2cc","Type":"ContainerStarted","Data":"d5f54b17f134ed5f1c1201f4c8dcabcd65cb2907dc3fc6003636d0e951c56f34"}
Sep 09 04:10:39.625297 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:39.625271    2580 generic.go:356] "Generic (PLEG): container finished" podID="b9065650-2dfb-4f3e-8345-df5108c8b445" containerID="667e9b2f4803110ccb74f66f66951b00d4c25355c83af7a013f219d016590e5b" exitCode=0
Sep 09 04:10:39.625435 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:39.625313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7826-139895587133104" event={"ID":"b9065650-2dfb-4f3e-8345-df5108c8b445","Type":"ContainerDied","Data":"667e9b2f4803110ccb74f66f66951b00d4c25355c83af7a013f219d016590e5b"}
Sep 09 04:10:39.649597 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:39.649544    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp" podStartSLOduration=1.8013522690000001 podStartE2EDuration="4.64952926s" totalImagesPullingTime="2.848176991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:10:39.646271347 +0000 UTC m=+760.595962028" watchObservedRunningTime="2026-09-09 04:10:39.64952926 +0000 UTC m=+760.599219915"
Sep 09 04:10:40.184213 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.184163    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:40.189458 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.189424    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:40.631568 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.631539    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:40.632580 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.632540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:10:40.785147 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.785122    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:40.817157 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.817129    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7826-139895587133104_b9065650-2dfb-4f3e-8345-df5108c8b445/test-curl-7826-139895587133104/0.log"
Sep 09 04:10:40.841245 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.841211    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7826-139895587133104"]
Sep 09 04:10:40.846406 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.846383    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7826-139895587133104"]
Sep 09 04:10:40.847233 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.847215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kwcb\" (UniqueName: \"kubernetes.io/projected/b9065650-2dfb-4f3e-8345-df5108c8b445-kube-api-access-8kwcb\") pod \"b9065650-2dfb-4f3e-8345-df5108c8b445\" (UID: \"b9065650-2dfb-4f3e-8345-df5108c8b445\") "
Sep 09 04:10:40.849285 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.849259    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9065650-2dfb-4f3e-8345-df5108c8b445-kube-api-access-8kwcb" (OuterVolumeSpecName: "kube-api-access-8kwcb") pod "b9065650-2dfb-4f3e-8345-df5108c8b445" (UID: "b9065650-2dfb-4f3e-8345-df5108c8b445"). InnerVolumeSpecName "kube-api-access-8kwcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:10:40.948073 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.947994    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kwcb\" (UniqueName: \"kubernetes.io/projected/b9065650-2dfb-4f3e-8345-df5108c8b445-kube-api-access-8kwcb\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:10:40.971145 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.971115    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7826-139895587133104"]
Sep 09 04:10:40.971488 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.971475    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9065650-2dfb-4f3e-8345-df5108c8b445" containerName="test-curl-7826-139895587133104"
Sep 09 04:10:40.971531 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.971490    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9065650-2dfb-4f3e-8345-df5108c8b445" containerName="test-curl-7826-139895587133104"
Sep 09 04:10:40.971566 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.971554    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9065650-2dfb-4f3e-8345-df5108c8b445" containerName="test-curl-7826-139895587133104"
Sep 09 04:10:40.974859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.974841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:40.981269 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:40.981247    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7826-139895587133104"]
Sep 09 04:10:41.048684 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:41.048647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmzhl\" (UniqueName: \"kubernetes.io/projected/215e8301-0f62-4568-aa1a-e1078b31d0ac-kube-api-access-dmzhl\") pod \"test-curl-7826-139895587133104\" (UID: \"215e8301-0f62-4568-aa1a-e1078b31d0ac\") " pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:41.149284 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:41.149241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmzhl\" (UniqueName: \"kubernetes.io/projected/215e8301-0f62-4568-aa1a-e1078b31d0ac-kube-api-access-dmzhl\") pod \"test-curl-7826-139895587133104\" (UID: \"215e8301-0f62-4568-aa1a-e1078b31d0ac\") " pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:41.157387 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:41.157360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmzhl\" (UniqueName: \"kubernetes.io/projected/215e8301-0f62-4568-aa1a-e1078b31d0ac-kube-api-access-dmzhl\") pod \"test-curl-7826-139895587133104\" (UID: \"215e8301-0f62-4568-aa1a-e1078b31d0ac\") " pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:41.286806 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:41.286708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:41.561539 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:41.561516    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7826-139895587133104"]
Sep 09 04:10:41.563943 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:10:41.563918    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod215e8301_0f62_4568_aa1a_e1078b31d0ac.slice/crio-74738be35ea88424a19c121597727296bd7749481f1370b91a19e5416cf8d12d WatchSource:0}: Error finding container 74738be35ea88424a19c121597727296bd7749481f1370b91a19e5416cf8d12d: Status 404 returned error can't find the container with id 74738be35ea88424a19c121597727296bd7749481f1370b91a19e5416cf8d12d
Sep 09 04:10:41.596382 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:41.596355    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9065650-2dfb-4f3e-8345-df5108c8b445" path="/var/lib/kubelet/pods/b9065650-2dfb-4f3e-8345-df5108c8b445/volumes"
Sep 09 04:10:41.639006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:41.638976    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:41.639394 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:41.639015    2580 scope.go:117] "RemoveContainer" containerID="667e9b2f4803110ccb74f66f66951b00d4c25355c83af7a013f219d016590e5b"
Sep 09 04:10:41.640304 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:41.640225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7826-139895587133104" event={"ID":"215e8301-0f62-4568-aa1a-e1078b31d0ac","Type":"ContainerStarted","Data":"74738be35ea88424a19c121597727296bd7749481f1370b91a19e5416cf8d12d"}
Sep 09 04:10:42.656532 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:42.656497    2580 generic.go:356] "Generic (PLEG): container finished" podID="215e8301-0f62-4568-aa1a-e1078b31d0ac" containerID="ffbdcb60c88073511591d37e80abba10485258bf023c5fc50ece7b78e41ea434" exitCode=0
Sep 09 04:10:42.656964 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:42.656592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7826-139895587133104" event={"ID":"215e8301-0f62-4568-aa1a-e1078b31d0ac","Type":"ContainerDied","Data":"ffbdcb60c88073511591d37e80abba10485258bf023c5fc50ece7b78e41ea434"}
Sep 09 04:10:43.770282 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.770245    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"]
Sep 09 04:10:43.775000 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.774978    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:43.777541 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.777518    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-359adf-serving-cert\""
Sep 09 04:10:43.789913 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.789885    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"]
Sep 09 04:10:43.824959 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.824933    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:43.846259 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.846101    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7826-139895587133104_215e8301-0f62-4568-aa1a-e1078b31d0ac/test-curl-7826-139895587133104/0.log"
Sep 09 04:10:43.871395 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.871363    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7826-139895587133104"]
Sep 09 04:10:43.872849 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.872832    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmzhl\" (UniqueName: \"kubernetes.io/projected/215e8301-0f62-4568-aa1a-e1078b31d0ac-kube-api-access-dmzhl\") pod \"215e8301-0f62-4568-aa1a-e1078b31d0ac\" (UID: \"215e8301-0f62-4568-aa1a-e1078b31d0ac\") "
Sep 09 04:10:43.872970 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.872952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-metrics-certs\") pod \"maas-api-e2e-inf-a-359adf-85489bd9c-4cljx\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:43.873036 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.872987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-359adf-85489bd9c-4cljx\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:43.873036 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.873014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlc2w\" (UniqueName: \"kubernetes.io/projected/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-kube-api-access-rlc2w\") pod \"maas-api-e2e-inf-a-359adf-85489bd9c-4cljx\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:43.873149 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.873122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-359adf-85489bd9c-4cljx\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:43.874998 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.874971    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7826-139895587133104"]
Sep 09 04:10:43.875218 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.875198    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215e8301-0f62-4568-aa1a-e1078b31d0ac-kube-api-access-dmzhl" (OuterVolumeSpecName: "kube-api-access-dmzhl") pod "215e8301-0f62-4568-aa1a-e1078b31d0ac" (UID: "215e8301-0f62-4568-aa1a-e1078b31d0ac"). InnerVolumeSpecName "kube-api-access-dmzhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:10:43.973736 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.973646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-359adf-85489bd9c-4cljx\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:43.973736 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.973703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-metrics-certs\") pod \"maas-api-e2e-inf-a-359adf-85489bd9c-4cljx\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:43.973736 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.973736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-359adf-85489bd9c-4cljx\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:43.974015 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.973756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlc2w\" (UniqueName: \"kubernetes.io/projected/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-kube-api-access-rlc2w\") pod \"maas-api-e2e-inf-a-359adf-85489bd9c-4cljx\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:43.974015 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.973842    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmzhl\" (UniqueName: \"kubernetes.io/projected/215e8301-0f62-4568-aa1a-e1078b31d0ac-kube-api-access-dmzhl\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:10:43.974015 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:10:43.973920    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-serving-cert: secret "maas-api-e2e-inf-a-359adf-serving-cert" not found
Sep 09 04:10:43.974157 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:10:43.974018    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-maas-api-tls podName:59ed0e1c-9b27-40e4-9b3c-a235ba8044ae nodeName:}" failed. No retries permitted until 2026-09-09 04:10:44.473994054 +0000 UTC m=+765.423684695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-maas-api-tls") pod "maas-api-e2e-inf-a-359adf-85489bd9c-4cljx" (UID: "59ed0e1c-9b27-40e4-9b3c-a235ba8044ae") : secret "maas-api-e2e-inf-a-359adf-serving-cert" not found
Sep 09 04:10:43.974499 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.974472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-359adf-85489bd9c-4cljx\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:43.976489 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.976467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-metrics-certs\") pod \"maas-api-e2e-inf-a-359adf-85489bd9c-4cljx\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:43.984393 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:43.984370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlc2w\" (UniqueName: \"kubernetes.io/projected/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-kube-api-access-rlc2w\") pod \"maas-api-e2e-inf-a-359adf-85489bd9c-4cljx\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:44.478738 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:44.478691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-359adf-85489bd9c-4cljx\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:44.481653 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:44.481623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-359adf-85489bd9c-4cljx\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:44.672551 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:44.672521    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:44.672737 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:44.672531    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74738be35ea88424a19c121597727296bd7749481f1370b91a19e5416cf8d12d"
Sep 09 04:10:44.686605 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:10:44.686571    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7826-139895587133104\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="215e8301-0f62-4568-aa1a-e1078b31d0ac" pod="opendatahub/test-curl-7826-139895587133104"
Sep 09 04:10:44.689703 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:44.689681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:44.969174 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:44.969147    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"]
Sep 09 04:10:44.971193 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:10:44.971168    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ed0e1c_9b27_40e4_9b3c_a235ba8044ae.slice/crio-8c34e025e1eaf6fa1ff6600b48b39e863cc9f9b95fab15a21fb7cbc820016895 WatchSource:0}: Error finding container 8c34e025e1eaf6fa1ff6600b48b39e863cc9f9b95fab15a21fb7cbc820016895: Status 404 returned error can't find the container with id 8c34e025e1eaf6fa1ff6600b48b39e863cc9f9b95fab15a21fb7cbc820016895
Sep 09 04:10:45.596919 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:45.596892    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="215e8301-0f62-4568-aa1a-e1078b31d0ac" path="/var/lib/kubelet/pods/215e8301-0f62-4568-aa1a-e1078b31d0ac/volumes"
Sep 09 04:10:45.680269 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:45.680235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx" event={"ID":"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae","Type":"ContainerStarted","Data":"cf778f6a9dfcc22047d13d23370860190526217e3c500f187166c4565e1af742"}
Sep 09 04:10:45.680457 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:45.680291    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx" event={"ID":"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae","Type":"ContainerStarted","Data":"8c34e025e1eaf6fa1ff6600b48b39e863cc9f9b95fab15a21fb7cbc820016895"}
Sep 09 04:10:45.680457 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:45.680365    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:45.697936 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:45.697887    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx" podStartSLOduration=2.393482813 podStartE2EDuration="2.697873738s" totalImagesPullingTime="304.390925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:10:45.697113424 +0000 UTC m=+766.646804078" watchObservedRunningTime="2026-09-09 04:10:45.697873738 +0000 UTC m=+766.647564391"
Sep 09 04:10:48.765387 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.765354    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"]
Sep 09 04:10:48.765961 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.765942    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="215e8301-0f62-4568-aa1a-e1078b31d0ac" containerName="test-curl-7826-139895587133104"
Sep 09 04:10:48.766045 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.765964    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="215e8301-0f62-4568-aa1a-e1078b31d0ac" containerName="test-curl-7826-139895587133104"
Sep 09 04:10:48.766107 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.766087    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="215e8301-0f62-4568-aa1a-e1078b31d0ac" containerName="test-curl-7826-139895587133104"
Sep 09 04:10:48.770590 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.770566    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:48.772840 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.772816    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-4ad5fa-serving-cert\""
Sep 09 04:10:48.780095 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.779487    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"]
Sep 09 04:10:48.815547 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.815512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-maas-api-tls\") pod \"maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:48.815734 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.815580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee75250d-6bd0-4db9-adcf-6c7392d33932-openshift-service-ca\") pod \"maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:48.815734 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.815671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmq2m\" (UniqueName: \"kubernetes.io/projected/ee75250d-6bd0-4db9-adcf-6c7392d33932-kube-api-access-nmq2m\") pod \"maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:48.815734 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.815701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-metrics-certs\") pod \"maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:48.916701 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.916663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmq2m\" (UniqueName: \"kubernetes.io/projected/ee75250d-6bd0-4db9-adcf-6c7392d33932-kube-api-access-nmq2m\") pod \"maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:48.916701 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.916704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-metrics-certs\") pod \"maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:48.916982 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.916743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-maas-api-tls\") pod \"maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:48.916982 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.916783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee75250d-6bd0-4db9-adcf-6c7392d33932-openshift-service-ca\") pod \"maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:48.916982 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:10:48.916926    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-serving-cert: secret "maas-api-e2e-autores-4ad5fa-serving-cert" not found
Sep 09 04:10:48.917134 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:10:48.917013    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-maas-api-tls podName:ee75250d-6bd0-4db9-adcf-6c7392d33932 nodeName:}" failed. No retries permitted until 2026-09-09 04:10:49.416993037 +0000 UTC m=+770.366683673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-maas-api-tls") pod "maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7" (UID: "ee75250d-6bd0-4db9-adcf-6c7392d33932") : secret "maas-api-e2e-autores-4ad5fa-serving-cert" not found
Sep 09 04:10:48.917423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.917402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee75250d-6bd0-4db9-adcf-6c7392d33932-openshift-service-ca\") pod \"maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:48.919476 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.919456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-metrics-certs\") pod \"maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:48.931223 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:48.931191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmq2m\" (UniqueName: \"kubernetes.io/projected/ee75250d-6bd0-4db9-adcf-6c7392d33932-kube-api-access-nmq2m\") pod \"maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:49.423340 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:49.422842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-maas-api-tls\") pod \"maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:49.426592 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:49.426531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-maas-api-tls\") pod \"maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:49.685553 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:49.685477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:49.999378 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:49.999349    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"]
Sep 09 04:10:50.000370 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:10:50.000346    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee75250d_6bd0_4db9_adcf_6c7392d33932.slice/crio-1b1a62e525a13ae39254f96e128a2c251c9c9ccaed18468866f9062cf5a1de9d WatchSource:0}: Error finding container 1b1a62e525a13ae39254f96e128a2c251c9c9ccaed18468866f9062cf5a1de9d: Status 404 returned error can't find the container with id 1b1a62e525a13ae39254f96e128a2c251c9c9ccaed18468866f9062cf5a1de9d
Sep 09 04:10:50.452887 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.452854    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"]
Sep 09 04:10:50.457723 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.457701    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:50.460335 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.460312    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-45a82a13-serving-cert\""
Sep 09 04:10:50.467178 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.467158    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"]
Sep 09 04:10:50.540018 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.539982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02f4fe9-af19-4d7e-883c-4201af1d7430-openshift-service-ca\") pod \"maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:50.540247 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.540043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02f4fe9-af19-4d7e-883c-4201af1d7430-maas-api-tls\") pod \"maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:50.540247 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.540099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02f4fe9-af19-4d7e-883c-4201af1d7430-metrics-certs\") pod \"maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:50.540247 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.540163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kckx\" (UniqueName: \"kubernetes.io/projected/d02f4fe9-af19-4d7e-883c-4201af1d7430-kube-api-access-4kckx\") pod \"maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:50.641456 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.641416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02f4fe9-af19-4d7e-883c-4201af1d7430-maas-api-tls\") pod \"maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:50.641456 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.641458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02f4fe9-af19-4d7e-883c-4201af1d7430-metrics-certs\") pod \"maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:50.641659 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.641519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kckx\" (UniqueName: \"kubernetes.io/projected/d02f4fe9-af19-4d7e-883c-4201af1d7430-kube-api-access-4kckx\") pod \"maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:50.641659 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.641573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02f4fe9-af19-4d7e-883c-4201af1d7430-openshift-service-ca\") pod \"maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:50.642341 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.642266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02f4fe9-af19-4d7e-883c-4201af1d7430-openshift-service-ca\") pod \"maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:50.644128 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.644099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02f4fe9-af19-4d7e-883c-4201af1d7430-metrics-certs\") pod \"maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:50.644241 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.644151    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02f4fe9-af19-4d7e-883c-4201af1d7430-maas-api-tls\") pod \"maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:50.649592 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.649567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kckx\" (UniqueName: \"kubernetes.io/projected/d02f4fe9-af19-4d7e-883c-4201af1d7430-kube-api-access-4kckx\") pod \"maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:50.737490 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.737408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7" event={"ID":"ee75250d-6bd0-4db9-adcf-6c7392d33932","Type":"ContainerStarted","Data":"1b1a62e525a13ae39254f96e128a2c251c9c9ccaed18468866f9062cf5a1de9d"}
Sep 09 04:10:50.772711 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:50.772684    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:51.080218 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:51.080184    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"]
Sep 09 04:10:51.081396 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:10:51.081367    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02f4fe9_af19_4d7e_883c_4201af1d7430.slice/crio-7ab25d3bdbf5ce9810601e881cca664ba8bdb1c2d809d77ff5a2965bb25a957f WatchSource:0}: Error finding container 7ab25d3bdbf5ce9810601e881cca664ba8bdb1c2d809d77ff5a2965bb25a957f: Status 404 returned error can't find the container with id 7ab25d3bdbf5ce9810601e881cca664ba8bdb1c2d809d77ff5a2965bb25a957f
Sep 09 04:10:51.692196 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:51.692174    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:10:51.748028 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:51.747986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz" event={"ID":"d02f4fe9-af19-4d7e-883c-4201af1d7430","Type":"ContainerStarted","Data":"4a42bb1d5ea092a2e4c36e6798f18efc33c92f1244ee2c67ddcc5652afd81f54"}
Sep 09 04:10:51.748209 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:51.748070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz" event={"ID":"d02f4fe9-af19-4d7e-883c-4201af1d7430","Type":"ContainerStarted","Data":"7ab25d3bdbf5ce9810601e881cca664ba8bdb1c2d809d77ff5a2965bb25a957f"}
Sep 09 04:10:51.748620 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:51.748596    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:51.750328 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:51.750181    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7" event={"ID":"ee75250d-6bd0-4db9-adcf-6c7392d33932","Type":"ContainerStarted","Data":"4f408b008cedce4e6162c017f0e95e8b10a9ae0a2d49a4aab246125f834df28c"}
Sep 09 04:10:51.750450 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:51.750398    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:51.767547 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:51.767484    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz" podStartSLOduration=1.200025939 podStartE2EDuration="1.767465248s" totalImagesPullingTime="567.439309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:10:51.76569849 +0000 UTC m=+772.715389145" watchObservedRunningTime="2026-09-09 04:10:51.767465248 +0000 UTC m=+772.717155903"
Sep 09 04:10:51.785983 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:51.785935    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7" podStartSLOduration=2.792149551 podStartE2EDuration="3.785921046s" totalImagesPullingTime="993.771495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:10:51.785160469 +0000 UTC m=+772.734851122" watchObservedRunningTime="2026-09-09 04:10:51.785921046 +0000 UTC m=+772.735611716"
Sep 09 04:10:56.671910 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:56.671866    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 04:10:56.672264 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:56.671924    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp" podUID="f5ec7505-fc62-4ce0-9548-28b278a1c2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 04:10:57.763491 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:57.763461    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:10:57.764074 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:57.764054    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:10:58.798094 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:58.798063    2580 generic.go:356] "Generic (PLEG): container finished" podID="f5ec7505-fc62-4ce0-9548-28b278a1c2cc" containerID="d5f54b17f134ed5f1c1201f4c8dcabcd65cb2907dc3fc6003636d0e951c56f34" exitCode=0
Sep 09 04:10:58.798516 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:58.798140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp" event={"ID":"f5ec7505-fc62-4ce0-9548-28b278a1c2cc","Type":"ContainerDied","Data":"d5f54b17f134ed5f1c1201f4c8dcabcd65cb2907dc3fc6003636d0e951c56f34"}
Sep 09 04:10:58.798595 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:58.798551    2580 scope.go:117] "RemoveContainer" containerID="d5f54b17f134ed5f1c1201f4c8dcabcd65cb2907dc3fc6003636d0e951c56f34"
Sep 09 04:10:58.800679 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:58.800644    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:10:58.800798 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:58.800732    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:10:58.800875 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:58.800801    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:10:59.598844 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:59.598819    2580 scope.go:117] "RemoveContainer" containerID="11e51b40309f2153859b95960dcebddeb83b1cd0bac83c4721a2d77b5d14113a"
Sep 09 04:10:59.825047 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:10:59.825012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp" event={"ID":"f5ec7505-fc62-4ce0-9548-28b278a1c2cc","Type":"ContainerStarted","Data":"a76596b9285bc3e46e5fb0ccc817c30d7c4039d04adeee683ed4e889e4ac692e"}
Sep 09 04:11:00.184725 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:00.184690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:11:00.185069 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:00.185045    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 09 04:11:00.185182 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:00.185119    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp" podUID="f5ec7505-fc62-4ce0-9548-28b278a1c2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 09 04:11:01.185308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:01.185268    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 09 04:11:01.185709 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:01.185330    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp" podUID="f5ec7505-fc62-4ce0-9548-28b278a1c2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 09 04:11:02.184550 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:02.184513    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 09 04:11:02.184706 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:02.184578    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp" podUID="f5ec7505-fc62-4ce0-9548-28b278a1c2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 09 04:11:03.184914 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:03.184878    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 09 04:11:03.185310 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:03.184942    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp" podUID="f5ec7505-fc62-4ce0-9548-28b278a1c2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 09 04:11:04.184641 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:04.184612    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 09 04:11:04.184912 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:04.184877    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp" podUID="f5ec7505-fc62-4ce0-9548-28b278a1c2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 09 04:11:05.339077 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:05.339047    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:11:05.339653 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:05.339632    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:11:05.340378 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:05.340358    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp"
Sep 09 04:11:07.501648 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.501614    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"]
Sep 09 04:11:07.506417 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.506392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.508990 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.508963    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-629522-openshift-default-dockercfg-gvx7k\""
Sep 09 04:11:07.518615 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.518588    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"]
Sep 09 04:11:07.606816 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.606749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768ee68e-ff97-4f05-8ac6-df6d464e5088-istiod-ca-cert\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.606816 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.606818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-token\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.607037 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.606848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-data\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.607037 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.606910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-podinfo\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.607136 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.607036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-workload-socket\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.607136 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.607062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-credential-socket\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.607136 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.607088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74pxr\" (UniqueName: \"kubernetes.io/projected/768ee68e-ff97-4f05-8ac6-df6d464e5088-kube-api-access-74pxr\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.607136 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.607118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-envoy\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.607297 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.607186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-workload-certs\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.708496 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.708464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-workload-certs\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.708693 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.708513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768ee68e-ff97-4f05-8ac6-df6d464e5088-istiod-ca-cert\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.708693 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.708548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-token\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.708693 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.708578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-data\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.708693 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.708611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-podinfo\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.708959 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.708707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-workload-socket\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.708959 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.708733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-credential-socket\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.708959 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.708759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74pxr\" (UniqueName: \"kubernetes.io/projected/768ee68e-ff97-4f05-8ac6-df6d464e5088-kube-api-access-74pxr\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.708959 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.708807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-envoy\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.709161 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.708967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-workload-certs\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.709161 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.709030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-data\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.709668 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.709643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768ee68e-ff97-4f05-8ac6-df6d464e5088-istiod-ca-cert\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.709932 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.709863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-credential-socket\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.710092 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.710070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-workload-socket\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.711595 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.711566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-envoy\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.711755 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.711733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-podinfo\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.716639 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.716621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-token\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.717652 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.717632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74pxr\" (UniqueName: \"kubernetes.io/projected/768ee68e-ff97-4f05-8ac6-df6d464e5088-kube-api-access-74pxr\") pod \"e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") " pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:07.819005 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:07.818927    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:08.097574 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:08.097545    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"]
Sep 09 04:11:08.098312 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:11:08.098285    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768ee68e_ff97_4f05_8ac6_df6d464e5088.slice/crio-81402966091ad942e990070fefb2e171187c834e4c6631c3c255596956048a8e WatchSource:0}: Error finding container 81402966091ad942e990070fefb2e171187c834e4c6631c3c255596956048a8e: Status 404 returned error can't find the container with id 81402966091ad942e990070fefb2e171187c834e4c6631c3c255596956048a8e
Sep 09 04:11:08.100489 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:08.100454    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:11:08.100569 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:08.100519    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:11:08.100569 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:08.100547    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:11:08.886149 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:08.886116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" event={"ID":"768ee68e-ff97-4f05-8ac6-df6d464e5088","Type":"ContainerStarted","Data":"6462eb5fefef8651ed109c98f3f1b81db3758c463c11d5f9c3005f4208360905"}
Sep 09 04:11:08.886534 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:08.886163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" event={"ID":"768ee68e-ff97-4f05-8ac6-df6d464e5088","Type":"ContainerStarted","Data":"81402966091ad942e990070fefb2e171187c834e4c6631c3c255596956048a8e"}
Sep 09 04:11:08.907637 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:08.907471    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" podStartSLOduration=1.907449163 podStartE2EDuration="1.907449163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:11:08.905813207 +0000 UTC m=+789.855503864" watchObservedRunningTime="2026-09-09 04:11:08.907449163 +0000 UTC m=+789.857139819"
Sep 09 04:11:09.819302 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:09.819261    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:09.824415 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:09.824384    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:09.892959 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:09.892934    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:09.893962 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:09.893944    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:12.097516 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.097310    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"]
Sep 09 04:11:12.102087 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.102064    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:12.106423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.106398    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-629522-serving-cert\""
Sep 09 04:11:12.112094 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.112070    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"]
Sep 09 04:11:12.256300 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.256265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscrc\" (UniqueName: \"kubernetes.io/projected/d1d89aea-10df-4903-8420-375251c9b197-kube-api-access-bscrc\") pod \"maas-api-e2e-inf-b-629522-7df77977db-8pnth\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:12.256497 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.256380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-maas-api-tls\") pod \"maas-api-e2e-inf-b-629522-7df77977db-8pnth\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:12.256497 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.256413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-metrics-certs\") pod \"maas-api-e2e-inf-b-629522-7df77977db-8pnth\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:12.256497 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.256470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d89aea-10df-4903-8420-375251c9b197-openshift-service-ca\") pod \"maas-api-e2e-inf-b-629522-7df77977db-8pnth\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:12.358105 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.358021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d89aea-10df-4903-8420-375251c9b197-openshift-service-ca\") pod \"maas-api-e2e-inf-b-629522-7df77977db-8pnth\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:12.358266 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.358102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscrc\" (UniqueName: \"kubernetes.io/projected/d1d89aea-10df-4903-8420-375251c9b197-kube-api-access-bscrc\") pod \"maas-api-e2e-inf-b-629522-7df77977db-8pnth\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:12.358266 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.358185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-maas-api-tls\") pod \"maas-api-e2e-inf-b-629522-7df77977db-8pnth\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:12.358266 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.358213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-metrics-certs\") pod \"maas-api-e2e-inf-b-629522-7df77977db-8pnth\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:12.358496 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:11:12.358360    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-serving-cert: secret "maas-api-e2e-inf-b-629522-serving-cert" not found
Sep 09 04:11:12.358496 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:11:12.358426    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-maas-api-tls podName:d1d89aea-10df-4903-8420-375251c9b197 nodeName:}" failed. No retries permitted until 2026-09-09 04:11:12.858404891 +0000 UTC m=+793.808095537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-maas-api-tls") pod "maas-api-e2e-inf-b-629522-7df77977db-8pnth" (UID: "d1d89aea-10df-4903-8420-375251c9b197") : secret "maas-api-e2e-inf-b-629522-serving-cert" not found
Sep 09 04:11:12.358804 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.358756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d89aea-10df-4903-8420-375251c9b197-openshift-service-ca\") pod \"maas-api-e2e-inf-b-629522-7df77977db-8pnth\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:12.360827 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.360807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-metrics-certs\") pod \"maas-api-e2e-inf-b-629522-7df77977db-8pnth\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:12.369558 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.369534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscrc\" (UniqueName: \"kubernetes.io/projected/d1d89aea-10df-4903-8420-375251c9b197-kube-api-access-bscrc\") pod \"maas-api-e2e-inf-b-629522-7df77977db-8pnth\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:12.862736 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.862696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-maas-api-tls\") pod \"maas-api-e2e-inf-b-629522-7df77977db-8pnth\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:12.865395 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:12.865365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-maas-api-tls\") pod \"maas-api-e2e-inf-b-629522-7df77977db-8pnth\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:13.022027 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:13.021985    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:13.309808 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:13.309755    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"]
Sep 09 04:11:13.313103 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:11:13.313065    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d89aea_10df_4903_8420_375251c9b197.slice/crio-a262d4f026bfacb7228fffcf5f7f7ef6be7ace4d507379baf26c32cee57b2c3d WatchSource:0}: Error finding container a262d4f026bfacb7228fffcf5f7f7ef6be7ace4d507379baf26c32cee57b2c3d: Status 404 returned error can't find the container with id a262d4f026bfacb7228fffcf5f7f7ef6be7ace4d507379baf26c32cee57b2c3d
Sep 09 04:11:13.921747 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:13.921712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth" event={"ID":"d1d89aea-10df-4903-8420-375251c9b197","Type":"ContainerStarted","Data":"a262d4f026bfacb7228fffcf5f7f7ef6be7ace4d507379baf26c32cee57b2c3d"}
Sep 09 04:11:14.929145 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:14.929112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth" event={"ID":"d1d89aea-10df-4903-8420-375251c9b197","Type":"ContainerStarted","Data":"bd59f34fc2ae14968c49a725c1d4e3fa21bdab91cec8923bf25df73dc80448d8"}
Sep 09 04:11:14.929563 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:14.929237    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:14.949787 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:14.949708    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth" podStartSLOduration=1.980072938 podStartE2EDuration="2.949685861s" totalImagesPullingTime="969.612923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:11:14.945285448 +0000 UTC m=+795.894976104" watchObservedRunningTime="2026-09-09 04:11:14.949685861 +0000 UTC m=+795.899376517"
Sep 09 04:11:20.940901 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:20.940823    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:11:25.933977 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:25.933923    2580 patch_prober.go:28] interesting pod/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 04:11:25.934551 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:25.934008    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 04:11:28.011253 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:28.011182    2580 generic.go:356] "Generic (PLEG): container finished" podID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerID="6462eb5fefef8651ed109c98f3f1b81db3758c463c11d5f9c3005f4208360905" exitCode=0
Sep 09 04:11:28.011639 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:28.011250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" event={"ID":"768ee68e-ff97-4f05-8ac6-df6d464e5088","Type":"ContainerDied","Data":"6462eb5fefef8651ed109c98f3f1b81db3758c463c11d5f9c3005f4208360905"}
Sep 09 04:11:28.011639 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:28.011617    2580 scope.go:117] "RemoveContainer" containerID="6462eb5fefef8651ed109c98f3f1b81db3758c463c11d5f9c3005f4208360905"
Sep 09 04:11:28.012710 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:28.012679    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:11:28.012817 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:28.012752    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:11:28.012870 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:28.012818    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:11:29.023659 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:29.023620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" event={"ID":"768ee68e-ff97-4f05-8ac6-df6d464e5088","Type":"ContainerStarted","Data":"62eff0db88dd7a559efa3c9c1b457c91e05e38ab37909f0cda5ae6d80b661243"}
Sep 09 04:11:29.819094 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:29.819050    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:29.819894 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:29.819864    2580 patch_prober.go:28] interesting pod/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 09 04:11:29.820017 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:29.819920    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 09 04:11:30.820857 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:30.820812    2580 patch_prober.go:28] interesting pod/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 09 04:11:30.821362 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:30.820884    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 09 04:11:30.951834 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:30.951803    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"]
Sep 09 04:11:30.952181 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:30.952136    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz" podUID="d02f4fe9-af19-4d7e-883c-4201af1d7430" containerName="maas-api" containerID="cri-o://4a42bb1d5ea092a2e4c36e6798f18efc33c92f1244ee2c67ddcc5652afd81f54" gracePeriod=30
Sep 09 04:11:31.266408 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.266383    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:11:31.350355 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.350324    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02f4fe9-af19-4d7e-883c-4201af1d7430-openshift-service-ca\") pod \"d02f4fe9-af19-4d7e-883c-4201af1d7430\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") "
Sep 09 04:11:31.350511 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.350379    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02f4fe9-af19-4d7e-883c-4201af1d7430-maas-api-tls\") pod \"d02f4fe9-af19-4d7e-883c-4201af1d7430\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") "
Sep 09 04:11:31.350511 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.350428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02f4fe9-af19-4d7e-883c-4201af1d7430-metrics-certs\") pod \"d02f4fe9-af19-4d7e-883c-4201af1d7430\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") "
Sep 09 04:11:31.350511 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.350459    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kckx\" (UniqueName: \"kubernetes.io/projected/d02f4fe9-af19-4d7e-883c-4201af1d7430-kube-api-access-4kckx\") pod \"d02f4fe9-af19-4d7e-883c-4201af1d7430\" (UID: \"d02f4fe9-af19-4d7e-883c-4201af1d7430\") "
Sep 09 04:11:31.351336 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.351296    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02f4fe9-af19-4d7e-883c-4201af1d7430-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d02f4fe9-af19-4d7e-883c-4201af1d7430" (UID: "d02f4fe9-af19-4d7e-883c-4201af1d7430"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:11:31.353965 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.353931    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02f4fe9-af19-4d7e-883c-4201af1d7430-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d02f4fe9-af19-4d7e-883c-4201af1d7430" (UID: "d02f4fe9-af19-4d7e-883c-4201af1d7430"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:11:31.353965 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.353945    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02f4fe9-af19-4d7e-883c-4201af1d7430-kube-api-access-4kckx" (OuterVolumeSpecName: "kube-api-access-4kckx") pod "d02f4fe9-af19-4d7e-883c-4201af1d7430" (UID: "d02f4fe9-af19-4d7e-883c-4201af1d7430"). InnerVolumeSpecName "kube-api-access-4kckx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:11:31.354118 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.354096    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02f4fe9-af19-4d7e-883c-4201af1d7430-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d02f4fe9-af19-4d7e-883c-4201af1d7430" (UID: "d02f4fe9-af19-4d7e-883c-4201af1d7430"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:11:31.452081 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.452046    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02f4fe9-af19-4d7e-883c-4201af1d7430-openshift-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:11:31.452081 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.452075    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02f4fe9-af19-4d7e-883c-4201af1d7430-maas-api-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:11:31.452081 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.452084    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d02f4fe9-af19-4d7e-883c-4201af1d7430-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:11:31.452308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.452094    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kckx\" (UniqueName: \"kubernetes.io/projected/d02f4fe9-af19-4d7e-883c-4201af1d7430-kube-api-access-4kckx\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:11:31.820199 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.820150    2580 patch_prober.go:28] interesting pod/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 09 04:11:31.820408 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:31.820223    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 09 04:11:32.053628 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:32.053542    2580 generic.go:356] "Generic (PLEG): container finished" podID="d02f4fe9-af19-4d7e-883c-4201af1d7430" containerID="4a42bb1d5ea092a2e4c36e6798f18efc33c92f1244ee2c67ddcc5652afd81f54" exitCode=0
Sep 09 04:11:32.053628 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:32.053589    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"
Sep 09 04:11:32.054168 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:32.053623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz" event={"ID":"d02f4fe9-af19-4d7e-883c-4201af1d7430","Type":"ContainerDied","Data":"4a42bb1d5ea092a2e4c36e6798f18efc33c92f1244ee2c67ddcc5652afd81f54"}
Sep 09 04:11:32.054168 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:32.053676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz" event={"ID":"d02f4fe9-af19-4d7e-883c-4201af1d7430","Type":"ContainerDied","Data":"7ab25d3bdbf5ce9810601e881cca664ba8bdb1c2d809d77ff5a2965bb25a957f"}
Sep 09 04:11:32.054168 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:32.053699    2580 scope.go:117] "RemoveContainer" containerID="4a42bb1d5ea092a2e4c36e6798f18efc33c92f1244ee2c67ddcc5652afd81f54"
Sep 09 04:11:32.063720 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:32.063704    2580 scope.go:117] "RemoveContainer" containerID="4a42bb1d5ea092a2e4c36e6798f18efc33c92f1244ee2c67ddcc5652afd81f54"
Sep 09 04:11:32.064032 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:11:32.064012    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a42bb1d5ea092a2e4c36e6798f18efc33c92f1244ee2c67ddcc5652afd81f54\": container with ID starting with 4a42bb1d5ea092a2e4c36e6798f18efc33c92f1244ee2c67ddcc5652afd81f54 not found: ID does not exist" containerID="4a42bb1d5ea092a2e4c36e6798f18efc33c92f1244ee2c67ddcc5652afd81f54"
Sep 09 04:11:32.064098 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:32.064040    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a42bb1d5ea092a2e4c36e6798f18efc33c92f1244ee2c67ddcc5652afd81f54"} err="failed to get container status \"4a42bb1d5ea092a2e4c36e6798f18efc33c92f1244ee2c67ddcc5652afd81f54\": rpc error: code = NotFound desc = could not find container \"4a42bb1d5ea092a2e4c36e6798f18efc33c92f1244ee2c67ddcc5652afd81f54\": container with ID starting with 4a42bb1d5ea092a2e4c36e6798f18efc33c92f1244ee2c67ddcc5652afd81f54 not found: ID does not exist"
Sep 09 04:11:32.071478 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:32.071455    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"]
Sep 09 04:11:32.075492 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:32.075474    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45a82a13-856675b4c5-ws6dz"]
Sep 09 04:11:32.819964 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:32.819927    2580 patch_prober.go:28] interesting pod/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 09 04:11:32.820278 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:32.820246    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 09 04:11:33.596752 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:33.596723    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d02f4fe9-af19-4d7e-883c-4201af1d7430" path="/var/lib/kubelet/pods/d02f4fe9-af19-4d7e-883c-4201af1d7430/volumes"
Sep 09 04:11:33.819820 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:33.819750    2580 patch_prober.go:28] interesting pod/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 09 04:11:33.820107 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:33.819859    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 09 04:11:34.823498 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:34.823465    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:34.824057 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:34.824035    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:34.824761 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:34.824740    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:11:35.565833 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.565799    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"]
Sep 09 04:11:35.566467 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.566449    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d02f4fe9-af19-4d7e-883c-4201af1d7430" containerName="maas-api"
Sep 09 04:11:35.566525 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.566471    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d02f4fe9-af19-4d7e-883c-4201af1d7430" containerName="maas-api"
Sep 09 04:11:35.566581 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.566570    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d02f4fe9-af19-4d7e-883c-4201af1d7430" containerName="maas-api"
Sep 09 04:11:35.571394 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.571376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:35.574947 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.574875    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3748ac1b-serving-cert\""
Sep 09 04:11:35.578491 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.578468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"]
Sep 09 04:11:35.589487 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.589463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43382bbc-b0ff-42c4-8767-13ec70be0890-openshift-service-ca\") pod \"maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:35.589751 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.589729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-metrics-certs\") pod \"maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:35.589933 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.589914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-maas-api-tls\") pod \"maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:35.590091 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.590073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrnm\" (UniqueName: \"kubernetes.io/projected/43382bbc-b0ff-42c4-8767-13ec70be0890-kube-api-access-tjrnm\") pod \"maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:35.691651 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.691620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43382bbc-b0ff-42c4-8767-13ec70be0890-openshift-service-ca\") pod \"maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:35.691851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.691715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-metrics-certs\") pod \"maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:35.691851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.691806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-maas-api-tls\") pod \"maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:35.691851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.691837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrnm\" (UniqueName: \"kubernetes.io/projected/43382bbc-b0ff-42c4-8767-13ec70be0890-kube-api-access-tjrnm\") pod \"maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:35.692021 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:11:35.691966    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-serving-cert: secret "maas-api-e2e-migrate-3748ac1b-serving-cert" not found
Sep 09 04:11:35.692082 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:11:35.692044    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-maas-api-tls podName:43382bbc-b0ff-42c4-8767-13ec70be0890 nodeName:}" failed. No retries permitted until 2026-09-09 04:11:36.192023228 +0000 UTC m=+817.141713873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-maas-api-tls") pod "maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j" (UID: "43382bbc-b0ff-42c4-8767-13ec70be0890") : secret "maas-api-e2e-migrate-3748ac1b-serving-cert" not found
Sep 09 04:11:35.692395 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.692373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43382bbc-b0ff-42c4-8767-13ec70be0890-openshift-service-ca\") pod \"maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:35.694416 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.694392    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-metrics-certs\") pod \"maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:35.702383 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:35.702359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrnm\" (UniqueName: \"kubernetes.io/projected/43382bbc-b0ff-42c4-8767-13ec70be0890-kube-api-access-tjrnm\") pod \"maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:36.198154 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:36.198122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-maas-api-tls\") pod \"maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:36.200714 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:36.200693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-maas-api-tls\") pod \"maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:36.486047 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:36.485961    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:36.764182 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:36.764153    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"]
Sep 09 04:11:36.765106 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:11:36.765080    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43382bbc_b0ff_42c4_8767_13ec70be0890.slice/crio-1589e509278972f243c0685d6be3b28b924a3c0a0003002778cc1af86c44af4c WatchSource:0}: Error finding container 1589e509278972f243c0685d6be3b28b924a3c0a0003002778cc1af86c44af4c: Status 404 returned error can't find the container with id 1589e509278972f243c0685d6be3b28b924a3c0a0003002778cc1af86c44af4c
Sep 09 04:11:37.093490 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:37.093448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j" event={"ID":"43382bbc-b0ff-42c4-8767-13ec70be0890","Type":"ContainerStarted","Data":"1589e509278972f243c0685d6be3b28b924a3c0a0003002778cc1af86c44af4c"}
Sep 09 04:11:38.100455 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:38.100418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j" event={"ID":"43382bbc-b0ff-42c4-8767-13ec70be0890","Type":"ContainerStarted","Data":"6e57a1292590a42d190fcf9260308b6c4ea8f1928269c12e0b61935e6694608a"}
Sep 09 04:11:38.100893 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:38.100481    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:11:38.118712 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:38.118663    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j" podStartSLOduration=2.8147272660000002 podStartE2EDuration="3.118647851s" totalImagesPullingTime="303.920585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:11:38.117802007 +0000 UTC m=+819.067492659" watchObservedRunningTime="2026-09-09 04:11:38.118647851 +0000 UTC m=+819.068338504"
Sep 09 04:11:44.113626 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:11:44.113599    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:12:06.874757 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:06.874721    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"]
Sep 09 04:12:06.875281 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:06.874982    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j" podUID="43382bbc-b0ff-42c4-8767-13ec70be0890" containerName="maas-api" containerID="cri-o://6e57a1292590a42d190fcf9260308b6c4ea8f1928269c12e0b61935e6694608a" gracePeriod=30
Sep 09 04:12:07.137366 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.137310    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:12:07.281012 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.280980    2580 generic.go:356] "Generic (PLEG): container finished" podID="43382bbc-b0ff-42c4-8767-13ec70be0890" containerID="6e57a1292590a42d190fcf9260308b6c4ea8f1928269c12e0b61935e6694608a" exitCode=0
Sep 09 04:12:07.281172 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.281025    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"
Sep 09 04:12:07.281172 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.281053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j" event={"ID":"43382bbc-b0ff-42c4-8767-13ec70be0890","Type":"ContainerDied","Data":"6e57a1292590a42d190fcf9260308b6c4ea8f1928269c12e0b61935e6694608a"}
Sep 09 04:12:07.281172 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.281102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j" event={"ID":"43382bbc-b0ff-42c4-8767-13ec70be0890","Type":"ContainerDied","Data":"1589e509278972f243c0685d6be3b28b924a3c0a0003002778cc1af86c44af4c"}
Sep 09 04:12:07.281172 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.281124    2580 scope.go:117] "RemoveContainer" containerID="6e57a1292590a42d190fcf9260308b6c4ea8f1928269c12e0b61935e6694608a"
Sep 09 04:12:07.290613 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.290597    2580 scope.go:117] "RemoveContainer" containerID="6e57a1292590a42d190fcf9260308b6c4ea8f1928269c12e0b61935e6694608a"
Sep 09 04:12:07.290913 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:12:07.290894    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e57a1292590a42d190fcf9260308b6c4ea8f1928269c12e0b61935e6694608a\": container with ID starting with 6e57a1292590a42d190fcf9260308b6c4ea8f1928269c12e0b61935e6694608a not found: ID does not exist" containerID="6e57a1292590a42d190fcf9260308b6c4ea8f1928269c12e0b61935e6694608a"
Sep 09 04:12:07.290977 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.290924    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e57a1292590a42d190fcf9260308b6c4ea8f1928269c12e0b61935e6694608a"} err="failed to get container status \"6e57a1292590a42d190fcf9260308b6c4ea8f1928269c12e0b61935e6694608a\": rpc error: code = NotFound desc = could not find container \"6e57a1292590a42d190fcf9260308b6c4ea8f1928269c12e0b61935e6694608a\": container with ID starting with 6e57a1292590a42d190fcf9260308b6c4ea8f1928269c12e0b61935e6694608a not found: ID does not exist"
Sep 09 04:12:07.296205 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.296190    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjrnm\" (UniqueName: \"kubernetes.io/projected/43382bbc-b0ff-42c4-8767-13ec70be0890-kube-api-access-tjrnm\") pod \"43382bbc-b0ff-42c4-8767-13ec70be0890\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") "
Sep 09 04:12:07.296256 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.296238    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43382bbc-b0ff-42c4-8767-13ec70be0890-openshift-service-ca\") pod \"43382bbc-b0ff-42c4-8767-13ec70be0890\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") "
Sep 09 04:12:07.296297 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.296287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-metrics-certs\") pod \"43382bbc-b0ff-42c4-8767-13ec70be0890\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") "
Sep 09 04:12:07.296338 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.296302    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-maas-api-tls\") pod \"43382bbc-b0ff-42c4-8767-13ec70be0890\" (UID: \"43382bbc-b0ff-42c4-8767-13ec70be0890\") "
Sep 09 04:12:07.296593 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.296571    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43382bbc-b0ff-42c4-8767-13ec70be0890-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "43382bbc-b0ff-42c4-8767-13ec70be0890" (UID: "43382bbc-b0ff-42c4-8767-13ec70be0890"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:12:07.298471 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.298448    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43382bbc-b0ff-42c4-8767-13ec70be0890-kube-api-access-tjrnm" (OuterVolumeSpecName: "kube-api-access-tjrnm") pod "43382bbc-b0ff-42c4-8767-13ec70be0890" (UID: "43382bbc-b0ff-42c4-8767-13ec70be0890"). InnerVolumeSpecName "kube-api-access-tjrnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:12:07.298471 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.298451    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "43382bbc-b0ff-42c4-8767-13ec70be0890" (UID: "43382bbc-b0ff-42c4-8767-13ec70be0890"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:12:07.298586 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.298552    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "43382bbc-b0ff-42c4-8767-13ec70be0890" (UID: "43382bbc-b0ff-42c4-8767-13ec70be0890"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:12:07.397511 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.397413    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:12:07.397511 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.397443    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43382bbc-b0ff-42c4-8767-13ec70be0890-maas-api-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:12:07.397511 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.397456    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjrnm\" (UniqueName: \"kubernetes.io/projected/43382bbc-b0ff-42c4-8767-13ec70be0890-kube-api-access-tjrnm\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:12:07.397511 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.397469    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43382bbc-b0ff-42c4-8767-13ec70be0890-openshift-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:12:07.605878 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.605846    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"]
Sep 09 04:12:07.609472 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:07.609450    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3748ac1b-865b4d6988-d649j"]
Sep 09 04:12:09.596212 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:09.596183    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43382bbc-b0ff-42c4-8767-13ec70be0890" path="/var/lib/kubelet/pods/43382bbc-b0ff-42c4-8767-13ec70be0890/volumes"
Sep 09 04:12:11.571712 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.571677    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"]
Sep 09 04:12:11.572344 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.572324    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43382bbc-b0ff-42c4-8767-13ec70be0890" containerName="maas-api"
Sep 09 04:12:11.572446 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.572346    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="43382bbc-b0ff-42c4-8767-13ec70be0890" containerName="maas-api"
Sep 09 04:12:11.572446 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.572432    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="43382bbc-b0ff-42c4-8767-13ec70be0890" containerName="maas-api"
Sep 09 04:12:11.576700 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.576682    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:11.578948 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.578926    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3dbd54be-serving-cert\""
Sep 09 04:12:11.587210 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.587135    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"]
Sep 09 04:12:11.630465 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.630435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3841ec4d-da33-49f0-9b13-54d3ba8bb400-maas-api-tls\") pod \"maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:11.630465 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.630470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3841ec4d-da33-49f0-9b13-54d3ba8bb400-openshift-service-ca\") pod \"maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:11.630700 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.630487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt7sv\" (UniqueName: \"kubernetes.io/projected/3841ec4d-da33-49f0-9b13-54d3ba8bb400-kube-api-access-jt7sv\") pod \"maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:11.630700 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.630601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3841ec4d-da33-49f0-9b13-54d3ba8bb400-metrics-certs\") pod \"maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:11.731153 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.731113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3841ec4d-da33-49f0-9b13-54d3ba8bb400-maas-api-tls\") pod \"maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:11.731153 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.731153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3841ec4d-da33-49f0-9b13-54d3ba8bb400-openshift-service-ca\") pod \"maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:11.731390 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.731178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt7sv\" (UniqueName: \"kubernetes.io/projected/3841ec4d-da33-49f0-9b13-54d3ba8bb400-kube-api-access-jt7sv\") pod \"maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:11.731390 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.731255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3841ec4d-da33-49f0-9b13-54d3ba8bb400-metrics-certs\") pod \"maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:11.731930 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.731843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3841ec4d-da33-49f0-9b13-54d3ba8bb400-openshift-service-ca\") pod \"maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:11.733808 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.733752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3841ec4d-da33-49f0-9b13-54d3ba8bb400-maas-api-tls\") pod \"maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:11.733916 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.733830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3841ec4d-da33-49f0-9b13-54d3ba8bb400-metrics-certs\") pod \"maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:11.739947 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.739928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt7sv\" (UniqueName: \"kubernetes.io/projected/3841ec4d-da33-49f0-9b13-54d3ba8bb400-kube-api-access-jt7sv\") pod \"maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:11.891204 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:11.891173    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:12.169233 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:12.169205    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"]
Sep 09 04:12:12.170906 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:12:12.170879    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3841ec4d_da33_49f0_9b13_54d3ba8bb400.slice/crio-d086649583228ef1365cf0037ceda727dd754b178673a974f35b95b67e695b6c WatchSource:0}: Error finding container d086649583228ef1365cf0037ceda727dd754b178673a974f35b95b67e695b6c: Status 404 returned error can't find the container with id d086649583228ef1365cf0037ceda727dd754b178673a974f35b95b67e695b6c
Sep 09 04:12:12.312854 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:12.312759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm" event={"ID":"3841ec4d-da33-49f0-9b13-54d3ba8bb400","Type":"ContainerStarted","Data":"d086649583228ef1365cf0037ceda727dd754b178673a974f35b95b67e695b6c"}
Sep 09 04:12:13.320429 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:13.320393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm" event={"ID":"3841ec4d-da33-49f0-9b13-54d3ba8bb400","Type":"ContainerStarted","Data":"b39ee53472d5d65a3505724345260d12808d3b00dd054ff581a15846010ce339"}
Sep 09 04:12:13.320895 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:13.320450    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:13.339481 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:13.339427    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm" podStartSLOduration=1.860710674 podStartE2EDuration="2.3394122s" totalImagesPullingTime="478.701526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:12:13.337463833 +0000 UTC m=+854.287154489" watchObservedRunningTime="2026-09-09 04:12:13.3394122 +0000 UTC m=+854.289102890"
Sep 09 04:12:19.332104 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:19.332070    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:42.127618 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.127583    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"]
Sep 09 04:12:42.128195 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.127814    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm" podUID="3841ec4d-da33-49f0-9b13-54d3ba8bb400" containerName="maas-api" containerID="cri-o://b39ee53472d5d65a3505724345260d12808d3b00dd054ff581a15846010ce339" gracePeriod=30
Sep 09 04:12:42.380477 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.380421    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:42.496863 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.496828    2580 generic.go:356] "Generic (PLEG): container finished" podID="3841ec4d-da33-49f0-9b13-54d3ba8bb400" containerID="b39ee53472d5d65a3505724345260d12808d3b00dd054ff581a15846010ce339" exitCode=0
Sep 09 04:12:42.497020 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.496871    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"
Sep 09 04:12:42.497020 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.496880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm" event={"ID":"3841ec4d-da33-49f0-9b13-54d3ba8bb400","Type":"ContainerDied","Data":"b39ee53472d5d65a3505724345260d12808d3b00dd054ff581a15846010ce339"}
Sep 09 04:12:42.497020 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.496929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm" event={"ID":"3841ec4d-da33-49f0-9b13-54d3ba8bb400","Type":"ContainerDied","Data":"d086649583228ef1365cf0037ceda727dd754b178673a974f35b95b67e695b6c"}
Sep 09 04:12:42.497020 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.496955    2580 scope.go:117] "RemoveContainer" containerID="b39ee53472d5d65a3505724345260d12808d3b00dd054ff581a15846010ce339"
Sep 09 04:12:42.507895 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.507868    2580 scope.go:117] "RemoveContainer" containerID="b39ee53472d5d65a3505724345260d12808d3b00dd054ff581a15846010ce339"
Sep 09 04:12:42.508209 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:12:42.508187    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b39ee53472d5d65a3505724345260d12808d3b00dd054ff581a15846010ce339\": container with ID starting with b39ee53472d5d65a3505724345260d12808d3b00dd054ff581a15846010ce339 not found: ID does not exist" containerID="b39ee53472d5d65a3505724345260d12808d3b00dd054ff581a15846010ce339"
Sep 09 04:12:42.508291 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.508225    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39ee53472d5d65a3505724345260d12808d3b00dd054ff581a15846010ce339"} err="failed to get container status \"b39ee53472d5d65a3505724345260d12808d3b00dd054ff581a15846010ce339\": rpc error: code = NotFound desc = could not find container \"b39ee53472d5d65a3505724345260d12808d3b00dd054ff581a15846010ce339\": container with ID starting with b39ee53472d5d65a3505724345260d12808d3b00dd054ff581a15846010ce339 not found: ID does not exist"
Sep 09 04:12:42.516529 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.516508    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3841ec4d-da33-49f0-9b13-54d3ba8bb400-openshift-service-ca\") pod \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") "
Sep 09 04:12:42.516672 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.516608    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jt7sv\" (UniqueName: \"kubernetes.io/projected/3841ec4d-da33-49f0-9b13-54d3ba8bb400-kube-api-access-jt7sv\") pod \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") "
Sep 09 04:12:42.516672 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.516643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3841ec4d-da33-49f0-9b13-54d3ba8bb400-metrics-certs\") pod \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") "
Sep 09 04:12:42.516819 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.516705    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3841ec4d-da33-49f0-9b13-54d3ba8bb400-maas-api-tls\") pod \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\" (UID: \"3841ec4d-da33-49f0-9b13-54d3ba8bb400\") "
Sep 09 04:12:42.516928 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.516900    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3841ec4d-da33-49f0-9b13-54d3ba8bb400-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3841ec4d-da33-49f0-9b13-54d3ba8bb400" (UID: "3841ec4d-da33-49f0-9b13-54d3ba8bb400"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:12:42.517045 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.517025    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3841ec4d-da33-49f0-9b13-54d3ba8bb400-openshift-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:12:42.519085 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.519046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3841ec4d-da33-49f0-9b13-54d3ba8bb400-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3841ec4d-da33-49f0-9b13-54d3ba8bb400" (UID: "3841ec4d-da33-49f0-9b13-54d3ba8bb400"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:12:42.519085 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.519069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3841ec4d-da33-49f0-9b13-54d3ba8bb400-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3841ec4d-da33-49f0-9b13-54d3ba8bb400" (UID: "3841ec4d-da33-49f0-9b13-54d3ba8bb400"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:12:42.519214 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.519159    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3841ec4d-da33-49f0-9b13-54d3ba8bb400-kube-api-access-jt7sv" (OuterVolumeSpecName: "kube-api-access-jt7sv") pod "3841ec4d-da33-49f0-9b13-54d3ba8bb400" (UID: "3841ec4d-da33-49f0-9b13-54d3ba8bb400"). InnerVolumeSpecName "kube-api-access-jt7sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:12:42.617690 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.617652    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt7sv\" (UniqueName: \"kubernetes.io/projected/3841ec4d-da33-49f0-9b13-54d3ba8bb400-kube-api-access-jt7sv\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:12:42.617690 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.617682    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3841ec4d-da33-49f0-9b13-54d3ba8bb400-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:12:42.617690 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.617691    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3841ec4d-da33-49f0-9b13-54d3ba8bb400-maas-api-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:12:42.822475 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.822447    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"]
Sep 09 04:12:42.825039 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:42.825020    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3dbd54be-7484d87b5c-j4zfm"]
Sep 09 04:12:43.596096 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:43.596058    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3841ec4d-da33-49f0-9b13-54d3ba8bb400" path="/var/lib/kubelet/pods/3841ec4d-da33-49f0-9b13-54d3ba8bb400/volumes"
Sep 09 04:12:44.286857 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.286827    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"]
Sep 09 04:12:44.287251 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.287238    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3841ec4d-da33-49f0-9b13-54d3ba8bb400" containerName="maas-api"
Sep 09 04:12:44.287296 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.287254    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3841ec4d-da33-49f0-9b13-54d3ba8bb400" containerName="maas-api"
Sep 09 04:12:44.287338 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.287317    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3841ec4d-da33-49f0-9b13-54d3ba8bb400" containerName="maas-api"
Sep 09 04:12:44.291400 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.291376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.293961 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.293943    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1be20a5f-openshift-default-dockercfg-djmb9\""
Sep 09 04:12:44.302294 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.302047    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"]
Sep 09 04:12:44.435344 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.435301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn5f2\" (UniqueName: \"kubernetes.io/projected/8f9c4f78-c21a-4cf2-abfa-46663415ef81-kube-api-access-bn5f2\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.435529 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.435357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-credential-socket\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.435529 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.435391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-podinfo\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.435529 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.435489    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-workload-socket\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.435529 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.435512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-data\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.435733 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.435551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-workload-certs\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.435733 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.435578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-envoy\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.435733 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.435652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istiod-ca-cert\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.435904 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.435795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-token\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.537192 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.537101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-workload-socket\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.537192 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.537139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-data\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.537192 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.537177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-workload-certs\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.537486 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.537202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-envoy\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.537486 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.537233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istiod-ca-cert\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.537486 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.537361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-token\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.537486 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.537406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn5f2\" (UniqueName: \"kubernetes.io/projected/8f9c4f78-c21a-4cf2-abfa-46663415ef81-kube-api-access-bn5f2\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.537486 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.537451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-credential-socket\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.537486 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.537476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-podinfo\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.537828 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.537550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-workload-socket\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.537828 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.537619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-workload-certs\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.537828 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.537801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-data\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.538010 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.537855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-credential-socket\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.538109 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.538089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istiod-ca-cert\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.540060 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.540021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-envoy\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.540162 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.540087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-podinfo\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.545806 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.545759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-token\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.546173 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.546154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn5f2\" (UniqueName: \"kubernetes.io/projected/8f9c4f78-c21a-4cf2-abfa-46663415ef81-kube-api-access-bn5f2\") pod \"e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") " pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.627273 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.627235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:44.907720 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.907693    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"]
Sep 09 04:12:44.910332 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:12:44.910297    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9c4f78_c21a_4cf2_abfa_46663415ef81.slice/crio-bc833ef21237c9e9f01ea32c0e07e27ce88871bf725e459e373642a94731275c WatchSource:0}: Error finding container bc833ef21237c9e9f01ea32c0e07e27ce88871bf725e459e373642a94731275c: Status 404 returned error can't find the container with id bc833ef21237c9e9f01ea32c0e07e27ce88871bf725e459e373642a94731275c
Sep 09 04:12:44.912682 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.912645    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:12:44.912800 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.912729    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:12:44.912875 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:44.912807    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:12:45.524754 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:45.524719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr" event={"ID":"8f9c4f78-c21a-4cf2-abfa-46663415ef81","Type":"ContainerStarted","Data":"8a4314a046d69f81b6582fc3c5ba61bf9e91686b3a5e4b7b237e42a8b3a019cd"}
Sep 09 04:12:45.524948 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:45.524821    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr" event={"ID":"8f9c4f78-c21a-4cf2-abfa-46663415ef81","Type":"ContainerStarted","Data":"bc833ef21237c9e9f01ea32c0e07e27ce88871bf725e459e373642a94731275c"}
Sep 09 04:12:45.547344 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:45.547291    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr" podStartSLOduration=1.547276909 podStartE2EDuration="1.547276909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:12:45.543852529 +0000 UTC m=+886.493543183" watchObservedRunningTime="2026-09-09 04:12:45.547276909 +0000 UTC m=+886.496967563"
Sep 09 04:12:45.628147 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:45.628119    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:45.634082 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:45.634055    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:46.532350 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:46.532317    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:46.533280 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:46.533260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:12:59.562201 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:59.562080    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:12:59.563104 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:12:59.563075    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:13:07.677289 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:07.677256    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerID="8a4314a046d69f81b6582fc3c5ba61bf9e91686b3a5e4b7b237e42a8b3a019cd" exitCode=0
Sep 09 04:13:07.677738 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:07.677333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr" event={"ID":"8f9c4f78-c21a-4cf2-abfa-46663415ef81","Type":"ContainerDied","Data":"8a4314a046d69f81b6582fc3c5ba61bf9e91686b3a5e4b7b237e42a8b3a019cd"}
Sep 09 04:13:07.677806 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:07.677735    2580 scope.go:117] "RemoveContainer" containerID="8a4314a046d69f81b6582fc3c5ba61bf9e91686b3a5e4b7b237e42a8b3a019cd"
Sep 09 04:13:07.678923 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:07.678885    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:13:07.679019 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:07.678966    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:13:07.679019 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:07.679002    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:13:08.685635 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:08.685598    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr" event={"ID":"8f9c4f78-c21a-4cf2-abfa-46663415ef81","Type":"ContainerStarted","Data":"006055e27febb348d29ad364873902cf4a6f98062570d9f14efb73f8dd96ed1b"}
Sep 09 04:13:09.628662 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:09.628228    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:13:09.628662 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:09.628354    2580 patch_prober.go:28] interesting pod/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 09 04:13:09.628662 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:09.628411    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr" podUID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 09 04:13:10.628229 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:10.628189    2580 patch_prober.go:28] interesting pod/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 09 04:13:10.628909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:10.628867    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr" podUID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 09 04:13:11.628306 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:11.627864    2580 patch_prober.go:28] interesting pod/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 09 04:13:11.628306 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:11.627930    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr" podUID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 09 04:13:12.627900 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:12.627858    2580 patch_prober.go:28] interesting pod/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 09 04:13:12.628108 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:12.627926    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr" podUID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 09 04:13:13.631191 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:13.631150    2580 patch_prober.go:28] interesting pod/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 09 04:13:13.631760 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:13.631722    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr" podUID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 09 04:13:14.632399 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:14.632369    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:13:14.632885 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:14.632800    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:13:14.633510 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:14.633493    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:13:19.230812 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:19.230748    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"]
Sep 09 04:13:19.231279 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:19.231070    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr" podUID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerName="istio-proxy" containerID="cri-o://006055e27febb348d29ad364873902cf4a6f98062570d9f14efb73f8dd96ed1b" gracePeriod=30
Sep 09 04:13:22.109267 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.109235    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"]
Sep 09 04:13:22.110325 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.110293    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7" podUID="ee75250d-6bd0-4db9-adcf-6c7392d33932" containerName="maas-api" containerID="cri-o://4f408b008cedce4e6162c017f0e95e8b10a9ae0a2d49a4aab246125f834df28c" gracePeriod=30
Sep 09 04:13:22.385648 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.385621    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:13:22.529479 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.529446    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-maas-api-tls\") pod \"ee75250d-6bd0-4db9-adcf-6c7392d33932\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") "
Sep 09 04:13:22.529670 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.529493    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee75250d-6bd0-4db9-adcf-6c7392d33932-openshift-service-ca\") pod \"ee75250d-6bd0-4db9-adcf-6c7392d33932\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") "
Sep 09 04:13:22.529670 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.529569    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmq2m\" (UniqueName: \"kubernetes.io/projected/ee75250d-6bd0-4db9-adcf-6c7392d33932-kube-api-access-nmq2m\") pod \"ee75250d-6bd0-4db9-adcf-6c7392d33932\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") "
Sep 09 04:13:22.529670 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.529634    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-metrics-certs\") pod \"ee75250d-6bd0-4db9-adcf-6c7392d33932\" (UID: \"ee75250d-6bd0-4db9-adcf-6c7392d33932\") "
Sep 09 04:13:22.529964 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.529935    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee75250d-6bd0-4db9-adcf-6c7392d33932-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee75250d-6bd0-4db9-adcf-6c7392d33932" (UID: "ee75250d-6bd0-4db9-adcf-6c7392d33932"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:13:22.530217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.530180    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee75250d-6bd0-4db9-adcf-6c7392d33932-openshift-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:13:22.532520 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.532490    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ee75250d-6bd0-4db9-adcf-6c7392d33932" (UID: "ee75250d-6bd0-4db9-adcf-6c7392d33932"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:13:22.533211 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.533189    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee75250d-6bd0-4db9-adcf-6c7392d33932" (UID: "ee75250d-6bd0-4db9-adcf-6c7392d33932"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:13:22.533315 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.533271    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee75250d-6bd0-4db9-adcf-6c7392d33932-kube-api-access-nmq2m" (OuterVolumeSpecName: "kube-api-access-nmq2m") pod "ee75250d-6bd0-4db9-adcf-6c7392d33932" (UID: "ee75250d-6bd0-4db9-adcf-6c7392d33932"). InnerVolumeSpecName "kube-api-access-nmq2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:13:22.630796 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.630679    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmq2m\" (UniqueName: \"kubernetes.io/projected/ee75250d-6bd0-4db9-adcf-6c7392d33932-kube-api-access-nmq2m\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:13:22.630796 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.630709    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:13:22.630796 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.630719    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee75250d-6bd0-4db9-adcf-6c7392d33932-maas-api-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:13:22.774022 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.773987    2580 generic.go:356] "Generic (PLEG): container finished" podID="ee75250d-6bd0-4db9-adcf-6c7392d33932" containerID="4f408b008cedce4e6162c017f0e95e8b10a9ae0a2d49a4aab246125f834df28c" exitCode=0
Sep 09 04:13:22.774197 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.774028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7" event={"ID":"ee75250d-6bd0-4db9-adcf-6c7392d33932","Type":"ContainerDied","Data":"4f408b008cedce4e6162c017f0e95e8b10a9ae0a2d49a4aab246125f834df28c"}
Sep 09 04:13:22.774197 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.774071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7" event={"ID":"ee75250d-6bd0-4db9-adcf-6c7392d33932","Type":"ContainerDied","Data":"1b1a62e525a13ae39254f96e128a2c251c9c9ccaed18468866f9062cf5a1de9d"}
Sep 09 04:13:22.774197 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.774091    2580 scope.go:117] "RemoveContainer" containerID="4f408b008cedce4e6162c017f0e95e8b10a9ae0a2d49a4aab246125f834df28c"
Sep 09 04:13:22.774197 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.774034    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"
Sep 09 04:13:22.784211 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.784188    2580 scope.go:117] "RemoveContainer" containerID="4f408b008cedce4e6162c017f0e95e8b10a9ae0a2d49a4aab246125f834df28c"
Sep 09 04:13:22.784510 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:13:22.784484    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f408b008cedce4e6162c017f0e95e8b10a9ae0a2d49a4aab246125f834df28c\": container with ID starting with 4f408b008cedce4e6162c017f0e95e8b10a9ae0a2d49a4aab246125f834df28c not found: ID does not exist" containerID="4f408b008cedce4e6162c017f0e95e8b10a9ae0a2d49a4aab246125f834df28c"
Sep 09 04:13:22.784602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.784516    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f408b008cedce4e6162c017f0e95e8b10a9ae0a2d49a4aab246125f834df28c"} err="failed to get container status \"4f408b008cedce4e6162c017f0e95e8b10a9ae0a2d49a4aab246125f834df28c\": rpc error: code = NotFound desc = could not find container \"4f408b008cedce4e6162c017f0e95e8b10a9ae0a2d49a4aab246125f834df28c\": container with ID starting with 4f408b008cedce4e6162c017f0e95e8b10a9ae0a2d49a4aab246125f834df28c not found: ID does not exist"
Sep 09 04:13:22.796068 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.796048    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"]
Sep 09 04:13:22.803186 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:22.803166    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4ad5fa-6d6c99f9cb-4fqp7"]
Sep 09 04:13:23.598152 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:23.598116    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee75250d-6bd0-4db9-adcf-6c7392d33932" path="/var/lib/kubelet/pods/ee75250d-6bd0-4db9-adcf-6c7392d33932/volumes"
Sep 09 04:13:24.485538 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.485516    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:13:24.648058 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648024    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-credential-socket\") pod \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") "
Sep 09 04:13:24.648514 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648097    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn5f2\" (UniqueName: \"kubernetes.io/projected/8f9c4f78-c21a-4cf2-abfa-46663415ef81-kube-api-access-bn5f2\") pod \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") "
Sep 09 04:13:24.648514 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648122    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-podinfo\") pod \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") "
Sep 09 04:13:24.648514 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648147    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-envoy\") pod \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") "
Sep 09 04:13:24.648514 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648174    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istiod-ca-cert\") pod \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") "
Sep 09 04:13:24.648514 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-data\") pod \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") "
Sep 09 04:13:24.648514 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648255    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-token\") pod \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") "
Sep 09 04:13:24.648514 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648297    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f9c4f78-c21a-4cf2-abfa-46663415ef81" (UID: "8f9c4f78-c21a-4cf2-abfa-46663415ef81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:13:24.648514 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648316    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-workload-certs\") pod \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") "
Sep 09 04:13:24.648514 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-workload-socket\") pod \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\" (UID: \"8f9c4f78-c21a-4cf2-abfa-46663415ef81\") "
Sep 09 04:13:24.648514 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f9c4f78-c21a-4cf2-abfa-46663415ef81" (UID: "8f9c4f78-c21a-4cf2-abfa-46663415ef81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:13:24.649336 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648639    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f9c4f78-c21a-4cf2-abfa-46663415ef81" (UID: "8f9c4f78-c21a-4cf2-abfa-46663415ef81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:13:24.649336 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648680    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-data\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:13:24.649336 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648701    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-credential-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:13:24.649336 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f9c4f78-c21a-4cf2-abfa-46663415ef81" (UID: "8f9c4f78-c21a-4cf2-abfa-46663415ef81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:13:24.649336 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.648946    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f9c4f78-c21a-4cf2-abfa-46663415ef81" (UID: "8f9c4f78-c21a-4cf2-abfa-46663415ef81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:13:24.650674 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.650639    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f9c4f78-c21a-4cf2-abfa-46663415ef81" (UID: "8f9c4f78-c21a-4cf2-abfa-46663415ef81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:13:24.650804 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.650644    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f9c4f78-c21a-4cf2-abfa-46663415ef81" (UID: "8f9c4f78-c21a-4cf2-abfa-46663415ef81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:13:24.651006 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.650985    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f9c4f78-c21a-4cf2-abfa-46663415ef81" (UID: "8f9c4f78-c21a-4cf2-abfa-46663415ef81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:13:24.651104 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.651082    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9c4f78-c21a-4cf2-abfa-46663415ef81-kube-api-access-bn5f2" (OuterVolumeSpecName: "kube-api-access-bn5f2") pod "8f9c4f78-c21a-4cf2-abfa-46663415ef81" (UID: "8f9c4f78-c21a-4cf2-abfa-46663415ef81"). InnerVolumeSpecName "kube-api-access-bn5f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:13:24.749392 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.749358    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-workload-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:13:24.749392 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.749386    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn5f2\" (UniqueName: \"kubernetes.io/projected/8f9c4f78-c21a-4cf2-abfa-46663415ef81-kube-api-access-bn5f2\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:13:24.749584 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.749399    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-podinfo\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:13:24.749584 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.749413    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-envoy\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:13:24.749584 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.749422    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istiod-ca-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:13:24.749584 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.749429    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9c4f78-c21a-4cf2-abfa-46663415ef81-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:13:24.749584 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.749436    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9c4f78-c21a-4cf2-abfa-46663415ef81-workload-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:13:24.796390 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.796359    2580 generic.go:356] "Generic (PLEG): container finished" podID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerID="006055e27febb348d29ad364873902cf4a6f98062570d9f14efb73f8dd96ed1b" exitCode=0
Sep 09 04:13:24.796568 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.796409    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"
Sep 09 04:13:24.796568 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.796450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr" event={"ID":"8f9c4f78-c21a-4cf2-abfa-46663415ef81","Type":"ContainerDied","Data":"006055e27febb348d29ad364873902cf4a6f98062570d9f14efb73f8dd96ed1b"}
Sep 09 04:13:24.796568 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.796512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr" event={"ID":"8f9c4f78-c21a-4cf2-abfa-46663415ef81","Type":"ContainerDied","Data":"bc833ef21237c9e9f01ea32c0e07e27ce88871bf725e459e373642a94731275c"}
Sep 09 04:13:24.796568 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.796543    2580 scope.go:117] "RemoveContainer" containerID="006055e27febb348d29ad364873902cf4a6f98062570d9f14efb73f8dd96ed1b"
Sep 09 04:13:24.807527 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.807508    2580 scope.go:117] "RemoveContainer" containerID="8a4314a046d69f81b6582fc3c5ba61bf9e91686b3a5e4b7b237e42a8b3a019cd"
Sep 09 04:13:24.816458 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.816438    2580 scope.go:117] "RemoveContainer" containerID="006055e27febb348d29ad364873902cf4a6f98062570d9f14efb73f8dd96ed1b"
Sep 09 04:13:24.816725 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:13:24.816707    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006055e27febb348d29ad364873902cf4a6f98062570d9f14efb73f8dd96ed1b\": container with ID starting with 006055e27febb348d29ad364873902cf4a6f98062570d9f14efb73f8dd96ed1b not found: ID does not exist" containerID="006055e27febb348d29ad364873902cf4a6f98062570d9f14efb73f8dd96ed1b"
Sep 09 04:13:24.816786 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.816734    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006055e27febb348d29ad364873902cf4a6f98062570d9f14efb73f8dd96ed1b"} err="failed to get container status \"006055e27febb348d29ad364873902cf4a6f98062570d9f14efb73f8dd96ed1b\": rpc error: code = NotFound desc = could not find container \"006055e27febb348d29ad364873902cf4a6f98062570d9f14efb73f8dd96ed1b\": container with ID starting with 006055e27febb348d29ad364873902cf4a6f98062570d9f14efb73f8dd96ed1b not found: ID does not exist"
Sep 09 04:13:24.816786 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.816751    2580 scope.go:117] "RemoveContainer" containerID="8a4314a046d69f81b6582fc3c5ba61bf9e91686b3a5e4b7b237e42a8b3a019cd"
Sep 09 04:13:24.817029 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:13:24.817011    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4314a046d69f81b6582fc3c5ba61bf9e91686b3a5e4b7b237e42a8b3a019cd\": container with ID starting with 8a4314a046d69f81b6582fc3c5ba61bf9e91686b3a5e4b7b237e42a8b3a019cd not found: ID does not exist" containerID="8a4314a046d69f81b6582fc3c5ba61bf9e91686b3a5e4b7b237e42a8b3a019cd"
Sep 09 04:13:24.817067 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.817033    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4314a046d69f81b6582fc3c5ba61bf9e91686b3a5e4b7b237e42a8b3a019cd"} err="failed to get container status \"8a4314a046d69f81b6582fc3c5ba61bf9e91686b3a5e4b7b237e42a8b3a019cd\": rpc error: code = NotFound desc = could not find container \"8a4314a046d69f81b6582fc3c5ba61bf9e91686b3a5e4b7b237e42a8b3a019cd\": container with ID starting with 8a4314a046d69f81b6582fc3c5ba61bf9e91686b3a5e4b7b237e42a8b3a019cd not found: ID does not exist"
Sep 09 04:13:24.822295 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.822274    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"]
Sep 09 04:13:24.826265 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:24.826244    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1be20a5f-openshift-default-7b457d69cb-w2jkr"]
Sep 09 04:13:25.596378 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:13:25.596348    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" path="/var/lib/kubelet/pods/8f9c4f78-c21a-4cf2-abfa-46663415ef81/volumes"
Sep 09 04:14:06.233294 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.233260    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"]
Sep 09 04:14:06.234314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.233917    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee75250d-6bd0-4db9-adcf-6c7392d33932" containerName="maas-api"
Sep 09 04:14:06.234314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.233936    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee75250d-6bd0-4db9-adcf-6c7392d33932" containerName="maas-api"
Sep 09 04:14:06.234314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.233952    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerName="istio-proxy"
Sep 09 04:14:06.234314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.233959    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerName="istio-proxy"
Sep 09 04:14:06.234314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.233971    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerName="istio-proxy"
Sep 09 04:14:06.234314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.233979    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerName="istio-proxy"
Sep 09 04:14:06.234314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.234091    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee75250d-6bd0-4db9-adcf-6c7392d33932" containerName="maas-api"
Sep 09 04:14:06.234314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.234102    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerName="istio-proxy"
Sep 09 04:14:06.238713 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.238692    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.241157 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.241134    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-b80654-openshift-default-dockercfg-5rwn8\""
Sep 09 04:14:06.246246 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.246216    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"]
Sep 09 04:14:06.324535 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.324491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-data\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.324737 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.324539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0be37e-de1f-4c5f-9050-539dd642c804-istiod-ca-cert\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.324737 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.324595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2npvs\" (UniqueName: \"kubernetes.io/projected/3a0be37e-de1f-4c5f-9050-539dd642c804-kube-api-access-2npvs\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.324737 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.324716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-token\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.324987 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.324756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-workload-certs\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.325072 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.325044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-envoy\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.325134 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.325096    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-credential-socket\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.325192 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.325154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-podinfo\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.325192 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.325181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-workload-socket\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.425930 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.425886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-data\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.426110 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.425943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0be37e-de1f-4c5f-9050-539dd642c804-istiod-ca-cert\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.426110 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.425976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2npvs\" (UniqueName: \"kubernetes.io/projected/3a0be37e-de1f-4c5f-9050-539dd642c804-kube-api-access-2npvs\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.426226 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.426119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-token\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.426226 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.426167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-workload-certs\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.426226 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.426217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-envoy\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.426380 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.426259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-credential-socket\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.426380 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.426295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-data\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.426380 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.426301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-podinfo\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.426380 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.426358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-workload-socket\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.426623 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.426599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-credential-socket\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.426691 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.426646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-workload-socket\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.426751 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.426688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0be37e-de1f-4c5f-9050-539dd642c804-istiod-ca-cert\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.426837 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.426803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-workload-certs\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.428726 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.428698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-envoy\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.428914 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.428896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-podinfo\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.434790 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.434733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-token\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.434881 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.434807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2npvs\" (UniqueName: \"kubernetes.io/projected/3a0be37e-de1f-4c5f-9050-539dd642c804-kube-api-access-2npvs\") pod \"e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") " pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.552110 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.552015    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:06.831793 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.831745    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"]
Sep 09 04:14:06.832658 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:14:06.832634    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0be37e_de1f_4c5f_9050_539dd642c804.slice/crio-b77e84a7dcf1c297dd37aa8acc5b4d42aa2899f16365113a17ba72d771a33ff2 WatchSource:0}: Error finding container b77e84a7dcf1c297dd37aa8acc5b4d42aa2899f16365113a17ba72d771a33ff2: Status 404 returned error can't find the container with id b77e84a7dcf1c297dd37aa8acc5b4d42aa2899f16365113a17ba72d771a33ff2
Sep 09 04:14:06.835003 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.834970    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:14:06.835108 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.835033    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:14:06.835108 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:06.835061    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:14:07.047695 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:07.047600    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c" event={"ID":"3a0be37e-de1f-4c5f-9050-539dd642c804","Type":"ContainerStarted","Data":"7a2d882229d923b0825a09dece996448dd8560d0a31ec7a8b2a6e91378a8659f"}
Sep 09 04:14:07.047695 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:07.047662    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c" event={"ID":"3a0be37e-de1f-4c5f-9050-539dd642c804","Type":"ContainerStarted","Data":"b77e84a7dcf1c297dd37aa8acc5b4d42aa2899f16365113a17ba72d771a33ff2"}
Sep 09 04:14:07.071607 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:07.071557    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c" podStartSLOduration=1.07154105 podStartE2EDuration="1.07154105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:14:07.0705495 +0000 UTC m=+968.020240156" watchObservedRunningTime="2026-09-09 04:14:07.07154105 +0000 UTC m=+968.021231704"
Sep 09 04:14:07.552872 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:07.552835    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:07.558180 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:07.558156    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:08.054090 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:08.054052    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:08.054941 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:08.054916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:30.189011 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:30.188981    2580 generic.go:356] "Generic (PLEG): container finished" podID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerID="7a2d882229d923b0825a09dece996448dd8560d0a31ec7a8b2a6e91378a8659f" exitCode=0
Sep 09 04:14:30.189356 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:30.189051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c" event={"ID":"3a0be37e-de1f-4c5f-9050-539dd642c804","Type":"ContainerDied","Data":"7a2d882229d923b0825a09dece996448dd8560d0a31ec7a8b2a6e91378a8659f"}
Sep 09 04:14:30.189478 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:30.189462    2580 scope.go:117] "RemoveContainer" containerID="7a2d882229d923b0825a09dece996448dd8560d0a31ec7a8b2a6e91378a8659f"
Sep 09 04:14:30.190541 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:30.190512    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:14:30.190628 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:30.190582    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:14:30.190628 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:30.190614    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:14:31.196783 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:31.196740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c" event={"ID":"3a0be37e-de1f-4c5f-9050-539dd642c804","Type":"ContainerStarted","Data":"6ac98526f1bcd0c3f4e4f3f76c62ee4ae00df3e4435aa530eaa19052df3539cc"}
Sep 09 04:14:31.553052 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:31.552967    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:31.553301 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:31.553282    2580 patch_prober.go:28] interesting pod/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 09 04:14:31.553356 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:31.553324    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c" podUID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 09 04:14:32.552964 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:32.552929    2580 patch_prober.go:28] interesting pod/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 09 04:14:32.553371 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:32.552979    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c" podUID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 09 04:14:33.552730 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:33.552680    2580 patch_prober.go:28] interesting pod/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 09 04:14:33.552941 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:33.552749    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c" podUID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 09 04:14:34.552630 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:34.552596    2580 patch_prober.go:28] interesting pod/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 09 04:14:34.553058 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:34.552663    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c" podUID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 09 04:14:35.553118 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:35.553082    2580 patch_prober.go:28] interesting pod/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Sep 09 04:14:35.553549 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:35.553154    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c" podUID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Sep 09 04:14:36.692407 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:36.692376    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:36.693481 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:36.693453    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:36.695490 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:36.695470    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:14:56.358513 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.358475    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"]
Sep 09 04:14:56.359236 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.359218    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9c4f78-c21a-4cf2-abfa-46663415ef81" containerName="istio-proxy"
Sep 09 04:14:56.362655 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.362636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.365196 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.365178    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f0f5e1c0-openshift-default-dockercfg-glzlz\""
Sep 09 04:14:56.372337 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.372316    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"]
Sep 09 04:14:56.414873 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.414843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-data\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.415043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.414888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-credential-socket\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.415043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.414927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-envoy\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.415043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.414955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-podinfo\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.415043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.414972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-workload-socket\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.415043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.414999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istiod-ca-cert\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.415311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.415118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-token\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.415311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.415170    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-workload-certs\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.415311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.415227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prjzw\" (UniqueName: \"kubernetes.io/projected/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-kube-api-access-prjzw\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.516633 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.516598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-workload-certs\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.516854 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.516658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prjzw\" (UniqueName: \"kubernetes.io/projected/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-kube-api-access-prjzw\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.516854 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.516700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-data\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.516854 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.516755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-credential-socket\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.516854 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.516831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-envoy\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.517136 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.517109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-workload-certs\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.517202 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.517116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-podinfo\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.517202 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.517181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-workload-socket\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.517314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.517123    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-credential-socket\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.517314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.517215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istiod-ca-cert\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.517314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.517222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-data\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.517314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.517262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-token\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.517531 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.517462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-workload-socket\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.518138 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.518110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istiod-ca-cert\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.519214 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.519192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-envoy\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.519614 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.519598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-podinfo\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.524598 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.524574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-token\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.524946 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.524926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prjzw\" (UniqueName: \"kubernetes.io/projected/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-kube-api-access-prjzw\") pod \"e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") " pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.675066 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.674983    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:56.963665 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.958036    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"]
Sep 09 04:14:56.966585 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.966542    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:14:56.966701 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.966626    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:14:56.966701 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:56.966685    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:14:57.367965 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:57.367929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" event={"ID":"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8","Type":"ContainerStarted","Data":"6eba31bf5539709eb730a2edf2a30bc0da93d5ac7e6a6cc2eb97181165218db5"}
Sep 09 04:14:57.368342 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:57.367983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" event={"ID":"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8","Type":"ContainerStarted","Data":"e081c5109d3941cc835bf65d5f8ec745eb6ec6b9b8ae1f629926b9ab1c05d5bd"}
Sep 09 04:14:57.392079 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:57.392034    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" podStartSLOduration=1.3920203789999999 podStartE2EDuration="1.392020379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:14:57.389460528 +0000 UTC m=+1018.339151182" watchObservedRunningTime="2026-09-09 04:14:57.392020379 +0000 UTC m=+1018.341711033"
Sep 09 04:14:57.675955 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:57.675862    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:58.682034 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:58.682003    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:59.380512 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:59.380474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:14:59.381511 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:14:59.381492    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:15:00.143453 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:00.143418    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn"]
Sep 09 04:15:00.147176 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:00.147155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn"
Sep 09 04:15:00.150133 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:00.150109    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:15:00.161326 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:00.161298    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn"
Sep 09 04:15:00.161555 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:00.161536    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn"]
Sep 09 04:15:00.470192 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:00.470158    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn"]
Sep 09 04:15:00.471168 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:15:00.471134    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f9745c_25ab_49bd_b23a_68c85b8ecabd.slice/crio-afd12dee1066246209befdbbf211d2690d8b0797b32a5d56e363d430699caa57 WatchSource:0}: Error finding container afd12dee1066246209befdbbf211d2690d8b0797b32a5d56e363d430699caa57: Status 404 returned error can't find the container with id afd12dee1066246209befdbbf211d2690d8b0797b32a5d56e363d430699caa57
Sep 09 04:15:00.474155 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:00.474135    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 04:15:01.395497 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:01.395463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn" event={"ID":"a3f9745c-25ab-49bd-b23a-68c85b8ecabd","Type":"ContainerStarted","Data":"afd12dee1066246209befdbbf211d2690d8b0797b32a5d56e363d430699caa57"}
Sep 09 04:15:03.410873 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:03.410837    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn" event={"ID":"a3f9745c-25ab-49bd-b23a-68c85b8ecabd","Type":"ContainerStarted","Data":"edde8cbc50d3fed0f6e66e9e66aa6219b8daaafaa02a25dfbadc8cea271dc5e3"}
Sep 09 04:15:03.428636 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:03.428586    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn" podStartSLOduration=0.986897075 podStartE2EDuration="3.428569574s" totalImagesPullingTime="2.441672499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:15:03.427267208 +0000 UTC m=+1024.376957861" watchObservedRunningTime="2026-09-09 04:15:03.428569574 +0000 UTC m=+1024.378260248"
Sep 09 04:15:04.418605 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:04.418566    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3f9745c-25ab-49bd-b23a-68c85b8ecabd" containerID="edde8cbc50d3fed0f6e66e9e66aa6219b8daaafaa02a25dfbadc8cea271dc5e3" exitCode=0
Sep 09 04:15:04.419019 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:04.418651    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn" event={"ID":"a3f9745c-25ab-49bd-b23a-68c85b8ecabd","Type":"ContainerDied","Data":"edde8cbc50d3fed0f6e66e9e66aa6219b8daaafaa02a25dfbadc8cea271dc5e3"}
Sep 09 04:15:05.564343 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:05.564318    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn"
Sep 09 04:15:06.432424 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:06.432397    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn"
Sep 09 04:15:06.432638 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:06.432439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn" event={"ID":"a3f9745c-25ab-49bd-b23a-68c85b8ecabd","Type":"ContainerDied","Data":"afd12dee1066246209befdbbf211d2690d8b0797b32a5d56e363d430699caa57"}
Sep 09 04:15:06.432638 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:06.432513    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afd12dee1066246209befdbbf211d2690d8b0797b32a5d56e363d430699caa57"
Sep 09 04:15:22.530282 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:22.530188    2580 generic.go:356] "Generic (PLEG): container finished" podID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerID="6eba31bf5539709eb730a2edf2a30bc0da93d5ac7e6a6cc2eb97181165218db5" exitCode=0
Sep 09 04:15:22.531054 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:22.530267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" event={"ID":"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8","Type":"ContainerDied","Data":"6eba31bf5539709eb730a2edf2a30bc0da93d5ac7e6a6cc2eb97181165218db5"}
Sep 09 04:15:22.531054 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:22.530832    2580 scope.go:117] "RemoveContainer" containerID="6eba31bf5539709eb730a2edf2a30bc0da93d5ac7e6a6cc2eb97181165218db5"
Sep 09 04:15:22.532148 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:22.532121    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:15:22.532223 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:22.532192    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:15:22.532284 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:22.532225    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:15:23.538686 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:23.538651    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" event={"ID":"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8","Type":"ContainerStarted","Data":"6d031d1d42cecdd993a9c15d5b90ad2f3c5f6f97a4c4e322d4629cac35aefd98"}
Sep 09 04:15:23.675316 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:23.675271    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:15:23.675566 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:23.675542    2580 patch_prober.go:28] interesting pod/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 09 04:15:23.675653 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:23.675598    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 09 04:15:24.676336 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:24.676298    2580 patch_prober.go:28] interesting pod/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 09 04:15:24.676732 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:24.676356    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 09 04:15:25.676072 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:25.676032    2580 patch_prober.go:28] interesting pod/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 09 04:15:25.676247 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:25.676103    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 09 04:15:26.675724 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:26.675686    2580 patch_prober.go:28] interesting pod/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 09 04:15:26.676129 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:26.675746    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 09 04:15:27.676213 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:27.676176    2580 patch_prober.go:28] interesting pod/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 09 04:15:27.676612 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:27.676245    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 09 04:15:28.675701 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:28.675665    2580 patch_prober.go:28] interesting pod/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 09 04:15:28.676032 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:28.675725    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 09 04:15:29.380958 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:29.380906    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:15:29.679480 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:29.679353    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:15:29.680282 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:29.680260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:15:49.098175 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:49.098139    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"]
Sep 09 04:15:49.098569 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:49.098346    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerName="istio-proxy" containerID="cri-o://6d031d1d42cecdd993a9c15d5b90ad2f3c5f6f97a4c4e322d4629cac35aefd98" gracePeriod=30
Sep 09 04:15:54.365921 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.365860    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:15:54.496311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496280    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-data\") pod \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") "
Sep 09 04:15:54.496311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496318    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-credential-socket\") pod \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") "
Sep 09 04:15:54.496606 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496341    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-workload-socket\") pod \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") "
Sep 09 04:15:54.496606 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496365    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-podinfo\") pod \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") "
Sep 09 04:15:54.496606 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496381    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-token\") pod \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") "
Sep 09 04:15:54.496606 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496465    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istiod-ca-cert\") pod \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") "
Sep 09 04:15:54.496606 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prjzw\" (UniqueName: \"kubernetes.io/projected/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-kube-api-access-prjzw\") pod \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") "
Sep 09 04:15:54.496606 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496518    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-envoy\") pod \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") "
Sep 09 04:15:54.496606 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496526    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-data" (OuterVolumeSpecName: "istio-data") pod "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" (UID: "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:15:54.496606 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496540    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-workload-certs\") pod \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\" (UID: \"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8\") "
Sep 09 04:15:54.496606 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" (UID: "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:15:54.497079 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496653    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" (UID: "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:15:54.497079 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496786    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" (UID: "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:15:54.497079 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496832    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" (UID: "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:15:54.497079 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496953    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istiod-ca-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:15:54.497079 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496972    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-workload-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:15:54.497079 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496986    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-data\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:15:54.497079 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.496999    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-credential-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:15:54.497079 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.497013    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-workload-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:15:54.498877 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.498844    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-kube-api-access-prjzw" (OuterVolumeSpecName: "kube-api-access-prjzw") pod "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" (UID: "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8"). InnerVolumeSpecName "kube-api-access-prjzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:15:54.499014 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.498869    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" (UID: "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:15:54.499214 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.499197    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" (UID: "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:15:54.499260 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.499221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-token" (OuterVolumeSpecName: "istio-token") pod "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" (UID: "97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:15:54.597939 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.597908    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prjzw\" (UniqueName: \"kubernetes.io/projected/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-kube-api-access-prjzw\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:15:54.597939 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.597935    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-envoy\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:15:54.597939 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.597944    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-podinfo\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:15:54.598166 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.597953    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:15:54.737487 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.737400    2580 generic.go:356] "Generic (PLEG): container finished" podID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerID="6d031d1d42cecdd993a9c15d5b90ad2f3c5f6f97a4c4e322d4629cac35aefd98" exitCode=0
Sep 09 04:15:54.737487 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.737447    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"
Sep 09 04:15:54.737703 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.737488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" event={"ID":"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8","Type":"ContainerDied","Data":"6d031d1d42cecdd993a9c15d5b90ad2f3c5f6f97a4c4e322d4629cac35aefd98"}
Sep 09 04:15:54.737703 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.737545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28" event={"ID":"97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8","Type":"ContainerDied","Data":"e081c5109d3941cc835bf65d5f8ec745eb6ec6b9b8ae1f629926b9ab1c05d5bd"}
Sep 09 04:15:54.737703 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.737569    2580 scope.go:117] "RemoveContainer" containerID="6d031d1d42cecdd993a9c15d5b90ad2f3c5f6f97a4c4e322d4629cac35aefd98"
Sep 09 04:15:54.747726 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.747708    2580 scope.go:117] "RemoveContainer" containerID="6eba31bf5539709eb730a2edf2a30bc0da93d5ac7e6a6cc2eb97181165218db5"
Sep 09 04:15:54.756144 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.756126    2580 scope.go:117] "RemoveContainer" containerID="6d031d1d42cecdd993a9c15d5b90ad2f3c5f6f97a4c4e322d4629cac35aefd98"
Sep 09 04:15:54.756413 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:15:54.756394    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d031d1d42cecdd993a9c15d5b90ad2f3c5f6f97a4c4e322d4629cac35aefd98\": container with ID starting with 6d031d1d42cecdd993a9c15d5b90ad2f3c5f6f97a4c4e322d4629cac35aefd98 not found: ID does not exist" containerID="6d031d1d42cecdd993a9c15d5b90ad2f3c5f6f97a4c4e322d4629cac35aefd98"
Sep 09 04:15:54.756461 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.756421    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d031d1d42cecdd993a9c15d5b90ad2f3c5f6f97a4c4e322d4629cac35aefd98"} err="failed to get container status \"6d031d1d42cecdd993a9c15d5b90ad2f3c5f6f97a4c4e322d4629cac35aefd98\": rpc error: code = NotFound desc = could not find container \"6d031d1d42cecdd993a9c15d5b90ad2f3c5f6f97a4c4e322d4629cac35aefd98\": container with ID starting with 6d031d1d42cecdd993a9c15d5b90ad2f3c5f6f97a4c4e322d4629cac35aefd98 not found: ID does not exist"
Sep 09 04:15:54.756461 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.756437    2580 scope.go:117] "RemoveContainer" containerID="6eba31bf5539709eb730a2edf2a30bc0da93d5ac7e6a6cc2eb97181165218db5"
Sep 09 04:15:54.756695 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:15:54.756676    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eba31bf5539709eb730a2edf2a30bc0da93d5ac7e6a6cc2eb97181165218db5\": container with ID starting with 6eba31bf5539709eb730a2edf2a30bc0da93d5ac7e6a6cc2eb97181165218db5 not found: ID does not exist" containerID="6eba31bf5539709eb730a2edf2a30bc0da93d5ac7e6a6cc2eb97181165218db5"
Sep 09 04:15:54.756745 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.756700    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eba31bf5539709eb730a2edf2a30bc0da93d5ac7e6a6cc2eb97181165218db5"} err="failed to get container status \"6eba31bf5539709eb730a2edf2a30bc0da93d5ac7e6a6cc2eb97181165218db5\": rpc error: code = NotFound desc = could not find container \"6eba31bf5539709eb730a2edf2a30bc0da93d5ac7e6a6cc2eb97181165218db5\": container with ID starting with 6eba31bf5539709eb730a2edf2a30bc0da93d5ac7e6a6cc2eb97181165218db5 not found: ID does not exist"
Sep 09 04:15:54.766153 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.766133    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"]
Sep 09 04:15:54.771881 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:54.771860    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f0f5e1c0-openshift-default-7987785456-l4f28"]
Sep 09 04:15:55.596917 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:15:55.596888    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" path="/var/lib/kubelet/pods/97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8/volumes"
Sep 09 04:16:04.184233 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.184202    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"]
Sep 09 04:16:04.184672 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.184656    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerName="istio-proxy"
Sep 09 04:16:04.184718 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.184675    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerName="istio-proxy"
Sep 09 04:16:04.184718 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.184692    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerName="istio-proxy"
Sep 09 04:16:04.184718 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.184698    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerName="istio-proxy"
Sep 09 04:16:04.184718 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.184704    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f9745c-25ab-49bd-b23a-68c85b8ecabd" containerName="cleanup"
Sep 09 04:16:04.184718 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.184709    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f9745c-25ab-49bd-b23a-68c85b8ecabd" containerName="cleanup"
Sep 09 04:16:04.184923 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.184797    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerName="istio-proxy"
Sep 09 04:16:04.184923 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.184807    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f9745c-25ab-49bd-b23a-68c85b8ecabd" containerName="cleanup"
Sep 09 04:16:04.189287 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.189266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.191695 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.191673    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-46ddac-openshift-default-dockercfg-l5v5p\""
Sep 09 04:16:04.192262 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.192239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-envoy\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.192360 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.192317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istiod-ca-cert\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.192360 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.192350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-podinfo\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.192477 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.192375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s64qn\" (UniqueName: \"kubernetes.io/projected/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-kube-api-access-s64qn\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.192477 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.192412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-credential-socket\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.192477 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.192458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-token\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.192675 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.192488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-data\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.192675 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.192510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-workload-socket\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.192675 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.192557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-workload-certs\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.199214 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.199190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"]
Sep 09 04:16:04.293365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.293314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-data\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.293365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.293368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-workload-socket\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.293624 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.293412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-workload-certs\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.293624 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.293467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-envoy\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.293624 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.293530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istiod-ca-cert\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.293624 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.293564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-podinfo\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.293624 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.293591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s64qn\" (UniqueName: \"kubernetes.io/projected/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-kube-api-access-s64qn\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.293931 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.293629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-credential-socket\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.293931 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.293652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-token\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.293931 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.293809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-data\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.294397 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.294375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-workload-certs\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.294511 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.294460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istiod-ca-cert\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.294843 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.294826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-workload-socket\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.295165 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.295149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-credential-socket\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.297719 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.297690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-podinfo\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.304069 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.304041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-envoy\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.307789 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.307702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-token\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.308678 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.308654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s64qn\" (UniqueName: \"kubernetes.io/projected/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-kube-api-access-s64qn\") pod \"e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") " pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.500914 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.500815    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:04.783383 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:16:04.783356    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca39590_7e3c_49d5_9b9e_14a78e7c26e8.slice/crio-66f45239ae8e898cc613fd7ec7075ef2c601b53d557ab4a1c715d1394aaa40d1 WatchSource:0}: Error finding container 66f45239ae8e898cc613fd7ec7075ef2c601b53d557ab4a1c715d1394aaa40d1: Status 404 returned error can't find the container with id 66f45239ae8e898cc613fd7ec7075ef2c601b53d557ab4a1c715d1394aaa40d1
Sep 09 04:16:04.786202 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.786161    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:04.786295 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.786266    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:04.786335 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.786310    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:04.789887 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.789868    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"]
Sep 09 04:16:04.798910 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:04.798887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts" event={"ID":"bca39590-7e3c-49d5-9b9e-14a78e7c26e8","Type":"ContainerStarted","Data":"66f45239ae8e898cc613fd7ec7075ef2c601b53d557ab4a1c715d1394aaa40d1"}
Sep 09 04:16:05.807660 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:05.807616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts" event={"ID":"bca39590-7e3c-49d5-9b9e-14a78e7c26e8","Type":"ContainerStarted","Data":"f827eec0183a7c59b4f0d90e7c273415d03df7e9ce1895fcca1aeed6f4cab64d"}
Sep 09 04:16:05.829129 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:05.829084    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts" podStartSLOduration=1.829070385 podStartE2EDuration="1.829070385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:16:05.828736662 +0000 UTC m=+1086.778427315" watchObservedRunningTime="2026-09-09 04:16:05.829070385 +0000 UTC m=+1086.778761038"
Sep 09 04:16:06.501915 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:06.501872    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:06.508064 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:06.508030    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:06.813923 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:06.813840    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:06.815010 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:06.814990    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:08.832607 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:08.832572    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"]
Sep 09 04:16:08.833056 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:08.833042    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="97aa50e4-b37e-4bb5-bd80-0b6c6a86bdd8" containerName="istio-proxy"
Sep 09 04:16:08.836228 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:08.836203    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:08.838680 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:08.838659    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-46ddac-serving-cert\""
Sep 09 04:16:08.845723 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:08.845699    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"]
Sep 09 04:16:08.937986 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:08.937947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:08.938162 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:08.938015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzdg\" (UniqueName: \"kubernetes.io/projected/f19e7a5c-ad13-4f07-9292-317c53eefe03-kube-api-access-6kzdg\") pod \"maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:08.938162 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:08.938141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:08.938307 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:08.938182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19e7a5c-ad13-4f07-9292-317c53eefe03-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:09.039501 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:09.039454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzdg\" (UniqueName: \"kubernetes.io/projected/f19e7a5c-ad13-4f07-9292-317c53eefe03-kube-api-access-6kzdg\") pod \"maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:09.039706 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:09.039539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:09.039706 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:09.039567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19e7a5c-ad13-4f07-9292-317c53eefe03-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:09.039706 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:16:09.039678    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-serving-cert: secret "maas-api-e2e-shared-a-w2-46ddac-serving-cert" not found
Sep 09 04:16:09.039916 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:09.039714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:09.039916 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:16:09.039750    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-maas-api-tls podName:f19e7a5c-ad13-4f07-9292-317c53eefe03 nodeName:}" failed. No retries permitted until 2026-09-09 04:16:09.539730927 +0000 UTC m=+1090.489421575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-maas-api-tls") pod "maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp" (UID: "f19e7a5c-ad13-4f07-9292-317c53eefe03") : secret "maas-api-e2e-shared-a-w2-46ddac-serving-cert" not found
Sep 09 04:16:09.040352 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:09.040321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19e7a5c-ad13-4f07-9292-317c53eefe03-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:09.042392 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:09.042362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:09.048747 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:09.048727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzdg\" (UniqueName: \"kubernetes.io/projected/f19e7a5c-ad13-4f07-9292-317c53eefe03-kube-api-access-6kzdg\") pod \"maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:09.544552 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:09.544507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:09.547185 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:09.547159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:09.751618 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:09.751587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:10.038124 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:10.038099    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"]
Sep 09 04:16:10.040014 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:16:10.039981    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19e7a5c_ad13_4f07_9292_317c53eefe03.slice/crio-90121fe23e191545bac32501c4073459cf4cb1c82450924a2c037aca76514f62 WatchSource:0}: Error finding container 90121fe23e191545bac32501c4073459cf4cb1c82450924a2c037aca76514f62: Status 404 returned error can't find the container with id 90121fe23e191545bac32501c4073459cf4cb1c82450924a2c037aca76514f62
Sep 09 04:16:10.843442 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:10.843389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp" event={"ID":"f19e7a5c-ad13-4f07-9292-317c53eefe03","Type":"ContainerStarted","Data":"5081c877a60a7581d62145a22bb19aaf10790b302e29c0372ba9780ad47b75cb"}
Sep 09 04:16:10.843618 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:10.843462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp" event={"ID":"f19e7a5c-ad13-4f07-9292-317c53eefe03","Type":"ContainerStarted","Data":"90121fe23e191545bac32501c4073459cf4cb1c82450924a2c037aca76514f62"}
Sep 09 04:16:10.843618 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:10.843527    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:10.862481 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:10.862434    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp" podStartSLOduration=2.147323613 podStartE2EDuration="2.862417942s" totalImagesPullingTime="715.094329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:16:10.86101797 +0000 UTC m=+1091.810708636" watchObservedRunningTime="2026-09-09 04:16:10.862417942 +0000 UTC m=+1091.812108596"
Sep 09 04:16:13.893270 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:13.893230    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"]
Sep 09 04:16:13.893707 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:13.893432    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c" podUID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerName="istio-proxy" containerID="cri-o://6ac98526f1bcd0c3f4e4f3f76c62ee4ae00df3e4435aa530eaa19052df3539cc" gracePeriod=30
Sep 09 04:16:14.592418 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.592383    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"]
Sep 09 04:16:14.596117 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.596095    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.598810 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.598758    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-1215c1-openshift-default-dockercfg-h4b79\""
Sep 09 04:16:14.609886 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.609853    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"]
Sep 09 04:16:14.697918 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.697883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-data\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.698110 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.697933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-credential-socket\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.698110 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.697984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-podinfo\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.698110 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.698048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-token\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.698110 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.698072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-envoy\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.698110 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.698103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnq82\" (UniqueName: \"kubernetes.io/projected/6ca8dc88-5e8b-423b-b190-3500de2bfae4-kube-api-access-lnq82\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.698385 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.698152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-workload-certs\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.698385 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.698210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-workload-socket\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.698385 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.698233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istiod-ca-cert\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.799289 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.799254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-data\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.799289 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.799293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-credential-socket\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.799534 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.799320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-podinfo\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.799534 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.799352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-token\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.799534 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.799368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-envoy\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.799534 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.799386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnq82\" (UniqueName: \"kubernetes.io/projected/6ca8dc88-5e8b-423b-b190-3500de2bfae4-kube-api-access-lnq82\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.799534 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.799414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-workload-certs\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.799534 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.799468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-workload-socket\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.799534 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.799492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istiod-ca-cert\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.799953 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.799711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-credential-socket\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.799953 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.799720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-data\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.800051 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.799981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-workload-certs\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.800121 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.800092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-workload-socket\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.800474 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.800451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istiod-ca-cert\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.801901 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.801858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-envoy\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.802139 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.802118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-podinfo\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.809573 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.809544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-token\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.810271 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.810246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnq82\" (UniqueName: \"kubernetes.io/projected/6ca8dc88-5e8b-423b-b190-3500de2bfae4-kube-api-access-lnq82\") pod \"e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") " pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:14.911452 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:14.911366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:15.193434 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:15.193409    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"]
Sep 09 04:16:15.195628 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:16:15.195602    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca8dc88_5e8b_423b_b190_3500de2bfae4.slice/crio-5659e5a05f23c4f053eb901bc079baaf387237dbec3a0e972a09ab3d888e6416 WatchSource:0}: Error finding container 5659e5a05f23c4f053eb901bc079baaf387237dbec3a0e972a09ab3d888e6416: Status 404 returned error can't find the container with id 5659e5a05f23c4f053eb901bc079baaf387237dbec3a0e972a09ab3d888e6416
Sep 09 04:16:15.198018 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:15.197984    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:15.198108 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:15.198052    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:15.198108 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:15.198078    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:15.873609 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:15.873570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn" event={"ID":"6ca8dc88-5e8b-423b-b190-3500de2bfae4","Type":"ContainerStarted","Data":"d77b5284bd50cb73c44eedeed9578527d188146aafab77649591681a6f43ac3b"}
Sep 09 04:16:15.873822 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:15.873630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn" event={"ID":"6ca8dc88-5e8b-423b-b190-3500de2bfae4","Type":"ContainerStarted","Data":"5659e5a05f23c4f053eb901bc079baaf387237dbec3a0e972a09ab3d888e6416"}
Sep 09 04:16:15.896147 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:15.896103    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn" podStartSLOduration=1.8960892980000001 podStartE2EDuration="1.896089298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:16:15.893613426 +0000 UTC m=+1096.843304081" watchObservedRunningTime="2026-09-09 04:16:15.896089298 +0000 UTC m=+1096.845779951"
Sep 09 04:16:15.912356 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:15.912333    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:16.855112 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:16.855084    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:16:16.918503 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:16.918468    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:17.884917 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:17.884889    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:17.885989 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:17.885972    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:16:19.159883 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.159825    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:16:19.244058 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244031    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-data\") pod \"3a0be37e-de1f-4c5f-9050-539dd642c804\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") "
Sep 09 04:16:19.244058 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244063    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-credential-socket\") pod \"3a0be37e-de1f-4c5f-9050-539dd642c804\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") "
Sep 09 04:16:19.244308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244080    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-workload-socket\") pod \"3a0be37e-de1f-4c5f-9050-539dd642c804\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") "
Sep 09 04:16:19.244308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244155    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-token\") pod \"3a0be37e-de1f-4c5f-9050-539dd642c804\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") "
Sep 09 04:16:19.244308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244197    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-workload-certs\") pod \"3a0be37e-de1f-4c5f-9050-539dd642c804\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") "
Sep 09 04:16:19.244308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244221    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-podinfo\") pod \"3a0be37e-de1f-4c5f-9050-539dd642c804\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") "
Sep 09 04:16:19.244523 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244303    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a0be37e-de1f-4c5f-9050-539dd642c804" (UID: "3a0be37e-de1f-4c5f-9050-539dd642c804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:16:19.244523 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0be37e-de1f-4c5f-9050-539dd642c804-istiod-ca-cert\") pod \"3a0be37e-de1f-4c5f-9050-539dd642c804\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") "
Sep 09 04:16:19.244523 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244360    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a0be37e-de1f-4c5f-9050-539dd642c804" (UID: "3a0be37e-de1f-4c5f-9050-539dd642c804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:16:19.244523 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244416    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2npvs\" (UniqueName: \"kubernetes.io/projected/3a0be37e-de1f-4c5f-9050-539dd642c804-kube-api-access-2npvs\") pod \"3a0be37e-de1f-4c5f-9050-539dd642c804\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") "
Sep 09 04:16:19.244523 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244460    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a0be37e-de1f-4c5f-9050-539dd642c804" (UID: "3a0be37e-de1f-4c5f-9050-539dd642c804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:16:19.244523 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244469    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-envoy\") pod \"3a0be37e-de1f-4c5f-9050-539dd642c804\" (UID: \"3a0be37e-de1f-4c5f-9050-539dd642c804\") "
Sep 09 04:16:19.244882 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244556    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a0be37e-de1f-4c5f-9050-539dd642c804" (UID: "3a0be37e-de1f-4c5f-9050-539dd642c804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:16:19.244882 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244664    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0be37e-de1f-4c5f-9050-539dd642c804-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a0be37e-de1f-4c5f-9050-539dd642c804" (UID: "3a0be37e-de1f-4c5f-9050-539dd642c804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:16:19.245095 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244957    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0be37e-de1f-4c5f-9050-539dd642c804-istiod-ca-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:16:19.245095 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244978    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-data\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:16:19.245095 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.244993    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-credential-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:16:19.245095 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.245007    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-workload-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:16:19.245363 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.245105    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-workload-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:16:19.246580 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.246552    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a0be37e-de1f-4c5f-9050-539dd642c804" (UID: "3a0be37e-de1f-4c5f-9050-539dd642c804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:16:19.246684 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.246618    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a0be37e-de1f-4c5f-9050-539dd642c804" (UID: "3a0be37e-de1f-4c5f-9050-539dd642c804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:16:19.246789 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.246754    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a0be37e-de1f-4c5f-9050-539dd642c804" (UID: "3a0be37e-de1f-4c5f-9050-539dd642c804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:16:19.246828 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.246793    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0be37e-de1f-4c5f-9050-539dd642c804-kube-api-access-2npvs" (OuterVolumeSpecName: "kube-api-access-2npvs") pod "3a0be37e-de1f-4c5f-9050-539dd642c804" (UID: "3a0be37e-de1f-4c5f-9050-539dd642c804"). InnerVolumeSpecName "kube-api-access-2npvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:16:19.346161 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.346122    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:16:19.346161 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.346154    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-podinfo\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:16:19.346161 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.346164    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2npvs\" (UniqueName: \"kubernetes.io/projected/3a0be37e-de1f-4c5f-9050-539dd642c804-kube-api-access-2npvs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:16:19.346161 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.346173    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0be37e-de1f-4c5f-9050-539dd642c804-istio-envoy\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:16:19.898742 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.898655    2580 generic.go:356] "Generic (PLEG): container finished" podID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerID="6ac98526f1bcd0c3f4e4f3f76c62ee4ae00df3e4435aa530eaa19052df3539cc" exitCode=0
Sep 09 04:16:19.898742 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.898702    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"
Sep 09 04:16:19.898979 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.898739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c" event={"ID":"3a0be37e-de1f-4c5f-9050-539dd642c804","Type":"ContainerDied","Data":"6ac98526f1bcd0c3f4e4f3f76c62ee4ae00df3e4435aa530eaa19052df3539cc"}
Sep 09 04:16:19.898979 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.898813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c" event={"ID":"3a0be37e-de1f-4c5f-9050-539dd642c804","Type":"ContainerDied","Data":"b77e84a7dcf1c297dd37aa8acc5b4d42aa2899f16365113a17ba72d771a33ff2"}
Sep 09 04:16:19.898979 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.898842    2580 scope.go:117] "RemoveContainer" containerID="6ac98526f1bcd0c3f4e4f3f76c62ee4ae00df3e4435aa530eaa19052df3539cc"
Sep 09 04:16:19.910806 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.910785    2580 scope.go:117] "RemoveContainer" containerID="7a2d882229d923b0825a09dece996448dd8560d0a31ec7a8b2a6e91378a8659f"
Sep 09 04:16:19.920097 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.920059    2580 scope.go:117] "RemoveContainer" containerID="6ac98526f1bcd0c3f4e4f3f76c62ee4ae00df3e4435aa530eaa19052df3539cc"
Sep 09 04:16:19.920396 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:16:19.920342    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac98526f1bcd0c3f4e4f3f76c62ee4ae00df3e4435aa530eaa19052df3539cc\": container with ID starting with 6ac98526f1bcd0c3f4e4f3f76c62ee4ae00df3e4435aa530eaa19052df3539cc not found: ID does not exist" containerID="6ac98526f1bcd0c3f4e4f3f76c62ee4ae00df3e4435aa530eaa19052df3539cc"
Sep 09 04:16:19.920396 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.920367    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"]
Sep 09 04:16:19.920591 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.920367    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac98526f1bcd0c3f4e4f3f76c62ee4ae00df3e4435aa530eaa19052df3539cc"} err="failed to get container status \"6ac98526f1bcd0c3f4e4f3f76c62ee4ae00df3e4435aa530eaa19052df3539cc\": rpc error: code = NotFound desc = could not find container \"6ac98526f1bcd0c3f4e4f3f76c62ee4ae00df3e4435aa530eaa19052df3539cc\": container with ID starting with 6ac98526f1bcd0c3f4e4f3f76c62ee4ae00df3e4435aa530eaa19052df3539cc not found: ID does not exist"
Sep 09 04:16:19.920591 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.920413    2580 scope.go:117] "RemoveContainer" containerID="7a2d882229d923b0825a09dece996448dd8560d0a31ec7a8b2a6e91378a8659f"
Sep 09 04:16:19.920882 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:16:19.920812    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2d882229d923b0825a09dece996448dd8560d0a31ec7a8b2a6e91378a8659f\": container with ID starting with 7a2d882229d923b0825a09dece996448dd8560d0a31ec7a8b2a6e91378a8659f not found: ID does not exist" containerID="7a2d882229d923b0825a09dece996448dd8560d0a31ec7a8b2a6e91378a8659f"
Sep 09 04:16:19.920882 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.920855    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2d882229d923b0825a09dece996448dd8560d0a31ec7a8b2a6e91378a8659f"} err="failed to get container status \"7a2d882229d923b0825a09dece996448dd8560d0a31ec7a8b2a6e91378a8659f\": rpc error: code = NotFound desc = could not find container \"7a2d882229d923b0825a09dece996448dd8560d0a31ec7a8b2a6e91378a8659f\": container with ID starting with 7a2d882229d923b0825a09dece996448dd8560d0a31ec7a8b2a6e91378a8659f not found: ID does not exist"
Sep 09 04:16:19.923704 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:19.923685    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-b80654-openshift-default-cfd4d7468-7pp4c"]
Sep 09 04:16:21.596101 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:21.596068    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a0be37e-de1f-4c5f-9050-539dd642c804" path="/var/lib/kubelet/pods/3a0be37e-de1f-4c5f-9050-539dd642c804/volumes"
Sep 09 04:16:33.311828 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.311785    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"]
Sep 09 04:16:33.312431 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.312413    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerName="istio-proxy"
Sep 09 04:16:33.312505 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.312433    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerName="istio-proxy"
Sep 09 04:16:33.312505 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.312451    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerName="istio-proxy"
Sep 09 04:16:33.312505 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.312459    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerName="istio-proxy"
Sep 09 04:16:33.312666 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.312585    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerName="istio-proxy"
Sep 09 04:16:33.312666 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.312598    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a0be37e-de1f-4c5f-9050-539dd642c804" containerName="istio-proxy"
Sep 09 04:16:33.316372 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.316351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.319180 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.319161    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-9f2be5-openshift-default-dockercfg-4slnm\""
Sep 09 04:16:33.329469 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.329443    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"]
Sep 09 04:16:33.489262 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.489218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnk7f\" (UniqueName: \"kubernetes.io/projected/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-kube-api-access-hnk7f\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.489468 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.489304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-credential-socket\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.489468 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.489341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-podinfo\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.489468 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.489381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-data\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.489468 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.489418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-envoy\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.489468 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.489458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istiod-ca-cert\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.489716 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.489506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-workload-certs\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.489716 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.489540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-token\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.489716 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.489564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-workload-socket\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.590190 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.590153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-data\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.590391 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.590199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-envoy\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.590391 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.590229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istiod-ca-cert\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.590391 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.590251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-workload-certs\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.590391 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.590282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-token\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.590391 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.590306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-workload-socket\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.590391 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.590355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnk7f\" (UniqueName: \"kubernetes.io/projected/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-kube-api-access-hnk7f\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.590803 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.590431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-credential-socket\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.590803 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.590466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-podinfo\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.590803 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.590757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-data\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.590995 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.590760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-workload-certs\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.591073 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.591051    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-workload-socket\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.591137 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.591063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-credential-socket\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.591465 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.591445    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istiod-ca-cert\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.593634 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.593571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-podinfo\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.593784 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.593728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-envoy\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.598576 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.598544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-token\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.598930 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.598908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnk7f\" (UniqueName: \"kubernetes.io/projected/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-kube-api-access-hnk7f\") pod \"e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") " pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.632744 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.632706    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:33.921909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.921877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"]
Sep 09 04:16:33.922284 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:16:33.922260    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b35da7_d29f_4d0c_9d43_ba3c02ef203e.slice/crio-2200d72ec268896941d558a44d05f9738d42c042e8d6d6d7d3750a2d15d0eaab WatchSource:0}: Error finding container 2200d72ec268896941d558a44d05f9738d42c042e8d6d6d7d3750a2d15d0eaab: Status 404 returned error can't find the container with id 2200d72ec268896941d558a44d05f9738d42c042e8d6d6d7d3750a2d15d0eaab
Sep 09 04:16:33.924716 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.924685    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:33.924848 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.924749    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:33.924848 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.924831    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:33.989752 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.989726    2580 generic.go:356] "Generic (PLEG): container finished" podID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerID="f827eec0183a7c59b4f0d90e7c273415d03df7e9ce1895fcca1aeed6f4cab64d" exitCode=0
Sep 09 04:16:33.989874 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.989807    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts" event={"ID":"bca39590-7e3c-49d5-9b9e-14a78e7c26e8","Type":"ContainerDied","Data":"f827eec0183a7c59b4f0d90e7c273415d03df7e9ce1895fcca1aeed6f4cab64d"}
Sep 09 04:16:33.990295 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.990277    2580 scope.go:117] "RemoveContainer" containerID="f827eec0183a7c59b4f0d90e7c273415d03df7e9ce1895fcca1aeed6f4cab64d"
Sep 09 04:16:33.991442 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.991408    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:33.991539 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.991495    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:33.991584 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.991539    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:33.991626 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.991589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" event={"ID":"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e","Type":"ContainerStarted","Data":"e3a1358bcf93a39e60e5559ad2023988576b812d823461a790a5b7c578678d42"}
Sep 09 04:16:33.991664 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:33.991633    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" event={"ID":"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e","Type":"ContainerStarted","Data":"2200d72ec268896941d558a44d05f9738d42c042e8d6d6d7d3750a2d15d0eaab"}
Sep 09 04:16:34.040971 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:34.040351    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" podStartSLOduration=1.040330635 podStartE2EDuration="1.040330635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:16:34.037089833 +0000 UTC m=+1114.986780487" watchObservedRunningTime="2026-09-09 04:16:34.040330635 +0000 UTC m=+1114.990021289"
Sep 09 04:16:34.633421 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:34.633383    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:34.639155 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:34.639129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:35.020852 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:35.019780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts" event={"ID":"bca39590-7e3c-49d5-9b9e-14a78e7c26e8","Type":"ContainerStarted","Data":"85cd24fcffc4240d5a9ad3988daf114e355f59f86ce0d43f62bc173afedb9d00"}
Sep 09 04:16:35.022522 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:35.021537    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:35.030794 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:35.029095    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:35.501750 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:35.501715    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:35.502948 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:35.502921    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 09 04:16:35.503212 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:35.503161    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts" podUID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 09 04:16:36.501411 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:36.501373    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 09 04:16:36.501933 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:36.501434    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts" podUID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 09 04:16:36.814261 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:36.814163    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:37.502284 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:37.502247    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 09 04:16:37.502894 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:37.502864    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts" podUID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 09 04:16:38.502280 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:38.502238    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 09 04:16:38.502751 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:38.502316    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts" podUID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 09 04:16:39.501242 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:39.501206    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 09 04:16:39.501439 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:39.501257    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts" podUID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 09 04:16:40.506900 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:40.506868    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:40.507949 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:40.507926    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:16:56.167204 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:56.167169    2580 generic.go:356] "Generic (PLEG): container finished" podID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerID="e3a1358bcf93a39e60e5559ad2023988576b812d823461a790a5b7c578678d42" exitCode=0
Sep 09 04:16:56.167614 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:56.167240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" event={"ID":"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e","Type":"ContainerDied","Data":"e3a1358bcf93a39e60e5559ad2023988576b812d823461a790a5b7c578678d42"}
Sep 09 04:16:56.167657 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:56.167638    2580 scope.go:117] "RemoveContainer" containerID="e3a1358bcf93a39e60e5559ad2023988576b812d823461a790a5b7c578678d42"
Sep 09 04:16:56.168882 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:56.168856    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:56.168958 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:56.168944    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:56.169168 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:56.168988    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:16:57.175252 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:57.175215    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" event={"ID":"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e","Type":"ContainerStarted","Data":"6c39a43d8e9ca4a6b1ee15ead1090349e94bd9df21eafe2943f6b17892d0c381"}
Sep 09 04:16:57.633321 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:57.633285    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:16:57.633614 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:57.633339    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 09 04:16:57.633811 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:57.633760    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 09 04:16:58.633470 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:58.633435    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 09 04:16:58.634004 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:58.633512    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 09 04:16:59.633472 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:59.633432    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 09 04:16:59.633672 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:59.633483    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 09 04:16:59.739318 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:16:59.736105    2580 scope.go:117] "RemoveContainer" containerID="ffbdcb60c88073511591d37e80abba10485258bf023c5fc50ece7b78e41ea434"
Sep 09 04:17:00.633755 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:00.633710    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 09 04:17:00.634171 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:00.633804    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 09 04:17:01.596678 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:01.596649    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7814-139976933218896"]
Sep 09 04:17:01.601544 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:01.601529    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:01.604787 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:01.604736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pcbxq\""
Sep 09 04:17:01.609184 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:01.609153    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7814-139976933218896"]
Sep 09 04:17:01.638335 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:01.638303    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 09 04:17:01.638903 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:01.638860    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 09 04:17:01.677141 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:01.677111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7pr5\" (UniqueName: \"kubernetes.io/projected/ff0c4c56-fecb-40dd-b2ba-32f444d7c191-kube-api-access-t7pr5\") pod \"test-curl-7814-139976933218896\" (UID: \"ff0c4c56-fecb-40dd-b2ba-32f444d7c191\") " pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:01.778185 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:01.778148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7pr5\" (UniqueName: \"kubernetes.io/projected/ff0c4c56-fecb-40dd-b2ba-32f444d7c191-kube-api-access-t7pr5\") pod \"test-curl-7814-139976933218896\" (UID: \"ff0c4c56-fecb-40dd-b2ba-32f444d7c191\") " pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:01.787237 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:01.787212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7pr5\" (UniqueName: \"kubernetes.io/projected/ff0c4c56-fecb-40dd-b2ba-32f444d7c191-kube-api-access-t7pr5\") pod \"test-curl-7814-139976933218896\" (UID: \"ff0c4c56-fecb-40dd-b2ba-32f444d7c191\") " pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:01.937976 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:01.937889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:02.212478 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:02.212295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7814-139976933218896"]
Sep 09 04:17:02.230927 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:17:02.221357    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff0c4c56_fecb_40dd_b2ba_32f444d7c191.slice/crio-6c04f17d238dc03c40b5b7de7857e0cef7d9f6425f3e07f63e01e7cbe3efc900 WatchSource:0}: Error finding container 6c04f17d238dc03c40b5b7de7857e0cef7d9f6425f3e07f63e01e7cbe3efc900: Status 404 returned error can't find the container with id 6c04f17d238dc03c40b5b7de7857e0cef7d9f6425f3e07f63e01e7cbe3efc900
Sep 09 04:17:02.637141 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:02.637111    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:17:02.637393 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:02.637375    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:17:02.638117 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:02.638094    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:17:03.219837 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:03.219800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7814-139976933218896" event={"ID":"ff0c4c56-fecb-40dd-b2ba-32f444d7c191","Type":"ContainerStarted","Data":"6c04f17d238dc03c40b5b7de7857e0cef7d9f6425f3e07f63e01e7cbe3efc900"}
Sep 09 04:17:05.236030 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:05.235993    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff0c4c56-fecb-40dd-b2ba-32f444d7c191" containerID="54da988c4d3bb016367af4caa1e6bc3d1d0b618bfb7d0b8b1f149a2e3aa0d9b8" exitCode=0
Sep 09 04:17:05.236665 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:05.236083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7814-139976933218896" event={"ID":"ff0c4c56-fecb-40dd-b2ba-32f444d7c191","Type":"ContainerDied","Data":"54da988c4d3bb016367af4caa1e6bc3d1d0b618bfb7d0b8b1f149a2e3aa0d9b8"}
Sep 09 04:17:06.380514 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.380490    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:06.414652 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.414620    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7814-139976933218896_ff0c4c56-fecb-40dd-b2ba-32f444d7c191/test-curl-7814-139976933218896/0.log"
Sep 09 04:17:06.441299 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.441264    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7814-139976933218896"]
Sep 09 04:17:06.445172 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.445148    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7814-139976933218896"]
Sep 09 04:17:06.528816 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.528715    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7pr5\" (UniqueName: \"kubernetes.io/projected/ff0c4c56-fecb-40dd-b2ba-32f444d7c191-kube-api-access-t7pr5\") pod \"ff0c4c56-fecb-40dd-b2ba-32f444d7c191\" (UID: \"ff0c4c56-fecb-40dd-b2ba-32f444d7c191\") "
Sep 09 04:17:06.530925 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.530901    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0c4c56-fecb-40dd-b2ba-32f444d7c191-kube-api-access-t7pr5" (OuterVolumeSpecName: "kube-api-access-t7pr5") pod "ff0c4c56-fecb-40dd-b2ba-32f444d7c191" (UID: "ff0c4c56-fecb-40dd-b2ba-32f444d7c191"). InnerVolumeSpecName "kube-api-access-t7pr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:17:06.558804 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.558756    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7814-139976933218896"]
Sep 09 04:17:06.559247 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.559229    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff0c4c56-fecb-40dd-b2ba-32f444d7c191" containerName="test-curl-7814-139976933218896"
Sep 09 04:17:06.559330 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.559249    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff0c4c56-fecb-40dd-b2ba-32f444d7c191" containerName="test-curl-7814-139976933218896"
Sep 09 04:17:06.559400 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.559385    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff0c4c56-fecb-40dd-b2ba-32f444d7c191" containerName="test-curl-7814-139976933218896"
Sep 09 04:17:06.562544 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.562525    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:06.568933 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.568906    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7814-139976933218896"]
Sep 09 04:17:06.629856 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.629822    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdwr9\" (UniqueName: \"kubernetes.io/projected/7ea59e0b-a6cf-49e7-af7f-0982ba506781-kube-api-access-zdwr9\") pod \"test-curl-7814-139976933218896\" (UID: \"7ea59e0b-a6cf-49e7-af7f-0982ba506781\") " pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:06.630027 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.630007    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7pr5\" (UniqueName: \"kubernetes.io/projected/ff0c4c56-fecb-40dd-b2ba-32f444d7c191-kube-api-access-t7pr5\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:06.730542 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.730501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdwr9\" (UniqueName: \"kubernetes.io/projected/7ea59e0b-a6cf-49e7-af7f-0982ba506781-kube-api-access-zdwr9\") pod \"test-curl-7814-139976933218896\" (UID: \"7ea59e0b-a6cf-49e7-af7f-0982ba506781\") " pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:06.739978 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.739941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdwr9\" (UniqueName: \"kubernetes.io/projected/7ea59e0b-a6cf-49e7-af7f-0982ba506781-kube-api-access-zdwr9\") pod \"test-curl-7814-139976933218896\" (UID: \"7ea59e0b-a6cf-49e7-af7f-0982ba506781\") " pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:06.873682 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:06.873651    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:07.147546 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:07.147513    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7814-139976933218896"]
Sep 09 04:17:07.148456 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:17:07.148435    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ea59e0b_a6cf_49e7_af7f_0982ba506781.slice/crio-707127a10018f9accb23ca49178e5724929d4f59799d188751b7b183ef6e8a31 WatchSource:0}: Error finding container 707127a10018f9accb23ca49178e5724929d4f59799d188751b7b183ef6e8a31: Status 404 returned error can't find the container with id 707127a10018f9accb23ca49178e5724929d4f59799d188751b7b183ef6e8a31
Sep 09 04:17:07.249431 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:07.249399    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:07.249591 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:07.249403    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c04f17d238dc03c40b5b7de7857e0cef7d9f6425f3e07f63e01e7cbe3efc900"
Sep 09 04:17:07.250677 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:07.250653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7814-139976933218896" event={"ID":"7ea59e0b-a6cf-49e7-af7f-0982ba506781","Type":"ContainerStarted","Data":"707127a10018f9accb23ca49178e5724929d4f59799d188751b7b183ef6e8a31"}
Sep 09 04:17:07.261320 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:07.261294    2580 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7814-139976933218896" oldPodUID="ff0c4c56-fecb-40dd-b2ba-32f444d7c191" podUID="7ea59e0b-a6cf-49e7-af7f-0982ba506781"
Sep 09 04:17:07.595875 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:07.595842    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff0c4c56-fecb-40dd-b2ba-32f444d7c191" path="/var/lib/kubelet/pods/ff0c4c56-fecb-40dd-b2ba-32f444d7c191/volumes"
Sep 09 04:17:09.266098 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:09.266068    2580 generic.go:356] "Generic (PLEG): container finished" podID="7ea59e0b-a6cf-49e7-af7f-0982ba506781" containerID="c342fffcd68d0d684c2d1cb69227116e29b1885a03fd1375146269fb26aadf0c" exitCode=0
Sep 09 04:17:09.266500 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:09.266156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7814-139976933218896" event={"ID":"7ea59e0b-a6cf-49e7-af7f-0982ba506781","Type":"ContainerDied","Data":"c342fffcd68d0d684c2d1cb69227116e29b1885a03fd1375146269fb26aadf0c"}
Sep 09 04:17:10.414650 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.414624    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:10.443679 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.443653    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7814-139976933218896_7ea59e0b-a6cf-49e7-af7f-0982ba506781/test-curl-7814-139976933218896/0.log"
Sep 09 04:17:10.472869 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.472843    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7814-139976933218896"]
Sep 09 04:17:10.476849 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.476826    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7814-139976933218896"]
Sep 09 04:17:10.569721 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.569642    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdwr9\" (UniqueName: \"kubernetes.io/projected/7ea59e0b-a6cf-49e7-af7f-0982ba506781-kube-api-access-zdwr9\") pod \"7ea59e0b-a6cf-49e7-af7f-0982ba506781\" (UID: \"7ea59e0b-a6cf-49e7-af7f-0982ba506781\") "
Sep 09 04:17:10.571860 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.571828    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea59e0b-a6cf-49e7-af7f-0982ba506781-kube-api-access-zdwr9" (OuterVolumeSpecName: "kube-api-access-zdwr9") pod "7ea59e0b-a6cf-49e7-af7f-0982ba506781" (UID: "7ea59e0b-a6cf-49e7-af7f-0982ba506781"). InnerVolumeSpecName "kube-api-access-zdwr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:17:10.603430 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.603402    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7814-139976932438496"]
Sep 09 04:17:10.603884 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.603868    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ea59e0b-a6cf-49e7-af7f-0982ba506781" containerName="test-curl-7814-139976933218896"
Sep 09 04:17:10.603884 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.603884    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ea59e0b-a6cf-49e7-af7f-0982ba506781" containerName="test-curl-7814-139976933218896"
Sep 09 04:17:10.603982 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.603950    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ea59e0b-a6cf-49e7-af7f-0982ba506781" containerName="test-curl-7814-139976933218896"
Sep 09 04:17:10.607043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.607029    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:10.614498 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.614472    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7814-139976932438496"]
Sep 09 04:17:10.670717 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.670678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wws\" (UniqueName: \"kubernetes.io/projected/0871f369-6fb8-46e8-a867-3b623d9f5927-kube-api-access-85wws\") pod \"test-curl-7814-139976932438496\" (UID: \"0871f369-6fb8-46e8-a867-3b623d9f5927\") " pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:10.670884 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.670732    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdwr9\" (UniqueName: \"kubernetes.io/projected/7ea59e0b-a6cf-49e7-af7f-0982ba506781-kube-api-access-zdwr9\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:10.771735 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.771697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wws\" (UniqueName: \"kubernetes.io/projected/0871f369-6fb8-46e8-a867-3b623d9f5927-kube-api-access-85wws\") pod \"test-curl-7814-139976932438496\" (UID: \"0871f369-6fb8-46e8-a867-3b623d9f5927\") " pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:10.782005 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.781982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wws\" (UniqueName: \"kubernetes.io/projected/0871f369-6fb8-46e8-a867-3b623d9f5927-kube-api-access-85wws\") pod \"test-curl-7814-139976932438496\" (UID: \"0871f369-6fb8-46e8-a867-3b623d9f5927\") " pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:10.918893 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:10.918855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:11.195568 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:11.195543    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7814-139976932438496"]
Sep 09 04:17:11.196661 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:17:11.196636    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0871f369_6fb8_46e8_a867_3b623d9f5927.slice/crio-0b08c6a5dbac0c97914013ee6307c89df53c7515709caf8614007e1c817e9e75 WatchSource:0}: Error finding container 0b08c6a5dbac0c97914013ee6307c89df53c7515709caf8614007e1c817e9e75: Status 404 returned error can't find the container with id 0b08c6a5dbac0c97914013ee6307c89df53c7515709caf8614007e1c817e9e75
Sep 09 04:17:11.280090 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:11.280053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7814-139976932438496" event={"ID":"0871f369-6fb8-46e8-a867-3b623d9f5927","Type":"ContainerStarted","Data":"0b08c6a5dbac0c97914013ee6307c89df53c7515709caf8614007e1c817e9e75"}
Sep 09 04:17:11.281429 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:11.281410    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:11.281429 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:11.281418    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="707127a10018f9accb23ca49178e5724929d4f59799d188751b7b183ef6e8a31"
Sep 09 04:17:11.292214 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:17:11.292192    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7814-139976933218896\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="7ea59e0b-a6cf-49e7-af7f-0982ba506781" pod="opendatahub/test-curl-7814-139976933218896"
Sep 09 04:17:11.597196 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:11.597161    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ea59e0b-a6cf-49e7-af7f-0982ba506781" path="/var/lib/kubelet/pods/7ea59e0b-a6cf-49e7-af7f-0982ba506781/volumes"
Sep 09 04:17:12.288407 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:12.288375    2580 generic.go:356] "Generic (PLEG): container finished" podID="0871f369-6fb8-46e8-a867-3b623d9f5927" containerID="c7aa61a0186398250d4c0c06ce92790d0abe983eca2b3afec107567deb483a65" exitCode=0
Sep 09 04:17:12.288579 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:12.288460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7814-139976932438496" event={"ID":"0871f369-6fb8-46e8-a867-3b623d9f5927","Type":"ContainerDied","Data":"c7aa61a0186398250d4c0c06ce92790d0abe983eca2b3afec107567deb483a65"}
Sep 09 04:17:13.436661 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.436631    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:13.460060 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.460037    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7814-139976932438496_0871f369-6fb8-46e8-a867-3b623d9f5927/test-curl-7814-139976932438496/0.log"
Sep 09 04:17:13.487110 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.487063    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7814-139976932438496"]
Sep 09 04:17:13.490925 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.490899    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7814-139976932438496"]
Sep 09 04:17:13.595667 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.595643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85wws\" (UniqueName: \"kubernetes.io/projected/0871f369-6fb8-46e8-a867-3b623d9f5927-kube-api-access-85wws\") pod \"0871f369-6fb8-46e8-a867-3b623d9f5927\" (UID: \"0871f369-6fb8-46e8-a867-3b623d9f5927\") "
Sep 09 04:17:13.597852 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.597825    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0871f369-6fb8-46e8-a867-3b623d9f5927-kube-api-access-85wws" (OuterVolumeSpecName: "kube-api-access-85wws") pod "0871f369-6fb8-46e8-a867-3b623d9f5927" (UID: "0871f369-6fb8-46e8-a867-3b623d9f5927"). InnerVolumeSpecName "kube-api-access-85wws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:17:13.611252 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.611228    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7814-139976932438496"]
Sep 09 04:17:13.611652 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.611640    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0871f369-6fb8-46e8-a867-3b623d9f5927" containerName="test-curl-7814-139976932438496"
Sep 09 04:17:13.611699 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.611654    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0871f369-6fb8-46e8-a867-3b623d9f5927" containerName="test-curl-7814-139976932438496"
Sep 09 04:17:13.611734 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.611722    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0871f369-6fb8-46e8-a867-3b623d9f5927" containerName="test-curl-7814-139976932438496"
Sep 09 04:17:13.614899 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.614883    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:13.620450 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.620428    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7814-139976932438496"]
Sep 09 04:17:13.697457 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.697419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99zrr\" (UniqueName: \"kubernetes.io/projected/7aa4d81e-1e36-4cdc-8c1e-74a191cba983-kube-api-access-99zrr\") pod \"test-curl-7814-139976932438496\" (UID: \"7aa4d81e-1e36-4cdc-8c1e-74a191cba983\") " pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:13.697630 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.697518    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85wws\" (UniqueName: \"kubernetes.io/projected/0871f369-6fb8-46e8-a867-3b623d9f5927-kube-api-access-85wws\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:13.799023 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.798974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99zrr\" (UniqueName: \"kubernetes.io/projected/7aa4d81e-1e36-4cdc-8c1e-74a191cba983-kube-api-access-99zrr\") pod \"test-curl-7814-139976932438496\" (UID: \"7aa4d81e-1e36-4cdc-8c1e-74a191cba983\") " pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:13.810819 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.810758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99zrr\" (UniqueName: \"kubernetes.io/projected/7aa4d81e-1e36-4cdc-8c1e-74a191cba983-kube-api-access-99zrr\") pod \"test-curl-7814-139976932438496\" (UID: \"7aa4d81e-1e36-4cdc-8c1e-74a191cba983\") " pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:13.925680 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:13.925605    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:14.201148 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:14.201122    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7814-139976932438496"]
Sep 09 04:17:14.201512 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:17:14.201492    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7aa4d81e_1e36_4cdc_8c1e_74a191cba983.slice/crio-dc21c3b1a1925bf4da1010315c9c152b44522687ec1d5e5f1e8e25f4dc7ed44c WatchSource:0}: Error finding container dc21c3b1a1925bf4da1010315c9c152b44522687ec1d5e5f1e8e25f4dc7ed44c: Status 404 returned error can't find the container with id dc21c3b1a1925bf4da1010315c9c152b44522687ec1d5e5f1e8e25f4dc7ed44c
Sep 09 04:17:14.302256 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:14.302213    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7814-139976932438496" event={"ID":"7aa4d81e-1e36-4cdc-8c1e-74a191cba983","Type":"ContainerStarted","Data":"dc21c3b1a1925bf4da1010315c9c152b44522687ec1d5e5f1e8e25f4dc7ed44c"}
Sep 09 04:17:14.303802 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:14.303779    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:14.303919 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:14.303815    2580 scope.go:117] "RemoveContainer" containerID="c7aa61a0186398250d4c0c06ce92790d0abe983eca2b3afec107567deb483a65"
Sep 09 04:17:15.312521 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:15.312427    2580 generic.go:356] "Generic (PLEG): container finished" podID="7aa4d81e-1e36-4cdc-8c1e-74a191cba983" containerID="3605db9041aa91deee497192adc16b4d1ae048a6f38b1b29116e5009f630bd61" exitCode=0
Sep 09 04:17:15.312934 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:15.312552    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7814-139976932438496" event={"ID":"7aa4d81e-1e36-4cdc-8c1e-74a191cba983","Type":"ContainerDied","Data":"3605db9041aa91deee497192adc16b4d1ae048a6f38b1b29116e5009f630bd61"}
Sep 09 04:17:15.596353 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:15.596320    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0871f369-6fb8-46e8-a867-3b623d9f5927" path="/var/lib/kubelet/pods/0871f369-6fb8-46e8-a867-3b623d9f5927/volumes"
Sep 09 04:17:16.455860 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:16.455836    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:16.479720 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:16.479689    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7814-139976932438496_7aa4d81e-1e36-4cdc-8c1e-74a191cba983/test-curl-7814-139976932438496/0.log"
Sep 09 04:17:16.506481 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:16.506418    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7814-139976932438496"]
Sep 09 04:17:16.509078 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:16.509055    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7814-139976932438496"]
Sep 09 04:17:16.627022 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:16.626990    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99zrr\" (UniqueName: \"kubernetes.io/projected/7aa4d81e-1e36-4cdc-8c1e-74a191cba983-kube-api-access-99zrr\") pod \"7aa4d81e-1e36-4cdc-8c1e-74a191cba983\" (UID: \"7aa4d81e-1e36-4cdc-8c1e-74a191cba983\") "
Sep 09 04:17:16.629179 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:16.629152    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa4d81e-1e36-4cdc-8c1e-74a191cba983-kube-api-access-99zrr" (OuterVolumeSpecName: "kube-api-access-99zrr") pod "7aa4d81e-1e36-4cdc-8c1e-74a191cba983" (UID: "7aa4d81e-1e36-4cdc-8c1e-74a191cba983"). InnerVolumeSpecName "kube-api-access-99zrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:17:16.728622 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:16.728592    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99zrr\" (UniqueName: \"kubernetes.io/projected/7aa4d81e-1e36-4cdc-8c1e-74a191cba983-kube-api-access-99zrr\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:17.327578 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:17.327548    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:17.327578 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:17.327577    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc21c3b1a1925bf4da1010315c9c152b44522687ec1d5e5f1e8e25f4dc7ed44c"
Sep 09 04:17:17.339423 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:17:17.339392    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7814-139976932438496\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="7aa4d81e-1e36-4cdc-8c1e-74a191cba983" pod="opendatahub/test-curl-7814-139976932438496"
Sep 09 04:17:17.596308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:17.596274    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa4d81e-1e36-4cdc-8c1e-74a191cba983" path="/var/lib/kubelet/pods/7aa4d81e-1e36-4cdc-8c1e-74a191cba983/volumes"
Sep 09 04:17:18.877637 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:18.877606    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"]
Sep 09 04:17:18.878113 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:18.877906    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn" podUID="6ca8dc88-5e8b-423b-b190-3500de2bfae4" containerName="istio-proxy" containerID="cri-o://d77b5284bd50cb73c44eedeed9578527d188146aafab77649591681a6f43ac3b" gracePeriod=30
Sep 09 04:17:22.282062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.281979    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"]
Sep 09 04:17:22.282438 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.282424    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa4d81e-1e36-4cdc-8c1e-74a191cba983" containerName="test-curl-7814-139976932438496"
Sep 09 04:17:22.282506 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.282441    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa4d81e-1e36-4cdc-8c1e-74a191cba983" containerName="test-curl-7814-139976932438496"
Sep 09 04:17:22.282564 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.282554    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa4d81e-1e36-4cdc-8c1e-74a191cba983" containerName="test-curl-7814-139976932438496"
Sep 09 04:17:22.286322 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.286301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:22.289068 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.289046    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-429d71ed-serving-cert\""
Sep 09 04:17:22.300469 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.300444    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"]
Sep 09 04:17:22.385609 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.385569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-maas-api-tls\") pod \"maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:22.386053 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.386024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkb5s\" (UniqueName: \"kubernetes.io/projected/926546b5-812b-43c5-8d8b-46d0af87b1d5-kube-api-access-xkb5s\") pod \"maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:22.386379 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.386355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/926546b5-812b-43c5-8d8b-46d0af87b1d5-openshift-service-ca\") pod \"maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:22.386494 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.386413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-metrics-certs\") pod \"maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:22.487723 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.487685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkb5s\" (UniqueName: \"kubernetes.io/projected/926546b5-812b-43c5-8d8b-46d0af87b1d5-kube-api-access-xkb5s\") pod \"maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:22.487937 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.487790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/926546b5-812b-43c5-8d8b-46d0af87b1d5-openshift-service-ca\") pod \"maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:22.487937 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.487821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-metrics-certs\") pod \"maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:22.487937 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.487850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-maas-api-tls\") pod \"maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:22.488047 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:17:22.487963    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-serving-cert: secret "maas-api-e2e-mt-429d71ed-serving-cert" not found
Sep 09 04:17:22.488131 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:17:22.488048    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-maas-api-tls podName:926546b5-812b-43c5-8d8b-46d0af87b1d5 nodeName:}" failed. No retries permitted until 2026-09-09 04:17:22.988027486 +0000 UTC m=+1163.937718122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-maas-api-tls") pod "maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj" (UID: "926546b5-812b-43c5-8d8b-46d0af87b1d5") : secret "maas-api-e2e-mt-429d71ed-serving-cert" not found
Sep 09 04:17:22.488631 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.488608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/926546b5-812b-43c5-8d8b-46d0af87b1d5-openshift-service-ca\") pod \"maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:22.490544 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.490523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-metrics-certs\") pod \"maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:22.502379 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.502346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkb5s\" (UniqueName: \"kubernetes.io/projected/926546b5-812b-43c5-8d8b-46d0af87b1d5-kube-api-access-xkb5s\") pod \"maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:22.994315 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.994279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-maas-api-tls\") pod \"maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:22.997123 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:22.997097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-maas-api-tls\") pod \"maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:23.198377 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:23.198342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:23.481374 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:23.481348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"]
Sep 09 04:17:23.482220 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:17:23.482189    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926546b5_812b_43c5_8d8b_46d0af87b1d5.slice/crio-b94543ae8a06f89a97083af8c1e8bb019145e30b53a159e1c60724e76818e053 WatchSource:0}: Error finding container b94543ae8a06f89a97083af8c1e8bb019145e30b53a159e1c60724e76818e053: Status 404 returned error can't find the container with id b94543ae8a06f89a97083af8c1e8bb019145e30b53a159e1c60724e76818e053
Sep 09 04:17:24.149866 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.149808    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:17:24.309273 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309242    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-envoy\") pod \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") "
Sep 09 04:17:24.309459 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309300    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-workload-socket\") pod \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") "
Sep 09 04:17:24.309459 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istiod-ca-cert\") pod \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") "
Sep 09 04:17:24.309459 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309369    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-workload-certs\") pod \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") "
Sep 09 04:17:24.309459 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309435    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnq82\" (UniqueName: \"kubernetes.io/projected/6ca8dc88-5e8b-423b-b190-3500de2bfae4-kube-api-access-lnq82\") pod \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") "
Sep 09 04:17:24.309691 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309463    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-podinfo\") pod \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") "
Sep 09 04:17:24.309691 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309489    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-data\") pod \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") "
Sep 09 04:17:24.309691 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309532    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-token\") pod \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") "
Sep 09 04:17:24.309691 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309556    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-credential-socket\") pod \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\" (UID: \"6ca8dc88-5e8b-423b-b190-3500de2bfae4\") "
Sep 09 04:17:24.309691 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ca8dc88-5e8b-423b-b190-3500de2bfae4" (UID: "6ca8dc88-5e8b-423b-b190-3500de2bfae4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:24.309984 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309728    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ca8dc88-5e8b-423b-b190-3500de2bfae4" (UID: "6ca8dc88-5e8b-423b-b190-3500de2bfae4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:17:24.309984 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309758    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ca8dc88-5e8b-423b-b190-3500de2bfae4" (UID: "6ca8dc88-5e8b-423b-b190-3500de2bfae4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:24.309984 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309916    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ca8dc88-5e8b-423b-b190-3500de2bfae4" (UID: "6ca8dc88-5e8b-423b-b190-3500de2bfae4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:24.310147 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.309999    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-workload-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:24.310147 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.310016    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istiod-ca-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:24.310147 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.310025    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-workload-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:24.310147 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.310034    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-credential-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:24.310341 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.310223    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ca8dc88-5e8b-423b-b190-3500de2bfae4" (UID: "6ca8dc88-5e8b-423b-b190-3500de2bfae4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:24.312243 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.312142    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ca8dc88-5e8b-423b-b190-3500de2bfae4" (UID: "6ca8dc88-5e8b-423b-b190-3500de2bfae4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:17:24.312243 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.312214    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ca8dc88-5e8b-423b-b190-3500de2bfae4" (UID: "6ca8dc88-5e8b-423b-b190-3500de2bfae4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:17:24.312397 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.312379    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca8dc88-5e8b-423b-b190-3500de2bfae4-kube-api-access-lnq82" (OuterVolumeSpecName: "kube-api-access-lnq82") pod "6ca8dc88-5e8b-423b-b190-3500de2bfae4" (UID: "6ca8dc88-5e8b-423b-b190-3500de2bfae4"). InnerVolumeSpecName "kube-api-access-lnq82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:17:24.312439 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.312380    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ca8dc88-5e8b-423b-b190-3500de2bfae4" (UID: "6ca8dc88-5e8b-423b-b190-3500de2bfae4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:17:24.378032 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.377995    2580 generic.go:356] "Generic (PLEG): container finished" podID="6ca8dc88-5e8b-423b-b190-3500de2bfae4" containerID="d77b5284bd50cb73c44eedeed9578527d188146aafab77649591681a6f43ac3b" exitCode=0
Sep 09 04:17:24.378200 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.378036    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"
Sep 09 04:17:24.378200 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.378072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn" event={"ID":"6ca8dc88-5e8b-423b-b190-3500de2bfae4","Type":"ContainerDied","Data":"d77b5284bd50cb73c44eedeed9578527d188146aafab77649591681a6f43ac3b"}
Sep 09 04:17:24.378200 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.378131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn" event={"ID":"6ca8dc88-5e8b-423b-b190-3500de2bfae4","Type":"ContainerDied","Data":"5659e5a05f23c4f053eb901bc079baaf387237dbec3a0e972a09ab3d888e6416"}
Sep 09 04:17:24.378200 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.378153    2580 scope.go:117] "RemoveContainer" containerID="d77b5284bd50cb73c44eedeed9578527d188146aafab77649591681a6f43ac3b"
Sep 09 04:17:24.380070 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.380046    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj" event={"ID":"926546b5-812b-43c5-8d8b-46d0af87b1d5","Type":"ContainerStarted","Data":"9fac7b01f2f05508a5d6e3a2945b9aebf15f8f5b743c78f35227654037b44114"}
Sep 09 04:17:24.380180 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.380095    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj" event={"ID":"926546b5-812b-43c5-8d8b-46d0af87b1d5","Type":"ContainerStarted","Data":"b94543ae8a06f89a97083af8c1e8bb019145e30b53a159e1c60724e76818e053"}
Sep 09 04:17:24.380249 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.380209    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:24.389686 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.389646    2580 scope.go:117] "RemoveContainer" containerID="d77b5284bd50cb73c44eedeed9578527d188146aafab77649591681a6f43ac3b"
Sep 09 04:17:24.389981 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:17:24.389963    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d77b5284bd50cb73c44eedeed9578527d188146aafab77649591681a6f43ac3b\": container with ID starting with d77b5284bd50cb73c44eedeed9578527d188146aafab77649591681a6f43ac3b not found: ID does not exist" containerID="d77b5284bd50cb73c44eedeed9578527d188146aafab77649591681a6f43ac3b"
Sep 09 04:17:24.390047 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.389990    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d77b5284bd50cb73c44eedeed9578527d188146aafab77649591681a6f43ac3b"} err="failed to get container status \"d77b5284bd50cb73c44eedeed9578527d188146aafab77649591681a6f43ac3b\": rpc error: code = NotFound desc = could not find container \"d77b5284bd50cb73c44eedeed9578527d188146aafab77649591681a6f43ac3b\": container with ID starting with d77b5284bd50cb73c44eedeed9578527d188146aafab77649591681a6f43ac3b not found: ID does not exist"
Sep 09 04:17:24.408619 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.408518    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj" podStartSLOduration=1.883242057 podStartE2EDuration="2.408497842s" totalImagesPullingTime="525.255785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:17:24.404067868 +0000 UTC m=+1165.353758523" watchObservedRunningTime="2026-09-09 04:17:24.408497842 +0000 UTC m=+1165.358188497"
Sep 09 04:17:24.410686 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.410662    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnq82\" (UniqueName: \"kubernetes.io/projected/6ca8dc88-5e8b-423b-b190-3500de2bfae4-kube-api-access-lnq82\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:24.410813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.410687    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-podinfo\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:24.410813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.410702    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-data\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:24.410813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.410715    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:24.410813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.410726    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca8dc88-5e8b-423b-b190-3500de2bfae4-istio-envoy\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:24.423844 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.423808    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"]
Sep 09 04:17:24.427495 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:24.427469    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-1215c1-openshift-default-7f75654d86-lh4xn"]
Sep 09 04:17:25.597482 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:25.597440    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ca8dc88-5e8b-423b-b190-3500de2bfae4" path="/var/lib/kubelet/pods/6ca8dc88-5e8b-423b-b190-3500de2bfae4/volumes"
Sep 09 04:17:30.394033 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:30.394003    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:17:45.793536 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:45.793471    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"]
Sep 09 04:17:45.794117 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:45.793735    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx" podUID="59ed0e1c-9b27-40e4-9b3c-a235ba8044ae" containerName="maas-api" containerID="cri-o://cf778f6a9dfcc22047d13d23370860190526217e3c500f187166c4565e1af742" gracePeriod=30
Sep 09 04:17:46.064736 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.064711    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:17:46.122107 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.122075    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-metrics-certs\") pod \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") "
Sep 09 04:17:46.122316 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.122116    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-maas-api-tls\") pod \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") "
Sep 09 04:17:46.122316 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.122228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-openshift-service-ca\") pod \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") "
Sep 09 04:17:46.122316 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.122262    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlc2w\" (UniqueName: \"kubernetes.io/projected/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-kube-api-access-rlc2w\") pod \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\" (UID: \"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae\") "
Sep 09 04:17:46.122808 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.122757    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59ed0e1c-9b27-40e4-9b3c-a235ba8044ae" (UID: "59ed0e1c-9b27-40e4-9b3c-a235ba8044ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:17:46.125315 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.125282    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-kube-api-access-rlc2w" (OuterVolumeSpecName: "kube-api-access-rlc2w") pod "59ed0e1c-9b27-40e4-9b3c-a235ba8044ae" (UID: "59ed0e1c-9b27-40e4-9b3c-a235ba8044ae"). InnerVolumeSpecName "kube-api-access-rlc2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:17:46.125560 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.125533    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59ed0e1c-9b27-40e4-9b3c-a235ba8044ae" (UID: "59ed0e1c-9b27-40e4-9b3c-a235ba8044ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:17:46.126385 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.126365    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "59ed0e1c-9b27-40e4-9b3c-a235ba8044ae" (UID: "59ed0e1c-9b27-40e4-9b3c-a235ba8044ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:17:46.223541 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.223502    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-openshift-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:46.223541 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.223544    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlc2w\" (UniqueName: \"kubernetes.io/projected/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-kube-api-access-rlc2w\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:46.223744 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.223560    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:46.223744 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.223573    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae-maas-api-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:17:46.523631 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.523598    2580 generic.go:356] "Generic (PLEG): container finished" podID="59ed0e1c-9b27-40e4-9b3c-a235ba8044ae" containerID="cf778f6a9dfcc22047d13d23370860190526217e3c500f187166c4565e1af742" exitCode=0
Sep 09 04:17:46.523834 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.523650    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"
Sep 09 04:17:46.523834 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.523687    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx" event={"ID":"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae","Type":"ContainerDied","Data":"cf778f6a9dfcc22047d13d23370860190526217e3c500f187166c4565e1af742"}
Sep 09 04:17:46.523834 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.523744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx" event={"ID":"59ed0e1c-9b27-40e4-9b3c-a235ba8044ae","Type":"ContainerDied","Data":"8c34e025e1eaf6fa1ff6600b48b39e863cc9f9b95fab15a21fb7cbc820016895"}
Sep 09 04:17:46.523834 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.523798    2580 scope.go:117] "RemoveContainer" containerID="cf778f6a9dfcc22047d13d23370860190526217e3c500f187166c4565e1af742"
Sep 09 04:17:46.534110 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.534090    2580 scope.go:117] "RemoveContainer" containerID="cf778f6a9dfcc22047d13d23370860190526217e3c500f187166c4565e1af742"
Sep 09 04:17:46.534384 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:17:46.534364    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf778f6a9dfcc22047d13d23370860190526217e3c500f187166c4565e1af742\": container with ID starting with cf778f6a9dfcc22047d13d23370860190526217e3c500f187166c4565e1af742 not found: ID does not exist" containerID="cf778f6a9dfcc22047d13d23370860190526217e3c500f187166c4565e1af742"
Sep 09 04:17:46.534457 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.534392    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf778f6a9dfcc22047d13d23370860190526217e3c500f187166c4565e1af742"} err="failed to get container status \"cf778f6a9dfcc22047d13d23370860190526217e3c500f187166c4565e1af742\": rpc error: code = NotFound desc = could not find container \"cf778f6a9dfcc22047d13d23370860190526217e3c500f187166c4565e1af742\": container with ID starting with cf778f6a9dfcc22047d13d23370860190526217e3c500f187166c4565e1af742 not found: ID does not exist"
Sep 09 04:17:46.550435 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.550403    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"]
Sep 09 04:17:46.555697 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:46.555671    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-359adf-85489bd9c-4cljx"]
Sep 09 04:17:47.595451 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:47.595421    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59ed0e1c-9b27-40e4-9b3c-a235ba8044ae" path="/var/lib/kubelet/pods/59ed0e1c-9b27-40e4-9b3c-a235ba8044ae/volumes"
Sep 09 04:17:59.611346 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:59.611315    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:17:59.616144 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:17:59.616122    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:18:09.197634 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.197594    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"]
Sep 09 04:18:09.198122 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.197877    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth" podUID="d1d89aea-10df-4903-8420-375251c9b197" containerName="maas-api" containerID="cri-o://bd59f34fc2ae14968c49a725c1d4e3fa21bdab91cec8923bf25df73dc80448d8" gracePeriod=30
Sep 09 04:18:09.230000 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.229970    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn"]
Sep 09 04:18:09.232601 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.232576    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-629522-29815455-x5zjn"]
Sep 09 04:18:09.449978 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.449920    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:18:09.544883 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.544851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-metrics-certs\") pod \"d1d89aea-10df-4903-8420-375251c9b197\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") "
Sep 09 04:18:09.545069 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.544899    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d89aea-10df-4903-8420-375251c9b197-openshift-service-ca\") pod \"d1d89aea-10df-4903-8420-375251c9b197\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") "
Sep 09 04:18:09.545069 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.544933    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bscrc\" (UniqueName: \"kubernetes.io/projected/d1d89aea-10df-4903-8420-375251c9b197-kube-api-access-bscrc\") pod \"d1d89aea-10df-4903-8420-375251c9b197\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") "
Sep 09 04:18:09.545069 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.544955    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-maas-api-tls\") pod \"d1d89aea-10df-4903-8420-375251c9b197\" (UID: \"d1d89aea-10df-4903-8420-375251c9b197\") "
Sep 09 04:18:09.545348 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.545316    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d89aea-10df-4903-8420-375251c9b197-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d1d89aea-10df-4903-8420-375251c9b197" (UID: "d1d89aea-10df-4903-8420-375251c9b197"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:18:09.547174 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.547150    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d1d89aea-10df-4903-8420-375251c9b197" (UID: "d1d89aea-10df-4903-8420-375251c9b197"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:18:09.547284 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.547171    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d1d89aea-10df-4903-8420-375251c9b197" (UID: "d1d89aea-10df-4903-8420-375251c9b197"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:18:09.547611 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.547577    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d89aea-10df-4903-8420-375251c9b197-kube-api-access-bscrc" (OuterVolumeSpecName: "kube-api-access-bscrc") pod "d1d89aea-10df-4903-8420-375251c9b197" (UID: "d1d89aea-10df-4903-8420-375251c9b197"). InnerVolumeSpecName "kube-api-access-bscrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:18:09.600964 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.600926    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3f9745c-25ab-49bd-b23a-68c85b8ecabd" path="/var/lib/kubelet/pods/a3f9745c-25ab-49bd-b23a-68c85b8ecabd/volumes"
Sep 09 04:18:09.645748 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.645705    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bscrc\" (UniqueName: \"kubernetes.io/projected/d1d89aea-10df-4903-8420-375251c9b197-kube-api-access-bscrc\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:18:09.645748 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.645742    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-maas-api-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:18:09.645748 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.645754    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1d89aea-10df-4903-8420-375251c9b197-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:18:09.645748 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.645779    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d89aea-10df-4903-8420-375251c9b197-openshift-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:18:09.675325 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.675293    2580 generic.go:356] "Generic (PLEG): container finished" podID="d1d89aea-10df-4903-8420-375251c9b197" containerID="bd59f34fc2ae14968c49a725c1d4e3fa21bdab91cec8923bf25df73dc80448d8" exitCode=0
Sep 09 04:18:09.675451 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.675339    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"
Sep 09 04:18:09.675451 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.675378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth" event={"ID":"d1d89aea-10df-4903-8420-375251c9b197","Type":"ContainerDied","Data":"bd59f34fc2ae14968c49a725c1d4e3fa21bdab91cec8923bf25df73dc80448d8"}
Sep 09 04:18:09.675451 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.675427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth" event={"ID":"d1d89aea-10df-4903-8420-375251c9b197","Type":"ContainerDied","Data":"a262d4f026bfacb7228fffcf5f7f7ef6be7ace4d507379baf26c32cee57b2c3d"}
Sep 09 04:18:09.675451 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.675450    2580 scope.go:117] "RemoveContainer" containerID="bd59f34fc2ae14968c49a725c1d4e3fa21bdab91cec8923bf25df73dc80448d8"
Sep 09 04:18:09.685094 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.685077    2580 scope.go:117] "RemoveContainer" containerID="bd59f34fc2ae14968c49a725c1d4e3fa21bdab91cec8923bf25df73dc80448d8"
Sep 09 04:18:09.685371 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:18:09.685353    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd59f34fc2ae14968c49a725c1d4e3fa21bdab91cec8923bf25df73dc80448d8\": container with ID starting with bd59f34fc2ae14968c49a725c1d4e3fa21bdab91cec8923bf25df73dc80448d8 not found: ID does not exist" containerID="bd59f34fc2ae14968c49a725c1d4e3fa21bdab91cec8923bf25df73dc80448d8"
Sep 09 04:18:09.685448 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.685387    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd59f34fc2ae14968c49a725c1d4e3fa21bdab91cec8923bf25df73dc80448d8"} err="failed to get container status \"bd59f34fc2ae14968c49a725c1d4e3fa21bdab91cec8923bf25df73dc80448d8\": rpc error: code = NotFound desc = could not find container \"bd59f34fc2ae14968c49a725c1d4e3fa21bdab91cec8923bf25df73dc80448d8\": container with ID starting with bd59f34fc2ae14968c49a725c1d4e3fa21bdab91cec8923bf25df73dc80448d8 not found: ID does not exist"
Sep 09 04:18:09.694357 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.694335    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"]
Sep 09 04:18:09.698034 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:09.698016    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-629522-7df77977db-8pnth"]
Sep 09 04:18:11.596569 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:11.596533    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1d89aea-10df-4903-8420-375251c9b197" path="/var/lib/kubelet/pods/d1d89aea-10df-4903-8420-375251c9b197/volumes"
Sep 09 04:18:16.413981 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:16.413946    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"]
Sep 09 04:18:16.414497 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:16.414188    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy" containerID="cri-o://62eff0db88dd7a559efa3c9c1b457c91e05e38ab37909f0cda5ae6d80b661243" gracePeriod=30
Sep 09 04:18:17.366727 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.366691    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"]
Sep 09 04:18:17.367123 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.367108    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ed0e1c-9b27-40e4-9b3c-a235ba8044ae" containerName="maas-api"
Sep 09 04:18:17.367189 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.367124    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ed0e1c-9b27-40e4-9b3c-a235ba8044ae" containerName="maas-api"
Sep 09 04:18:17.367189 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.367140    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d89aea-10df-4903-8420-375251c9b197" containerName="maas-api"
Sep 09 04:18:17.367189 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.367146    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d89aea-10df-4903-8420-375251c9b197" containerName="maas-api"
Sep 09 04:18:17.367189 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.367159    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca8dc88-5e8b-423b-b190-3500de2bfae4" containerName="istio-proxy"
Sep 09 04:18:17.367189 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.367164    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca8dc88-5e8b-423b-b190-3500de2bfae4" containerName="istio-proxy"
Sep 09 04:18:17.367440 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.367241    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d89aea-10df-4903-8420-375251c9b197" containerName="maas-api"
Sep 09 04:18:17.367440 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.367251    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ed0e1c-9b27-40e4-9b3c-a235ba8044ae" containerName="maas-api"
Sep 09 04:18:17.367440 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.367260    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca8dc88-5e8b-423b-b190-3500de2bfae4" containerName="istio-proxy"
Sep 09 04:18:17.371751 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.371733    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.375968 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.375943    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-426183-openshift-default-dockercfg-vl7fd\""
Sep 09 04:18:17.401329 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.401306    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"]
Sep 09 04:18:17.418344 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.418241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79sv6\" (UniqueName: \"kubernetes.io/projected/0793bc60-b1ec-46bd-99a5-828a08edd26f-kube-api-access-79sv6\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.418813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.418480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-token\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.418813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.418523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-credential-socket\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.418813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.418550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-data\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.418813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.418623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-podinfo\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.418813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.418657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-workload-certs\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.418813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.418706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-envoy\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.418813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.418753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-workload-socket\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.418813 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.418804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0793bc60-b1ec-46bd-99a5-828a08edd26f-istiod-ca-cert\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.520217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.520178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-podinfo\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.520217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.520220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-workload-certs\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.520475 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.520247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-envoy\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.520475 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.520284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-workload-socket\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.520475 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.520306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0793bc60-b1ec-46bd-99a5-828a08edd26f-istiod-ca-cert\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.520475 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.520343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79sv6\" (UniqueName: \"kubernetes.io/projected/0793bc60-b1ec-46bd-99a5-828a08edd26f-kube-api-access-79sv6\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.520475 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.520410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-token\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.520475 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.520438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-credential-socket\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.520475 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.520459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-data\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.520924 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.520666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-workload-certs\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.520924 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.520675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-workload-socket\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.520924 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.520840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-data\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.521113 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.520963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-credential-socket\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.521285 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.521252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0793bc60-b1ec-46bd-99a5-828a08edd26f-istiod-ca-cert\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.522808 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.522784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-envoy\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.523221 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.523199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-podinfo\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.535677 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.535650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-token\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.535777 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.535697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79sv6\" (UniqueName: \"kubernetes.io/projected/0793bc60-b1ec-46bd-99a5-828a08edd26f-kube-api-access-79sv6\") pod \"e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") " pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.681956 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.681867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:17.966316 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.966289    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"]
Sep 09 04:18:17.967386 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:18:17.967359    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0793bc60_b1ec_46bd_99a5_828a08edd26f.slice/crio-f939f72afbb82a0ec287880d4f66c2cb4397f1ce97ec9aa4c292ec55c1ba9827 WatchSource:0}: Error finding container f939f72afbb82a0ec287880d4f66c2cb4397f1ce97ec9aa4c292ec55c1ba9827: Status 404 returned error can't find the container with id f939f72afbb82a0ec287880d4f66c2cb4397f1ce97ec9aa4c292ec55c1ba9827
Sep 09 04:18:17.969625 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.969588    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:18:17.969707 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.969652    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:18:17.969707 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:17.969679    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:18:18.737098 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:18.737057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" event={"ID":"0793bc60-b1ec-46bd-99a5-828a08edd26f","Type":"ContainerStarted","Data":"61ad871ea48f1a674d788c4f2870aec3c2a0a8fd04ddd76101b4e95af3e09af4"}
Sep 09 04:18:18.737574 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:18.737146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" event={"ID":"0793bc60-b1ec-46bd-99a5-828a08edd26f","Type":"ContainerStarted","Data":"f939f72afbb82a0ec287880d4f66c2cb4397f1ce97ec9aa4c292ec55c1ba9827"}
Sep 09 04:18:18.760757 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:18.760706    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" podStartSLOduration=1.7606915029999999 podStartE2EDuration="1.760691503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:18:18.759091053 +0000 UTC m=+1219.708781710" watchObservedRunningTime="2026-09-09 04:18:18.760691503 +0000 UTC m=+1219.710382157"
Sep 09 04:18:19.682545 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:19.682513    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:19.688209 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:19.688184    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:19.742706 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:19.742677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:19.744026 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:19.744010    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:19.826141 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:19.826100    2580 patch_prober.go:28] interesting pod/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 04:18:19.826325 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:19.826160    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 04:18:21.665166 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.665142    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:18:21.757423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.757334    2580 generic.go:356] "Generic (PLEG): container finished" podID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerID="62eff0db88dd7a559efa3c9c1b457c91e05e38ab37909f0cda5ae6d80b661243" exitCode=0
Sep 09 04:18:21.757423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.757389    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"
Sep 09 04:18:21.757602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.757425    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" event={"ID":"768ee68e-ff97-4f05-8ac6-df6d464e5088","Type":"ContainerDied","Data":"62eff0db88dd7a559efa3c9c1b457c91e05e38ab37909f0cda5ae6d80b661243"}
Sep 09 04:18:21.757602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.757482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr" event={"ID":"768ee68e-ff97-4f05-8ac6-df6d464e5088","Type":"ContainerDied","Data":"81402966091ad942e990070fefb2e171187c834e4c6631c3c255596956048a8e"}
Sep 09 04:18:21.757602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.757506    2580 scope.go:117] "RemoveContainer" containerID="62eff0db88dd7a559efa3c9c1b457c91e05e38ab37909f0cda5ae6d80b661243"
Sep 09 04:18:21.762100 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762076    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-data\") pod \"768ee68e-ff97-4f05-8ac6-df6d464e5088\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") "
Sep 09 04:18:21.762231 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762128    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74pxr\" (UniqueName: \"kubernetes.io/projected/768ee68e-ff97-4f05-8ac6-df6d464e5088-kube-api-access-74pxr\") pod \"768ee68e-ff97-4f05-8ac6-df6d464e5088\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") "
Sep 09 04:18:21.762231 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762161    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-credential-socket\") pod \"768ee68e-ff97-4f05-8ac6-df6d464e5088\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") "
Sep 09 04:18:21.762231 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762200    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-workload-socket\") pod \"768ee68e-ff97-4f05-8ac6-df6d464e5088\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") "
Sep 09 04:18:21.762231 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768ee68e-ff97-4f05-8ac6-df6d464e5088-istiod-ca-cert\") pod \"768ee68e-ff97-4f05-8ac6-df6d464e5088\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") "
Sep 09 04:18:21.762489 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762285    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-envoy\") pod \"768ee68e-ff97-4f05-8ac6-df6d464e5088\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") "
Sep 09 04:18:21.762489 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762311    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-podinfo\") pod \"768ee68e-ff97-4f05-8ac6-df6d464e5088\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") "
Sep 09 04:18:21.762489 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-token\") pod \"768ee68e-ff97-4f05-8ac6-df6d464e5088\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") "
Sep 09 04:18:21.762489 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762398    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-workload-certs\") pod \"768ee68e-ff97-4f05-8ac6-df6d464e5088\" (UID: \"768ee68e-ff97-4f05-8ac6-df6d464e5088\") "
Sep 09 04:18:21.762489 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762454    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "768ee68e-ff97-4f05-8ac6-df6d464e5088" (UID: "768ee68e-ff97-4f05-8ac6-df6d464e5088"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:18:21.762755 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762511    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-data" (OuterVolumeSpecName: "istio-data") pod "768ee68e-ff97-4f05-8ac6-df6d464e5088" (UID: "768ee68e-ff97-4f05-8ac6-df6d464e5088"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:18:21.762755 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762539    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "768ee68e-ff97-4f05-8ac6-df6d464e5088" (UID: "768ee68e-ff97-4f05-8ac6-df6d464e5088"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:18:21.762755 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762727    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "768ee68e-ff97-4f05-8ac6-df6d464e5088" (UID: "768ee68e-ff97-4f05-8ac6-df6d464e5088"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:18:21.762951 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762883    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-data\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:18:21.762951 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762902    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-credential-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:18:21.762951 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762918    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-workload-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:18:21.762951 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.762933    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-workload-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:18:21.763319 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.763020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768ee68e-ff97-4f05-8ac6-df6d464e5088-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "768ee68e-ff97-4f05-8ac6-df6d464e5088" (UID: "768ee68e-ff97-4f05-8ac6-df6d464e5088"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:18:21.764814 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.764793    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "768ee68e-ff97-4f05-8ac6-df6d464e5088" (UID: "768ee68e-ff97-4f05-8ac6-df6d464e5088"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:18:21.765117 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.765089    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-token" (OuterVolumeSpecName: "istio-token") pod "768ee68e-ff97-4f05-8ac6-df6d464e5088" (UID: "768ee68e-ff97-4f05-8ac6-df6d464e5088"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:18:21.765257 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.765240    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "768ee68e-ff97-4f05-8ac6-df6d464e5088" (UID: "768ee68e-ff97-4f05-8ac6-df6d464e5088"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:18:21.765319 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.765261    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768ee68e-ff97-4f05-8ac6-df6d464e5088-kube-api-access-74pxr" (OuterVolumeSpecName: "kube-api-access-74pxr") pod "768ee68e-ff97-4f05-8ac6-df6d464e5088" (UID: "768ee68e-ff97-4f05-8ac6-df6d464e5088"). InnerVolumeSpecName "kube-api-access-74pxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:18:21.769997 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.769979    2580 scope.go:117] "RemoveContainer" containerID="6462eb5fefef8651ed109c98f3f1b81db3758c463c11d5f9c3005f4208360905"
Sep 09 04:18:21.787202 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.787180    2580 scope.go:117] "RemoveContainer" containerID="62eff0db88dd7a559efa3c9c1b457c91e05e38ab37909f0cda5ae6d80b661243"
Sep 09 04:18:21.787463 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:18:21.787445    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62eff0db88dd7a559efa3c9c1b457c91e05e38ab37909f0cda5ae6d80b661243\": container with ID starting with 62eff0db88dd7a559efa3c9c1b457c91e05e38ab37909f0cda5ae6d80b661243 not found: ID does not exist" containerID="62eff0db88dd7a559efa3c9c1b457c91e05e38ab37909f0cda5ae6d80b661243"
Sep 09 04:18:21.787510 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.787471    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62eff0db88dd7a559efa3c9c1b457c91e05e38ab37909f0cda5ae6d80b661243"} err="failed to get container status \"62eff0db88dd7a559efa3c9c1b457c91e05e38ab37909f0cda5ae6d80b661243\": rpc error: code = NotFound desc = could not find container \"62eff0db88dd7a559efa3c9c1b457c91e05e38ab37909f0cda5ae6d80b661243\": container with ID starting with 62eff0db88dd7a559efa3c9c1b457c91e05e38ab37909f0cda5ae6d80b661243 not found: ID does not exist"
Sep 09 04:18:21.787510 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.787488    2580 scope.go:117] "RemoveContainer" containerID="6462eb5fefef8651ed109c98f3f1b81db3758c463c11d5f9c3005f4208360905"
Sep 09 04:18:21.787747 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:18:21.787728    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6462eb5fefef8651ed109c98f3f1b81db3758c463c11d5f9c3005f4208360905\": container with ID starting with 6462eb5fefef8651ed109c98f3f1b81db3758c463c11d5f9c3005f4208360905 not found: ID does not exist" containerID="6462eb5fefef8651ed109c98f3f1b81db3758c463c11d5f9c3005f4208360905"
Sep 09 04:18:21.787834 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.787751    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6462eb5fefef8651ed109c98f3f1b81db3758c463c11d5f9c3005f4208360905"} err="failed to get container status \"6462eb5fefef8651ed109c98f3f1b81db3758c463c11d5f9c3005f4208360905\": rpc error: code = NotFound desc = could not find container \"6462eb5fefef8651ed109c98f3f1b81db3758c463c11d5f9c3005f4208360905\": container with ID starting with 6462eb5fefef8651ed109c98f3f1b81db3758c463c11d5f9c3005f4208360905 not found: ID does not exist"
Sep 09 04:18:21.864013 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.863976    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74pxr\" (UniqueName: \"kubernetes.io/projected/768ee68e-ff97-4f05-8ac6-df6d464e5088-kube-api-access-74pxr\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:18:21.864013 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.864010    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768ee68e-ff97-4f05-8ac6-df6d464e5088-istiod-ca-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:18:21.864209 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.864026    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-envoy\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:18:21.864209 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.864039    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-podinfo\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:18:21.864209 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:21.864051    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768ee68e-ff97-4f05-8ac6-df6d464e5088-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:18:22.010864 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.010785    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"]
Sep 09 04:18:22.011455 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.011425    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy"
Sep 09 04:18:22.011455 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.011445    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy"
Sep 09 04:18:22.011627 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.011466    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy"
Sep 09 04:18:22.011627 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.011475    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy"
Sep 09 04:18:22.011627 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.011595    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy"
Sep 09 04:18:22.016741 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.016717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.020365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.020294    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-426183-serving-cert\""
Sep 09 04:18:22.027846 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.026673    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"]
Sep 09 04:18:22.066035 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.066004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95bs4\" (UniqueName: \"kubernetes.io/projected/c5051f50-3e61-4b4f-9358-be04824a254c-kube-api-access-95bs4\") pod \"maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.066181 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.066054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.066235 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.066218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.066273 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.066258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5051f50-3e61-4b4f-9358-be04824a254c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.088044 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.088011    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"]
Sep 09 04:18:22.098336 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.098307    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-629522-openshift-default-c75f574d8-cfvcr"]
Sep 09 04:18:22.167379 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.167339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.167566 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.167392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5051f50-3e61-4b4f-9358-be04824a254c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.167566 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.167450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95bs4\" (UniqueName: \"kubernetes.io/projected/c5051f50-3e61-4b4f-9358-be04824a254c-kube-api-access-95bs4\") pod \"maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.167566 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.167469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.167699 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:18:22.167580    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-serving-cert: secret "maas-api-e2e-shared-a-w3-426183-serving-cert" not found
Sep 09 04:18:22.167699 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:18:22.167649    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-maas-api-tls podName:c5051f50-3e61-4b4f-9358-be04824a254c nodeName:}" failed. No retries permitted until 2026-09-09 04:18:22.667627722 +0000 UTC m=+1223.617318363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-maas-api-tls") pod "maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96" (UID: "c5051f50-3e61-4b4f-9358-be04824a254c") : secret "maas-api-e2e-shared-a-w3-426183-serving-cert" not found
Sep 09 04:18:22.168217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.168195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5051f50-3e61-4b4f-9358-be04824a254c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.170041 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.170016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.188421 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.188389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95bs4\" (UniqueName: \"kubernetes.io/projected/c5051f50-3e61-4b4f-9358-be04824a254c-kube-api-access-95bs4\") pod \"maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.672917 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.672881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.675468 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.675445    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:22.932691 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:22.932606    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:23.210824 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:23.210792    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"]
Sep 09 04:18:23.212616 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:18:23.212580    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5051f50_3e61_4b4f_9358_be04824a254c.slice/crio-c8c88f7839be94910d5ec2b1e178455ab46528727da325ac68df94ad9f1d397e WatchSource:0}: Error finding container c8c88f7839be94910d5ec2b1e178455ab46528727da325ac68df94ad9f1d397e: Status 404 returned error can't find the container with id c8c88f7839be94910d5ec2b1e178455ab46528727da325ac68df94ad9f1d397e
Sep 09 04:18:23.597542 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:23.597513    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" path="/var/lib/kubelet/pods/768ee68e-ff97-4f05-8ac6-df6d464e5088/volumes"
Sep 09 04:18:23.772792 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:23.772683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96" event={"ID":"c5051f50-3e61-4b4f-9358-be04824a254c","Type":"ContainerStarted","Data":"f888cf4dddd87a52f4d4dc529ea7914f3614647dcc8af2c853e801dad184cc49"}
Sep 09 04:18:23.772792 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:23.772751    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:23.773183 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:23.772811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96" event={"ID":"c5051f50-3e61-4b4f-9358-be04824a254c","Type":"ContainerStarted","Data":"c8c88f7839be94910d5ec2b1e178455ab46528727da325ac68df94ad9f1d397e"}
Sep 09 04:18:23.793201 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:23.793146    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96" podStartSLOduration=2.497392261 podStartE2EDuration="2.79313054s" totalImagesPullingTime="295.738279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:18:23.790442698 +0000 UTC m=+1224.740133353" watchObservedRunningTime="2026-09-09 04:18:23.79313054 +0000 UTC m=+1224.742821194"
Sep 09 04:18:29.784357 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:29.784323    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:18:38.883734 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:38.883700    2580 generic.go:356] "Generic (PLEG): container finished" podID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerID="61ad871ea48f1a674d788c4f2870aec3c2a0a8fd04ddd76101b4e95af3e09af4" exitCode=0
Sep 09 04:18:38.884262 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:38.883791    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" event={"ID":"0793bc60-b1ec-46bd-99a5-828a08edd26f","Type":"ContainerDied","Data":"61ad871ea48f1a674d788c4f2870aec3c2a0a8fd04ddd76101b4e95af3e09af4"}
Sep 09 04:18:38.884333 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:38.884272    2580 scope.go:117] "RemoveContainer" containerID="61ad871ea48f1a674d788c4f2870aec3c2a0a8fd04ddd76101b4e95af3e09af4"
Sep 09 04:18:38.885444 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:38.885403    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:18:38.885532 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:38.885494    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:18:38.885573 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:38.885537    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:18:39.891700 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:39.891664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" event={"ID":"0793bc60-b1ec-46bd-99a5-828a08edd26f","Type":"ContainerStarted","Data":"90c1f5291048765deb6e222fc50c985afae55832eae6f0aed4e452ae1f7351c2"}
Sep 09 04:18:40.681988 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:40.681951    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:40.682292 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:40.682274    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 09 04:18:40.682368 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:40.682326    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 09 04:18:41.683241 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:41.683202    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 09 04:18:41.683655 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:41.683261    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 09 04:18:42.682866 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:42.682828    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 09 04:18:42.683134 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:42.682895    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 09 04:18:43.683316 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:43.683272    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 09 04:18:43.683724 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:43.683345    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 09 04:18:44.682421 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:44.682385    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 09 04:18:44.682620 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:44.682454    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 09 04:18:45.687809 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:45.687779    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:45.688205 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:45.688051    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:45.688825 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:45.688808    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:18:46.409350 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.409309    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"]
Sep 09 04:18:46.409897 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.409877    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="768ee68e-ff97-4f05-8ac6-df6d464e5088" containerName="istio-proxy"
Sep 09 04:18:46.413340 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.413321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.416398 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.416315    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-9c44b5-openshift-default-dockercfg-ldbpc\""
Sep 09 04:18:46.428132 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.428109    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"]
Sep 09 04:18:46.513074 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.512444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-credential-socket\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.513074 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.512500    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-workload-socket\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.513074 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.512612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d40511-21fb-460a-8ccb-ed86caf19e3a-istiod-ca-cert\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.513074 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.512668    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpjz\" (UniqueName: \"kubernetes.io/projected/45d40511-21fb-460a-8ccb-ed86caf19e3a-kube-api-access-rzpjz\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.513074 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.512741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-podinfo\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.513074 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.512843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-envoy\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.513074 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.512875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-data\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.513074 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.512913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-workload-certs\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.513074 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.512943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-token\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.614163 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.614125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-credential-socket\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.614345 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.614168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-workload-socket\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.614345 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.614227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d40511-21fb-460a-8ccb-ed86caf19e3a-istiod-ca-cert\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.614345 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.614271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpjz\" (UniqueName: \"kubernetes.io/projected/45d40511-21fb-460a-8ccb-ed86caf19e3a-kube-api-access-rzpjz\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.614345 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.614333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-podinfo\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.614573 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.614406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-envoy\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.614573 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.614436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-data\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.614687 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.614584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-workload-certs\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.614687 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.614626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-token\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.614832 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.614701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-credential-socket\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.615060 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.615029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d40511-21fb-460a-8ccb-ed86caf19e3a-istiod-ca-cert\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.615170 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.615045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-workload-certs\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.615170 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.615103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-data\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.615284 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.615174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-workload-socket\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.617057 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.617034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-podinfo\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.617215 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.617199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-envoy\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.624014 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.623994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpjz\" (UniqueName: \"kubernetes.io/projected/45d40511-21fb-460a-8ccb-ed86caf19e3a-kube-api-access-rzpjz\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.624102 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.624068    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-token\") pod \"e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") " pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:46.724253 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:46.724175    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:47.009683 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:47.009648    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"]
Sep 09 04:18:47.010692 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:18:47.010668    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d40511_21fb_460a_8ccb_ed86caf19e3a.slice/crio-e177bcc123dffae60e0bf85dd6b66c709817e2235e3636566d685b9c211eeca8 WatchSource:0}: Error finding container e177bcc123dffae60e0bf85dd6b66c709817e2235e3636566d685b9c211eeca8: Status 404 returned error can't find the container with id e177bcc123dffae60e0bf85dd6b66c709817e2235e3636566d685b9c211eeca8
Sep 09 04:18:47.013280 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:47.013239    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:18:47.013366 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:47.013321    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:18:47.013366 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:47.013360    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:18:47.950665 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:47.950632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" event={"ID":"45d40511-21fb-460a-8ccb-ed86caf19e3a","Type":"ContainerStarted","Data":"fd99db5a2e6b2d9e3fcffedba63d06f86e224eac622b303318816757fce973cc"}
Sep 09 04:18:47.951086 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:47.950682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" event={"ID":"45d40511-21fb-460a-8ccb-ed86caf19e3a","Type":"ContainerStarted","Data":"e177bcc123dffae60e0bf85dd6b66c709817e2235e3636566d685b9c211eeca8"}
Sep 09 04:18:47.976261 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:47.976213    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" podStartSLOduration=1.976197548 podStartE2EDuration="1.976197548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:18:47.975607554 +0000 UTC m=+1248.925298211" watchObservedRunningTime="2026-09-09 04:18:47.976197548 +0000 UTC m=+1248.925888201"
Sep 09 04:18:48.725159 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:48.725117    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:48.730492 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:48.730465    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:48.958233 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:48.958205    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:18:48.959370 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:18:48.959350    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:19:18.959931 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:18.959894    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Sep 09 04:19:18.960339 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:18.959964    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 09 04:19:19.182934 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:19.182901    2580 generic.go:356] "Generic (PLEG): container finished" podID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerID="fd99db5a2e6b2d9e3fcffedba63d06f86e224eac622b303318816757fce973cc" exitCode=0
Sep 09 04:19:19.183095 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:19.182960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" event={"ID":"45d40511-21fb-460a-8ccb-ed86caf19e3a","Type":"ContainerDied","Data":"fd99db5a2e6b2d9e3fcffedba63d06f86e224eac622b303318816757fce973cc"}
Sep 09 04:19:19.183373 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:19.183357    2580 scope.go:117] "RemoveContainer" containerID="fd99db5a2e6b2d9e3fcffedba63d06f86e224eac622b303318816757fce973cc"
Sep 09 04:19:19.184506 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:19.184476    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:19:19.184602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:19.184546    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:19:19.184602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:19.184577    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:19:20.190669 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:20.190630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" event={"ID":"45d40511-21fb-460a-8ccb-ed86caf19e3a","Type":"ContainerStarted","Data":"dd6d668a8841206fb5eb72fc518d3c597b8864f395b1d655d4f88a88ed5b3f94"}
Sep 09 04:19:20.724785 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:20.724733    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:19:20.725148 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:20.725126    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Sep 09 04:19:20.725311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:20.725289    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 09 04:19:21.724894 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:21.724852    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Sep 09 04:19:21.725387 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:21.724923    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 09 04:19:22.725280 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:22.725236    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Sep 09 04:19:22.725750 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:22.725303    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 09 04:19:23.724916 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:23.724881    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Sep 09 04:19:23.725216 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:23.725187    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 09 04:19:24.725509 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:24.725466    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Sep 09 04:19:24.726036 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:24.725543    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 09 04:19:25.729308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:25.729277    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:19:25.729695 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:25.729663    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:19:25.730461 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:25.730442    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:19:48.929663 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:48.929625    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"]
Sep 09 04:19:48.930283 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:48.929918    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj" podUID="926546b5-812b-43c5-8d8b-46d0af87b1d5" containerName="maas-api" containerID="cri-o://9fac7b01f2f05508a5d6e3a2945b9aebf15f8f5b743c78f35227654037b44114" gracePeriod=30
Sep 09 04:19:49.186734 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.186678    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:19:49.343048 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.343016    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-metrics-certs\") pod \"926546b5-812b-43c5-8d8b-46d0af87b1d5\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") "
Sep 09 04:19:49.343254 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.343083    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-maas-api-tls\") pod \"926546b5-812b-43c5-8d8b-46d0af87b1d5\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") "
Sep 09 04:19:49.343254 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.343150    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkb5s\" (UniqueName: \"kubernetes.io/projected/926546b5-812b-43c5-8d8b-46d0af87b1d5-kube-api-access-xkb5s\") pod \"926546b5-812b-43c5-8d8b-46d0af87b1d5\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") "
Sep 09 04:19:49.344105 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.343652    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/926546b5-812b-43c5-8d8b-46d0af87b1d5-openshift-service-ca\") pod \"926546b5-812b-43c5-8d8b-46d0af87b1d5\" (UID: \"926546b5-812b-43c5-8d8b-46d0af87b1d5\") "
Sep 09 04:19:49.344337 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.344047    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926546b5-812b-43c5-8d8b-46d0af87b1d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "926546b5-812b-43c5-8d8b-46d0af87b1d5" (UID: "926546b5-812b-43c5-8d8b-46d0af87b1d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:19:49.344810 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.344755    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/926546b5-812b-43c5-8d8b-46d0af87b1d5-openshift-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:19:49.346135 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.346100    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "926546b5-812b-43c5-8d8b-46d0af87b1d5" (UID: "926546b5-812b-43c5-8d8b-46d0af87b1d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:19:49.346314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.346158    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926546b5-812b-43c5-8d8b-46d0af87b1d5-kube-api-access-xkb5s" (OuterVolumeSpecName: "kube-api-access-xkb5s") pod "926546b5-812b-43c5-8d8b-46d0af87b1d5" (UID: "926546b5-812b-43c5-8d8b-46d0af87b1d5"). InnerVolumeSpecName "kube-api-access-xkb5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:19:49.348114 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.348075    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "926546b5-812b-43c5-8d8b-46d0af87b1d5" (UID: "926546b5-812b-43c5-8d8b-46d0af87b1d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:19:49.381903 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.381862    2580 generic.go:356] "Generic (PLEG): container finished" podID="926546b5-812b-43c5-8d8b-46d0af87b1d5" containerID="9fac7b01f2f05508a5d6e3a2945b9aebf15f8f5b743c78f35227654037b44114" exitCode=0
Sep 09 04:19:49.382040 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.381912    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"
Sep 09 04:19:49.382040 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.381932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj" event={"ID":"926546b5-812b-43c5-8d8b-46d0af87b1d5","Type":"ContainerDied","Data":"9fac7b01f2f05508a5d6e3a2945b9aebf15f8f5b743c78f35227654037b44114"}
Sep 09 04:19:49.382040 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.381992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj" event={"ID":"926546b5-812b-43c5-8d8b-46d0af87b1d5","Type":"ContainerDied","Data":"b94543ae8a06f89a97083af8c1e8bb019145e30b53a159e1c60724e76818e053"}
Sep 09 04:19:49.382040 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.382017    2580 scope.go:117] "RemoveContainer" containerID="9fac7b01f2f05508a5d6e3a2945b9aebf15f8f5b743c78f35227654037b44114"
Sep 09 04:19:49.392593 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.392570    2580 scope.go:117] "RemoveContainer" containerID="9fac7b01f2f05508a5d6e3a2945b9aebf15f8f5b743c78f35227654037b44114"
Sep 09 04:19:49.392948 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:19:49.392928    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fac7b01f2f05508a5d6e3a2945b9aebf15f8f5b743c78f35227654037b44114\": container with ID starting with 9fac7b01f2f05508a5d6e3a2945b9aebf15f8f5b743c78f35227654037b44114 not found: ID does not exist" containerID="9fac7b01f2f05508a5d6e3a2945b9aebf15f8f5b743c78f35227654037b44114"
Sep 09 04:19:49.393016 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.392960    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fac7b01f2f05508a5d6e3a2945b9aebf15f8f5b743c78f35227654037b44114"} err="failed to get container status \"9fac7b01f2f05508a5d6e3a2945b9aebf15f8f5b743c78f35227654037b44114\": rpc error: code = NotFound desc = could not find container \"9fac7b01f2f05508a5d6e3a2945b9aebf15f8f5b743c78f35227654037b44114\": container with ID starting with 9fac7b01f2f05508a5d6e3a2945b9aebf15f8f5b743c78f35227654037b44114 not found: ID does not exist"
Sep 09 04:19:49.405494 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.405470    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"]
Sep 09 04:19:49.409473 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.409448    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-429d71ed-f8d8fd74f-br7sj"]
Sep 09 04:19:49.446223 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.446145    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:19:49.446223 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.446170    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/926546b5-812b-43c5-8d8b-46d0af87b1d5-maas-api-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:19:49.446223 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.446183    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkb5s\" (UniqueName: \"kubernetes.io/projected/926546b5-812b-43c5-8d8b-46d0af87b1d5-kube-api-access-xkb5s\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:19:49.596582 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:19:49.596554    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="926546b5-812b-43c5-8d8b-46d0af87b1d5" path="/var/lib/kubelet/pods/926546b5-812b-43c5-8d8b-46d0af87b1d5/volumes"
Sep 09 04:21:28.456117 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.456078    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"]
Sep 09 04:21:28.456730 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.456714    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926546b5-812b-43c5-8d8b-46d0af87b1d5" containerName="maas-api"
Sep 09 04:21:28.456824 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.456732    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="926546b5-812b-43c5-8d8b-46d0af87b1d5" containerName="maas-api"
Sep 09 04:21:28.456867 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.456842    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="926546b5-812b-43c5-8d8b-46d0af87b1d5" containerName="maas-api"
Sep 09 04:21:28.459955 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.459939    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.462523 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.462499    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-85732c\"/\"default-dockercfg-scmzk\""
Sep 09 04:21:28.462677 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.462559    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-85732c\"/\"kube-root-ca.crt\""
Sep 09 04:21:28.463915 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.463896    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-85732c\"/\"openshift-service-ca.crt\""
Sep 09 04:21:28.464027 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.463894    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-85732c\"/\"e2e-conflict-b-4afe0483-kserve-self-signed-certs\""
Sep 09 04:21:28.469943 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.469921    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"]
Sep 09 04:21:28.488736 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.488710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-dshm\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.488886 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.488797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-tmp-dir\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.488886 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.488839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481cbb10-c919-4dd0-b751-5f277b4420d2-tls-certs\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.488886 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.488862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-home\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.489040 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.488896    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22vrx\" (UniqueName: \"kubernetes.io/projected/481cbb10-c919-4dd0-b751-5f277b4420d2-kube-api-access-22vrx\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.489040 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.488921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-model-cache\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.589483 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.589447    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-home\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.589660 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.589507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22vrx\" (UniqueName: \"kubernetes.io/projected/481cbb10-c919-4dd0-b751-5f277b4420d2-kube-api-access-22vrx\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.589660 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.589551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-model-cache\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.589660 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.589596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-dshm\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.589901 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.589783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-tmp-dir\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.589901 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.589824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-home\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.589901 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.589878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481cbb10-c919-4dd0-b751-5f277b4420d2-tls-certs\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.590077 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.589992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-model-cache\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.590139 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.590114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-tmp-dir\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.592044 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.592021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-dshm\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.592336 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.592319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481cbb10-c919-4dd0-b751-5f277b4420d2-tls-certs\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.599040 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.599016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22vrx\" (UniqueName: \"kubernetes.io/projected/481cbb10-c919-4dd0-b751-5f277b4420d2-kube-api-access-22vrx\") pod \"e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") " pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:28.770376 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:28.770285    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:29.046675 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:29.046643    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"]
Sep 09 04:21:29.046840 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:21:29.046696    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod481cbb10_c919_4dd0_b751_5f277b4420d2.slice/crio-a1fd10881eb49b26c0678aa19ec7621273d8625596696e1796bc40e2a3597420 WatchSource:0}: Error finding container a1fd10881eb49b26c0678aa19ec7621273d8625596696e1796bc40e2a3597420: Status 404 returned error can't find the container with id a1fd10881eb49b26c0678aa19ec7621273d8625596696e1796bc40e2a3597420
Sep 09 04:21:30.004842 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:30.004803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd" event={"ID":"481cbb10-c919-4dd0-b751-5f277b4420d2","Type":"ContainerStarted","Data":"4fab0718df397d214ee473dedaae3c75eff0b2aed26dcc47127bd5cad86f837f"}
Sep 09 04:21:30.005346 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:30.004857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd" event={"ID":"481cbb10-c919-4dd0-b751-5f277b4420d2","Type":"ContainerStarted","Data":"a1fd10881eb49b26c0678aa19ec7621273d8625596696e1796bc40e2a3597420"}
Sep 09 04:21:30.023941 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:30.023893    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd" podStartSLOduration=2.023879304 podStartE2EDuration="2.023879304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:21:30.020707175 +0000 UTC m=+1410.970397846" watchObservedRunningTime="2026-09-09 04:21:30.023879304 +0000 UTC m=+1410.973569958"
Sep 09 04:21:38.771140 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:38.771100    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:38.771600 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:38.771160    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:38.783749 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:38.783723    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:39.076005 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:39.075928    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:21:45.720077 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:45.720042    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"]
Sep 09 04:21:52.854955 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:52.854860    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"]
Sep 09 04:21:52.858954 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:52.858932    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:52.861812 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:52.861788    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f95c96-openshift-default-dockercfg-bmlf5\""
Sep 09 04:21:52.871130 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:52.871101    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"]
Sep 09 04:21:52.920161 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:52.920122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-workload-socket\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:52.920317 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:52.920174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istiod-ca-cert\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:52.920317 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:52.920203    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-credential-socket\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:52.920317 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:52.920266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-envoy\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:52.920433 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:52.920333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-token\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:52.920433 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:52.920376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-data\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:52.920433 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:52.920393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-podinfo\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:52.920541 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:52.920485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-workload-certs\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:52.920541 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:52.920511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkqhx\" (UniqueName: \"kubernetes.io/projected/87a7bcd9-7dc3-4bbc-8486-d203c199d256-kube-api-access-nkqhx\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022024 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.021981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istiod-ca-cert\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022194 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.022033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-credential-socket\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022194 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.022092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-envoy\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022194 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.022139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-token\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022194 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.022181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-data\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022439 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.022209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-podinfo\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022439 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.022407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-workload-certs\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022548 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.022453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkqhx\" (UniqueName: \"kubernetes.io/projected/87a7bcd9-7dc3-4bbc-8486-d203c199d256-kube-api-access-nkqhx\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022548 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.022539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-workload-socket\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022656 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.022577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-credential-socket\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022710 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.022656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-data\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022827 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.022803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-workload-certs\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022910 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.022849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istiod-ca-cert\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.022910 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.022897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-workload-socket\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.024731 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.024708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-envoy\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.025265 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.025246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-podinfo\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.034652 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.034624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-token\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.034794 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.034693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkqhx\" (UniqueName: \"kubernetes.io/projected/87a7bcd9-7dc3-4bbc-8486-d203c199d256-kube-api-access-nkqhx\") pod \"e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") " pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.173446 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.173359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:53.455443 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.455413    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"]
Sep 09 04:21:53.456481 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:21:53.456458    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a7bcd9_7dc3_4bbc_8486_d203c199d256.slice/crio-fd25e455feabe9acd612203c193b7ab719c10027b6622ac4d84c44da9d9ba47f WatchSource:0}: Error finding container fd25e455feabe9acd612203c193b7ab719c10027b6622ac4d84c44da9d9ba47f: Status 404 returned error can't find the container with id fd25e455feabe9acd612203c193b7ab719c10027b6622ac4d84c44da9d9ba47f
Sep 09 04:21:53.458997 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.458963    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:21:53.459098 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.459030    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:21:53.459098 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:53.459057    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:21:54.172485 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:54.172443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf" event={"ID":"87a7bcd9-7dc3-4bbc-8486-d203c199d256","Type":"ContainerStarted","Data":"563327a0ada10a5feeef13927e7e158e0eaac28e7a1b74c77d07f584098a4ad5"}
Sep 09 04:21:54.172947 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:54.172541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf" event={"ID":"87a7bcd9-7dc3-4bbc-8486-d203c199d256","Type":"ContainerStarted","Data":"fd25e455feabe9acd612203c193b7ab719c10027b6622ac4d84c44da9d9ba47f"}
Sep 09 04:21:54.173751 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:54.173731    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:54.178745 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:54.178712    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:54.194315 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:54.194268    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf" podStartSLOduration=2.194256242 podStartE2EDuration="2.194256242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:21:54.191195329 +0000 UTC m=+1435.140885985" watchObservedRunningTime="2026-09-09 04:21:54.194256242 +0000 UTC m=+1435.143946958"
Sep 09 04:21:55.181311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:55.181276    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:55.182387 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:55.182351    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:21:59.846426 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:21:59.846391    2580 scope.go:117] "RemoveContainer" containerID="edde8cbc50d3fed0f6e66e9e66aa6219b8daaafaa02a25dfbadc8cea271dc5e3"
Sep 09 04:22:00.740933 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:00.740883    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd" podUID="481cbb10-c919-4dd0-b751-5f277b4420d2" containerName="main" containerID="cri-o://4fab0718df397d214ee473dedaae3c75eff0b2aed26dcc47127bd5cad86f837f" gracePeriod=45
Sep 09 04:22:00.994972 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:00.994908    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:22:01.102144 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.102113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-model-cache\") pod \"481cbb10-c919-4dd0-b751-5f277b4420d2\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") "
Sep 09 04:22:01.102334 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.102165    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481cbb10-c919-4dd0-b751-5f277b4420d2-tls-certs\") pod \"481cbb10-c919-4dd0-b751-5f277b4420d2\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") "
Sep 09 04:22:01.102334 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.102211    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-dshm\") pod \"481cbb10-c919-4dd0-b751-5f277b4420d2\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") "
Sep 09 04:22:01.102334 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.102233    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22vrx\" (UniqueName: \"kubernetes.io/projected/481cbb10-c919-4dd0-b751-5f277b4420d2-kube-api-access-22vrx\") pod \"481cbb10-c919-4dd0-b751-5f277b4420d2\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") "
Sep 09 04:22:01.102334 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.102302    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-home\") pod \"481cbb10-c919-4dd0-b751-5f277b4420d2\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") "
Sep 09 04:22:01.102574 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.102378    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-tmp-dir\") pod \"481cbb10-c919-4dd0-b751-5f277b4420d2\" (UID: \"481cbb10-c919-4dd0-b751-5f277b4420d2\") "
Sep 09 04:22:01.102655 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.102593    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-home" (OuterVolumeSpecName: "home") pod "481cbb10-c919-4dd0-b751-5f277b4420d2" (UID: "481cbb10-c919-4dd0-b751-5f277b4420d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:22:01.102719 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.102665    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "481cbb10-c919-4dd0-b751-5f277b4420d2" (UID: "481cbb10-c919-4dd0-b751-5f277b4420d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:22:01.102796 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.102732    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "481cbb10-c919-4dd0-b751-5f277b4420d2" (UID: "481cbb10-c919-4dd0-b751-5f277b4420d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:22:01.102860 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.102796    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-home\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:22:01.102860 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.102815    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-tmp-dir\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:22:01.104799 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.104752    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481cbb10-c919-4dd0-b751-5f277b4420d2-kube-api-access-22vrx" (OuterVolumeSpecName: "kube-api-access-22vrx") pod "481cbb10-c919-4dd0-b751-5f277b4420d2" (UID: "481cbb10-c919-4dd0-b751-5f277b4420d2"). InnerVolumeSpecName "kube-api-access-22vrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:22:01.105080 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.105060    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481cbb10-c919-4dd0-b751-5f277b4420d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "481cbb10-c919-4dd0-b751-5f277b4420d2" (UID: "481cbb10-c919-4dd0-b751-5f277b4420d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:22:01.105238 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.105215    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-dshm" (OuterVolumeSpecName: "dshm") pod "481cbb10-c919-4dd0-b751-5f277b4420d2" (UID: "481cbb10-c919-4dd0-b751-5f277b4420d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:22:01.204344 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.204311    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-model-cache\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:22:01.204344 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.204347    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481cbb10-c919-4dd0-b751-5f277b4420d2-tls-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:22:01.204344 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.204359    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481cbb10-c919-4dd0-b751-5f277b4420d2-dshm\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:22:01.204572 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.204372    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22vrx\" (UniqueName: \"kubernetes.io/projected/481cbb10-c919-4dd0-b751-5f277b4420d2-kube-api-access-22vrx\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:22:01.229757 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.229725    2580 generic.go:356] "Generic (PLEG): container finished" podID="481cbb10-c919-4dd0-b751-5f277b4420d2" containerID="4fab0718df397d214ee473dedaae3c75eff0b2aed26dcc47127bd5cad86f837f" exitCode=0
Sep 09 04:22:01.229936 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.229787    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"
Sep 09 04:22:01.229936 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.229817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd" event={"ID":"481cbb10-c919-4dd0-b751-5f277b4420d2","Type":"ContainerDied","Data":"4fab0718df397d214ee473dedaae3c75eff0b2aed26dcc47127bd5cad86f837f"}
Sep 09 04:22:01.229936 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.229875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd" event={"ID":"481cbb10-c919-4dd0-b751-5f277b4420d2","Type":"ContainerDied","Data":"a1fd10881eb49b26c0678aa19ec7621273d8625596696e1796bc40e2a3597420"}
Sep 09 04:22:01.229936 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.229893    2580 scope.go:117] "RemoveContainer" containerID="4fab0718df397d214ee473dedaae3c75eff0b2aed26dcc47127bd5cad86f837f"
Sep 09 04:22:01.240027 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.240012    2580 scope.go:117] "RemoveContainer" containerID="4fab0718df397d214ee473dedaae3c75eff0b2aed26dcc47127bd5cad86f837f"
Sep 09 04:22:01.240302 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:22:01.240286    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fab0718df397d214ee473dedaae3c75eff0b2aed26dcc47127bd5cad86f837f\": container with ID starting with 4fab0718df397d214ee473dedaae3c75eff0b2aed26dcc47127bd5cad86f837f not found: ID does not exist" containerID="4fab0718df397d214ee473dedaae3c75eff0b2aed26dcc47127bd5cad86f837f"
Sep 09 04:22:01.240343 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.240310    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fab0718df397d214ee473dedaae3c75eff0b2aed26dcc47127bd5cad86f837f"} err="failed to get container status \"4fab0718df397d214ee473dedaae3c75eff0b2aed26dcc47127bd5cad86f837f\": rpc error: code = NotFound desc = could not find container \"4fab0718df397d214ee473dedaae3c75eff0b2aed26dcc47127bd5cad86f837f\": container with ID starting with 4fab0718df397d214ee473dedaae3c75eff0b2aed26dcc47127bd5cad86f837f not found: ID does not exist"
Sep 09 04:22:01.252680 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.252610    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"]
Sep 09 04:22:01.255452 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.255432    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-85732c/e2e-conflict-b-4afe0483-kserve-5dc97c5f78-jtnpd"]
Sep 09 04:22:01.596098 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:01.596069    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="481cbb10-c919-4dd0-b751-5f277b4420d2" path="/var/lib/kubelet/pods/481cbb10-c919-4dd0-b751-5f277b4420d2/volumes"
Sep 09 04:22:16.324662 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:16.324572    2580 generic.go:356] "Generic (PLEG): container finished" podID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerID="563327a0ada10a5feeef13927e7e158e0eaac28e7a1b74c77d07f584098a4ad5" exitCode=0
Sep 09 04:22:16.324662 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:16.324617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf" event={"ID":"87a7bcd9-7dc3-4bbc-8486-d203c199d256","Type":"ContainerDied","Data":"563327a0ada10a5feeef13927e7e158e0eaac28e7a1b74c77d07f584098a4ad5"}
Sep 09 04:22:16.325194 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:16.325030    2580 scope.go:117] "RemoveContainer" containerID="563327a0ada10a5feeef13927e7e158e0eaac28e7a1b74c77d07f584098a4ad5"
Sep 09 04:22:16.326196 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:16.326161    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:22:16.326311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:16.326248    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:22:16.326311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:16.326297    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:22:17.336181 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:17.336139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf" event={"ID":"87a7bcd9-7dc3-4bbc-8486-d203c199d256","Type":"ContainerStarted","Data":"bf113e78db68dc0b65f963659953061cfe7dd95b750da647221330196d7cc665"}
Sep 09 04:22:18.174478 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:18.174434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:22:18.174706 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:18.174676    2580 patch_prober.go:28] interesting pod/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 09 04:22:18.174818 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:18.174746    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf" podUID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 09 04:22:19.174687 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:19.174646    2580 patch_prober.go:28] interesting pod/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 09 04:22:19.175107 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:19.174722    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf" podUID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 09 04:22:20.174488 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:20.174450    2580 patch_prober.go:28] interesting pod/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 09 04:22:20.174687 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:20.174504    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf" podUID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 09 04:22:21.174598 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.174562    2580 patch_prober.go:28] interesting pod/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 09 04:22:21.175017 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.174613    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf" podUID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 09 04:22:21.823697 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.823663    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"]
Sep 09 04:22:21.828790 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.824667    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="481cbb10-c919-4dd0-b751-5f277b4420d2" containerName="main"
Sep 09 04:22:21.828981 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.828963    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="481cbb10-c919-4dd0-b751-5f277b4420d2" containerName="main"
Sep 09 04:22:21.829233 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.829220    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="481cbb10-c919-4dd0-b751-5f277b4420d2" containerName="main"
Sep 09 04:22:21.833601 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.833579    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:21.841623 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.841200    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-acfe5b-openshift-default-dockercfg-mzb7b\""
Sep 09 04:22:21.845415 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.845355    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"]
Sep 09 04:22:21.922791 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.918626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-credential-socket\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:21.922791 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.918692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-workload-socket\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:21.922791 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.918806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-workload-certs\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:21.922791 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.918864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505aa41e-5775-4636-aa7d-fe082483949a-istiod-ca-cert\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:21.922791 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.918919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505aa41e-5775-4636-aa7d-fe082483949a-istio-podinfo\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:21.922791 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.918964    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505aa41e-5775-4636-aa7d-fe082483949a-istio-token\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:21.922791 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.919053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm5b7\" (UniqueName: \"kubernetes.io/projected/505aa41e-5775-4636-aa7d-fe082483949a-kube-api-access-rm5b7\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:21.922791 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.919098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-istio-envoy\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:21.922791 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:21.919143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-istio-data\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.020640 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.020593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505aa41e-5775-4636-aa7d-fe082483949a-istiod-ca-cert\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.020881 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.020667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505aa41e-5775-4636-aa7d-fe082483949a-istio-podinfo\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.020881 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.020694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505aa41e-5775-4636-aa7d-fe082483949a-istio-token\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.020881 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.020785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm5b7\" (UniqueName: \"kubernetes.io/projected/505aa41e-5775-4636-aa7d-fe082483949a-kube-api-access-rm5b7\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.020881 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.020816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-istio-envoy\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.020881 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.020862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-istio-data\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.021180 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.020929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-credential-socket\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.021180 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.020958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-workload-socket\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.021180 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.021032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-workload-certs\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.021535 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.021504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-workload-certs\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.022216 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.022185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505aa41e-5775-4636-aa7d-fe082483949a-istiod-ca-cert\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.023386 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.023363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-istio-data\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.024109 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.023992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-workload-socket\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.024604 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.024583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-credential-socket\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.025672 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.025373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505aa41e-5775-4636-aa7d-fe082483949a-istio-podinfo\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.030042 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.030020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-istio-envoy\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.035582 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.035559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm5b7\" (UniqueName: \"kubernetes.io/projected/505aa41e-5775-4636-aa7d-fe082483949a-kube-api-access-rm5b7\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.036128 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.036075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505aa41e-5775-4636-aa7d-fe082483949a-istio-token\") pod \"e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") " pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.159228 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.159192    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:22.174058 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.174028    2580 patch_prober.go:28] interesting pod/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 09 04:22:22.174129 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.174090    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf" podUID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 09 04:22:22.457898 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.457872    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"]
Sep 09 04:22:22.461013 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:22:22.460975    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505aa41e_5775_4636_aa7d_fe082483949a.slice/crio-be3f9561ffa32d9c5d3d6891aa7f00a3377d1639490a0f27079a3710e5cb441e WatchSource:0}: Error finding container be3f9561ffa32d9c5d3d6891aa7f00a3377d1639490a0f27079a3710e5cb441e: Status 404 returned error can't find the container with id be3f9561ffa32d9c5d3d6891aa7f00a3377d1639490a0f27079a3710e5cb441e
Sep 09 04:22:22.463832 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.463760    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:22:22.463961 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.463879    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:22:22.463961 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:22.463923    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:22:23.178047 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:23.178017    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:22:23.178323 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:23.178301    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:22:23.179060 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:23.179039    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:22:23.377816 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:23.377748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" event={"ID":"505aa41e-5775-4636-aa7d-fe082483949a","Type":"ContainerStarted","Data":"61911cb036e9bb78dece2d96dabb091db0427af3cc0fb3c6f7eef8c47f41712b"}
Sep 09 04:22:23.378000 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:23.377849    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" event={"ID":"505aa41e-5775-4636-aa7d-fe082483949a","Type":"ContainerStarted","Data":"be3f9561ffa32d9c5d3d6891aa7f00a3377d1639490a0f27079a3710e5cb441e"}
Sep 09 04:22:23.401309 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:23.401266    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" podStartSLOduration=2.401252713 podStartE2EDuration="2.401252713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:22:23.397712917 +0000 UTC m=+1464.347403571" watchObservedRunningTime="2026-09-09 04:22:23.401252713 +0000 UTC m=+1464.350943367"
Sep 09 04:22:24.159716 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:24.159673    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:24.165321 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:24.165291    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:24.384172 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:24.384133    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:24.384846 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:24.384828    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:26.581180 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.581147    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"]
Sep 09 04:22:26.585123 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.585097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:26.587379 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.587354    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-acfe5b-serving-cert\""
Sep 09 04:22:26.601005 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.600983    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"]
Sep 09 04:22:26.672207 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.672175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-openshift-service-ca\") pod \"maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:26.672350 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.672213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-maas-api-tls\") pod \"maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:26.672350 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.672242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-metrics-certs\") pod \"maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:26.672447 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.672372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zlxv\" (UniqueName: \"kubernetes.io/projected/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-kube-api-access-2zlxv\") pod \"maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:26.773280 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.773249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-metrics-certs\") pod \"maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:26.773485 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.773313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zlxv\" (UniqueName: \"kubernetes.io/projected/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-kube-api-access-2zlxv\") pod \"maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:26.773485 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.773383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-openshift-service-ca\") pod \"maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:26.773485 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.773406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-maas-api-tls\") pod \"maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:26.773671 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:22:26.773493    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-serving-cert: secret "maas-api-e2e-api-b-acfe5b-serving-cert" not found
Sep 09 04:22:26.773671 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:22:26.773545    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-maas-api-tls podName:bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9 nodeName:}" failed. No retries permitted until 2026-09-09 04:22:27.273530691 +0000 UTC m=+1468.223221328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-maas-api-tls") pod "maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w" (UID: "bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9") : secret "maas-api-e2e-api-b-acfe5b-serving-cert" not found
Sep 09 04:22:26.774174 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.774150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-openshift-service-ca\") pod \"maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:26.775978 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.775956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-metrics-certs\") pod \"maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:26.781648 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:26.781631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zlxv\" (UniqueName: \"kubernetes.io/projected/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-kube-api-access-2zlxv\") pod \"maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:27.278439 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:27.278403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-maas-api-tls\") pod \"maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:27.281014 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:27.280992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-maas-api-tls\") pod \"maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:27.504447 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:27.504406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:27.792506 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:27.792482    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"]
Sep 09 04:22:27.794203 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:22:27.794164    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbdc20bb_a1fa_4c75_a363_e9ce0622f2b9.slice/crio-0b3933f0ca4c81c7ccda264054ba70bda6c470c688d652071a5cfaa9ac87c09d WatchSource:0}: Error finding container 0b3933f0ca4c81c7ccda264054ba70bda6c470c688d652071a5cfaa9ac87c09d: Status 404 returned error can't find the container with id 0b3933f0ca4c81c7ccda264054ba70bda6c470c688d652071a5cfaa9ac87c09d
Sep 09 04:22:27.795552 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:27.795532    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 04:22:28.411721 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:28.411581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w" event={"ID":"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9","Type":"ContainerStarted","Data":"0b3933f0ca4c81c7ccda264054ba70bda6c470c688d652071a5cfaa9ac87c09d"}
Sep 09 04:22:29.422672 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:29.422636    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w" event={"ID":"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9","Type":"ContainerStarted","Data":"ee65f6b010e7f6defe3d0fe9ab4ed7046786ae38b912977c9657cb40cab926c5"}
Sep 09 04:22:29.423119 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:29.422813    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:29.439889 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:29.439825    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w" podStartSLOduration=2.2513145039999998 podStartE2EDuration="3.439808227s" totalImagesPullingTime="1.188493723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:22:29.439371849 +0000 UTC m=+1470.389062503" watchObservedRunningTime="2026-09-09 04:22:29.439808227 +0000 UTC m=+1470.389498882"
Sep 09 04:22:35.436077 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:35.436047    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:22:51.587283 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:51.587250    2580 generic.go:356] "Generic (PLEG): container finished" podID="505aa41e-5775-4636-aa7d-fe082483949a" containerID="61911cb036e9bb78dece2d96dabb091db0427af3cc0fb3c6f7eef8c47f41712b" exitCode=0
Sep 09 04:22:51.587701 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:51.587329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" event={"ID":"505aa41e-5775-4636-aa7d-fe082483949a","Type":"ContainerDied","Data":"61911cb036e9bb78dece2d96dabb091db0427af3cc0fb3c6f7eef8c47f41712b"}
Sep 09 04:22:51.587747 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:51.587717    2580 scope.go:117] "RemoveContainer" containerID="61911cb036e9bb78dece2d96dabb091db0427af3cc0fb3c6f7eef8c47f41712b"
Sep 09 04:22:51.588951 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:51.588918    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:22:51.589089 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:51.588988    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:22:51.589089 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:51.589024    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:22:52.595016 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:52.594984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" event={"ID":"505aa41e-5775-4636-aa7d-fe082483949a","Type":"ContainerStarted","Data":"b3e778df9705f8215e08958091a55501be44628236958b5c0910a61f2845c68b"}
Sep 09 04:22:53.159481 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:53.159446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:53.160609 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:53.160580    2580 patch_prober.go:28] interesting pod/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 09 04:22:53.160785 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:53.160638    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" podUID="505aa41e-5775-4636-aa7d-fe082483949a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 09 04:22:54.159624 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:54.159585    2580 patch_prober.go:28] interesting pod/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 09 04:22:54.160032 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:54.159650    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" podUID="505aa41e-5775-4636-aa7d-fe082483949a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 09 04:22:54.384112 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:54.384074    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:55.160357 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:55.160323    2580 patch_prober.go:28] interesting pod/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 09 04:22:55.160848 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:55.160383    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" podUID="505aa41e-5775-4636-aa7d-fe082483949a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 09 04:22:56.160218 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:56.160173    2580 patch_prober.go:28] interesting pod/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 09 04:22:56.160416 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:56.160246    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" podUID="505aa41e-5775-4636-aa7d-fe082483949a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 09 04:22:57.160381 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:57.160339    2580 patch_prober.go:28] interesting pod/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 09 04:22:57.160599 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:57.160415    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" podUID="505aa41e-5775-4636-aa7d-fe082483949a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 09 04:22:58.164201 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:58.164168    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:58.165209 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:58.165190    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:22:59.656055 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:59.656027    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:22:59.663921 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:22:59.663899    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:23:15.782484 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:15.782451    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"]
Sep 09 04:23:15.783091 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:15.782720    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf" podUID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerName="istio-proxy" containerID="cri-o://bf113e78db68dc0b65f963659953061cfe7dd95b750da647221330196d7cc665" gracePeriod=30
Sep 09 04:23:21.040394 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.040353    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:23:21.132242 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.132210    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-token\") pod \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") "
Sep 09 04:23:21.132242 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.132248    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-podinfo\") pod \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") "
Sep 09 04:23:21.132548 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.132287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istiod-ca-cert\") pod \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") "
Sep 09 04:23:21.132548 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.132409    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-credential-socket\") pod \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") "
Sep 09 04:23:21.132548 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.132480    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkqhx\" (UniqueName: \"kubernetes.io/projected/87a7bcd9-7dc3-4bbc-8486-d203c199d256-kube-api-access-nkqhx\") pod \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") "
Sep 09 04:23:21.132548 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.132512    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-envoy\") pod \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") "
Sep 09 04:23:21.132808 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.132577    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-data\") pod \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") "
Sep 09 04:23:21.132808 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.132620    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-workload-certs\") pod \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") "
Sep 09 04:23:21.132808 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.132635    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "87a7bcd9-7dc3-4bbc-8486-d203c199d256" (UID: "87a7bcd9-7dc3-4bbc-8486-d203c199d256"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:23:21.132808 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.132641    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "87a7bcd9-7dc3-4bbc-8486-d203c199d256" (UID: "87a7bcd9-7dc3-4bbc-8486-d203c199d256"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:23:21.132808 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.132656    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-workload-socket\") pod \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\" (UID: \"87a7bcd9-7dc3-4bbc-8486-d203c199d256\") "
Sep 09 04:23:21.133253 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.132889    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-data" (OuterVolumeSpecName: "istio-data") pod "87a7bcd9-7dc3-4bbc-8486-d203c199d256" (UID: "87a7bcd9-7dc3-4bbc-8486-d203c199d256"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:23:21.133253 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.132904    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "87a7bcd9-7dc3-4bbc-8486-d203c199d256" (UID: "87a7bcd9-7dc3-4bbc-8486-d203c199d256"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:23:21.133253 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.133040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "87a7bcd9-7dc3-4bbc-8486-d203c199d256" (UID: "87a7bcd9-7dc3-4bbc-8486-d203c199d256"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:23:21.133253 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.133198    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-data\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:21.133253 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.133219    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-workload-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:21.133253 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.133233    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-workload-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:21.133253 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.133247    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istiod-ca-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:21.133573 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.133261    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-credential-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:21.134601 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.134578    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "87a7bcd9-7dc3-4bbc-8486-d203c199d256" (UID: "87a7bcd9-7dc3-4bbc-8486-d203c199d256"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:23:21.135061 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.135040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-token" (OuterVolumeSpecName: "istio-token") pod "87a7bcd9-7dc3-4bbc-8486-d203c199d256" (UID: "87a7bcd9-7dc3-4bbc-8486-d203c199d256"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:23:21.135176 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.135157    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "87a7bcd9-7dc3-4bbc-8486-d203c199d256" (UID: "87a7bcd9-7dc3-4bbc-8486-d203c199d256"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:23:21.135176 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.135164    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a7bcd9-7dc3-4bbc-8486-d203c199d256-kube-api-access-nkqhx" (OuterVolumeSpecName: "kube-api-access-nkqhx") pod "87a7bcd9-7dc3-4bbc-8486-d203c199d256" (UID: "87a7bcd9-7dc3-4bbc-8486-d203c199d256"). InnerVolumeSpecName "kube-api-access-nkqhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:23:21.234277 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.234239    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:21.234277 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.234272    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-podinfo\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:21.234277 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.234281    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkqhx\" (UniqueName: \"kubernetes.io/projected/87a7bcd9-7dc3-4bbc-8486-d203c199d256-kube-api-access-nkqhx\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:21.234517 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.234291    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a7bcd9-7dc3-4bbc-8486-d203c199d256-istio-envoy\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:21.815757 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.815725    2580 generic.go:356] "Generic (PLEG): container finished" podID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerID="bf113e78db68dc0b65f963659953061cfe7dd95b750da647221330196d7cc665" exitCode=0
Sep 09 04:23:21.815952 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.815786    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"
Sep 09 04:23:21.815952 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.815799    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf" event={"ID":"87a7bcd9-7dc3-4bbc-8486-d203c199d256","Type":"ContainerDied","Data":"bf113e78db68dc0b65f963659953061cfe7dd95b750da647221330196d7cc665"}
Sep 09 04:23:21.815952 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.815857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf" event={"ID":"87a7bcd9-7dc3-4bbc-8486-d203c199d256","Type":"ContainerDied","Data":"fd25e455feabe9acd612203c193b7ab719c10027b6622ac4d84c44da9d9ba47f"}
Sep 09 04:23:21.815952 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.815884    2580 scope.go:117] "RemoveContainer" containerID="bf113e78db68dc0b65f963659953061cfe7dd95b750da647221330196d7cc665"
Sep 09 04:23:21.829095 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.829075    2580 scope.go:117] "RemoveContainer" containerID="563327a0ada10a5feeef13927e7e158e0eaac28e7a1b74c77d07f584098a4ad5"
Sep 09 04:23:21.836402 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.836381    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"]
Sep 09 04:23:21.839036 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.839023    2580 scope.go:117] "RemoveContainer" containerID="bf113e78db68dc0b65f963659953061cfe7dd95b750da647221330196d7cc665"
Sep 09 04:23:21.839367 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:23:21.839344    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf113e78db68dc0b65f963659953061cfe7dd95b750da647221330196d7cc665\": container with ID starting with bf113e78db68dc0b65f963659953061cfe7dd95b750da647221330196d7cc665 not found: ID does not exist" containerID="bf113e78db68dc0b65f963659953061cfe7dd95b750da647221330196d7cc665"
Sep 09 04:23:21.839436 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.839376    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf113e78db68dc0b65f963659953061cfe7dd95b750da647221330196d7cc665"} err="failed to get container status \"bf113e78db68dc0b65f963659953061cfe7dd95b750da647221330196d7cc665\": rpc error: code = NotFound desc = could not find container \"bf113e78db68dc0b65f963659953061cfe7dd95b750da647221330196d7cc665\": container with ID starting with bf113e78db68dc0b65f963659953061cfe7dd95b750da647221330196d7cc665 not found: ID does not exist"
Sep 09 04:23:21.839436 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.839392    2580 scope.go:117] "RemoveContainer" containerID="563327a0ada10a5feeef13927e7e158e0eaac28e7a1b74c77d07f584098a4ad5"
Sep 09 04:23:21.839701 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:23:21.839662    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563327a0ada10a5feeef13927e7e158e0eaac28e7a1b74c77d07f584098a4ad5\": container with ID starting with 563327a0ada10a5feeef13927e7e158e0eaac28e7a1b74c77d07f584098a4ad5 not found: ID does not exist" containerID="563327a0ada10a5feeef13927e7e158e0eaac28e7a1b74c77d07f584098a4ad5"
Sep 09 04:23:21.839701 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.839681    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563327a0ada10a5feeef13927e7e158e0eaac28e7a1b74c77d07f584098a4ad5"} err="failed to get container status \"563327a0ada10a5feeef13927e7e158e0eaac28e7a1b74c77d07f584098a4ad5\": rpc error: code = NotFound desc = could not find container \"563327a0ada10a5feeef13927e7e158e0eaac28e7a1b74c77d07f584098a4ad5\": container with ID starting with 563327a0ada10a5feeef13927e7e158e0eaac28e7a1b74c77d07f584098a4ad5 not found: ID does not exist"
Sep 09 04:23:21.842053 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:21.842035    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f95c96-openshift-default-7688d75c4d-vxkjf"]
Sep 09 04:23:23.596477 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:23.596448    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" path="/var/lib/kubelet/pods/87a7bcd9-7dc3-4bbc-8486-d203c199d256/volumes"
Sep 09 04:23:31.645046 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.645009    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"]
Sep 09 04:23:31.645592 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.645268    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w" podUID="bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9" containerName="maas-api" containerID="cri-o://ee65f6b010e7f6defe3d0fe9ab4ed7046786ae38b912977c9657cb40cab926c5" gracePeriod=30
Sep 09 04:23:31.887520 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.887495    2580 generic.go:356] "Generic (PLEG): container finished" podID="bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9" containerID="ee65f6b010e7f6defe3d0fe9ab4ed7046786ae38b912977c9657cb40cab926c5" exitCode=0
Sep 09 04:23:31.887653 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.887545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w" event={"ID":"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9","Type":"ContainerDied","Data":"ee65f6b010e7f6defe3d0fe9ab4ed7046786ae38b912977c9657cb40cab926c5"}
Sep 09 04:23:31.887653 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.887585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w" event={"ID":"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9","Type":"ContainerDied","Data":"0b3933f0ca4c81c7ccda264054ba70bda6c470c688d652071a5cfaa9ac87c09d"}
Sep 09 04:23:31.887653 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.887600    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b3933f0ca4c81c7ccda264054ba70bda6c470c688d652071a5cfaa9ac87c09d"
Sep 09 04:23:31.900943 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.900887    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:23:31.940027 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.939992    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-openshift-service-ca\") pod \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") "
Sep 09 04:23:31.940188 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.940050    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-metrics-certs\") pod \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") "
Sep 09 04:23:31.940188 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.940162    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zlxv\" (UniqueName: \"kubernetes.io/projected/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-kube-api-access-2zlxv\") pod \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") "
Sep 09 04:23:31.940311 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.940187    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-maas-api-tls\") pod \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\" (UID: \"bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9\") "
Sep 09 04:23:31.940370 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.940330    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9" (UID: "bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:23:31.940556 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.940538    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-openshift-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:31.942469 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.942434    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-kube-api-access-2zlxv" (OuterVolumeSpecName: "kube-api-access-2zlxv") pod "bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9" (UID: "bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9"). InnerVolumeSpecName "kube-api-access-2zlxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:23:31.942602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.942537    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9" (UID: "bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:23:31.942799 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:31.942744    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9" (UID: "bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:23:32.041660 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:32.041626    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:32.041660 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:32.041659    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zlxv\" (UniqueName: \"kubernetes.io/projected/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-kube-api-access-2zlxv\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:32.042012 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:32.041697    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9-maas-api-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:32.894047 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:32.894005    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"
Sep 09 04:23:32.918713 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:32.918684    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"]
Sep 09 04:23:32.922976 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:32.922952    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-acfe5b-5c8bf74f49-rrg5w"]
Sep 09 04:23:33.596309 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:33.596276    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9" path="/var/lib/kubelet/pods/bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9/volumes"
Sep 09 04:23:39.329625 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:39.329593    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"]
Sep 09 04:23:39.330134 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:39.329880    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" podUID="505aa41e-5775-4636-aa7d-fe082483949a" containerName="istio-proxy" containerID="cri-o://b3e778df9705f8215e08958091a55501be44628236958b5c0910a61f2845c68b" gracePeriod=30
Sep 09 04:23:43.167275 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:43.167232    2580 patch_prober.go:28] interesting pod/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 04:23:43.167844 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:43.167301    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" podUID="505aa41e-5775-4636-aa7d-fe082483949a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 04:23:44.593295 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.593244    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:23:44.660550 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.660520    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-credential-socket\") pod \"505aa41e-5775-4636-aa7d-fe082483949a\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") "
Sep 09 04:23:44.660550 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.660558    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505aa41e-5775-4636-aa7d-fe082483949a-istio-token\") pod \"505aa41e-5775-4636-aa7d-fe082483949a\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") "
Sep 09 04:23:44.660836 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.660596    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-istio-data\") pod \"505aa41e-5775-4636-aa7d-fe082483949a\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") "
Sep 09 04:23:44.660836 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.660614    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505aa41e-5775-4636-aa7d-fe082483949a-istio-podinfo\") pod \"505aa41e-5775-4636-aa7d-fe082483949a\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") "
Sep 09 04:23:44.660836 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.660636    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505aa41e-5775-4636-aa7d-fe082483949a-istiod-ca-cert\") pod \"505aa41e-5775-4636-aa7d-fe082483949a\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") "
Sep 09 04:23:44.660836 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.660683    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm5b7\" (UniqueName: \"kubernetes.io/projected/505aa41e-5775-4636-aa7d-fe082483949a-kube-api-access-rm5b7\") pod \"505aa41e-5775-4636-aa7d-fe082483949a\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") "
Sep 09 04:23:44.660836 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.660719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-workload-certs\") pod \"505aa41e-5775-4636-aa7d-fe082483949a\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") "
Sep 09 04:23:44.660836 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.660808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-workload-socket\") pod \"505aa41e-5775-4636-aa7d-fe082483949a\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") "
Sep 09 04:23:44.661197 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.660848    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-istio-envoy\") pod \"505aa41e-5775-4636-aa7d-fe082483949a\" (UID: \"505aa41e-5775-4636-aa7d-fe082483949a\") "
Sep 09 04:23:44.661197 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.660880    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "505aa41e-5775-4636-aa7d-fe082483949a" (UID: "505aa41e-5775-4636-aa7d-fe082483949a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:23:44.661197 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.660886    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-istio-data" (OuterVolumeSpecName: "istio-data") pod "505aa41e-5775-4636-aa7d-fe082483949a" (UID: "505aa41e-5775-4636-aa7d-fe082483949a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:23:44.661197 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.661059    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "505aa41e-5775-4636-aa7d-fe082483949a" (UID: "505aa41e-5775-4636-aa7d-fe082483949a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:23:44.661197 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.661108    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "505aa41e-5775-4636-aa7d-fe082483949a" (UID: "505aa41e-5775-4636-aa7d-fe082483949a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:23:44.661197 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.661156    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505aa41e-5775-4636-aa7d-fe082483949a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "505aa41e-5775-4636-aa7d-fe082483949a" (UID: "505aa41e-5775-4636-aa7d-fe082483949a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:23:44.661666 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.661275    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-workload-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:44.661666 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.661293    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-workload-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:44.661666 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.661307    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-credential-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:44.661666 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.661321    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-istio-data\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:44.661666 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.661335    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505aa41e-5775-4636-aa7d-fe082483949a-istiod-ca-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:44.663205 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.663176    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/505aa41e-5775-4636-aa7d-fe082483949a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "505aa41e-5775-4636-aa7d-fe082483949a" (UID: "505aa41e-5775-4636-aa7d-fe082483949a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:23:44.663205 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.663186    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "505aa41e-5775-4636-aa7d-fe082483949a" (UID: "505aa41e-5775-4636-aa7d-fe082483949a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:23:44.663650 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.663634    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505aa41e-5775-4636-aa7d-fe082483949a-kube-api-access-rm5b7" (OuterVolumeSpecName: "kube-api-access-rm5b7") pod "505aa41e-5775-4636-aa7d-fe082483949a" (UID: "505aa41e-5775-4636-aa7d-fe082483949a"). InnerVolumeSpecName "kube-api-access-rm5b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:23:44.663691 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.663661    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505aa41e-5775-4636-aa7d-fe082483949a-istio-token" (OuterVolumeSpecName: "istio-token") pod "505aa41e-5775-4636-aa7d-fe082483949a" (UID: "505aa41e-5775-4636-aa7d-fe082483949a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:23:44.762613 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.762581    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm5b7\" (UniqueName: \"kubernetes.io/projected/505aa41e-5775-4636-aa7d-fe082483949a-kube-api-access-rm5b7\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:44.762613 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.762607    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505aa41e-5775-4636-aa7d-fe082483949a-istio-envoy\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:44.762613 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.762617    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505aa41e-5775-4636-aa7d-fe082483949a-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:44.762858 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.762625    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505aa41e-5775-4636-aa7d-fe082483949a-istio-podinfo\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:44.976264 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.976183    2580 generic.go:356] "Generic (PLEG): container finished" podID="505aa41e-5775-4636-aa7d-fe082483949a" containerID="b3e778df9705f8215e08958091a55501be44628236958b5c0910a61f2845c68b" exitCode=0
Sep 09 04:23:44.976264 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.976221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" event={"ID":"505aa41e-5775-4636-aa7d-fe082483949a","Type":"ContainerDied","Data":"b3e778df9705f8215e08958091a55501be44628236958b5c0910a61f2845c68b"}
Sep 09 04:23:44.976264 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.976235    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"
Sep 09 04:23:44.976264 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.976264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn" event={"ID":"505aa41e-5775-4636-aa7d-fe082483949a","Type":"ContainerDied","Data":"be3f9561ffa32d9c5d3d6891aa7f00a3377d1639490a0f27079a3710e5cb441e"}
Sep 09 04:23:44.976554 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.976287    2580 scope.go:117] "RemoveContainer" containerID="b3e778df9705f8215e08958091a55501be44628236958b5c0910a61f2845c68b"
Sep 09 04:23:44.987287 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:44.987272    2580 scope.go:117] "RemoveContainer" containerID="61911cb036e9bb78dece2d96dabb091db0427af3cc0fb3c6f7eef8c47f41712b"
Sep 09 04:23:45.000255 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:45.000233    2580 scope.go:117] "RemoveContainer" containerID="b3e778df9705f8215e08958091a55501be44628236958b5c0910a61f2845c68b"
Sep 09 04:23:45.000530 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:23:45.000510    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e778df9705f8215e08958091a55501be44628236958b5c0910a61f2845c68b\": container with ID starting with b3e778df9705f8215e08958091a55501be44628236958b5c0910a61f2845c68b not found: ID does not exist" containerID="b3e778df9705f8215e08958091a55501be44628236958b5c0910a61f2845c68b"
Sep 09 04:23:45.000581 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:45.000542    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e778df9705f8215e08958091a55501be44628236958b5c0910a61f2845c68b"} err="failed to get container status \"b3e778df9705f8215e08958091a55501be44628236958b5c0910a61f2845c68b\": rpc error: code = NotFound desc = could not find container \"b3e778df9705f8215e08958091a55501be44628236958b5c0910a61f2845c68b\": container with ID starting with b3e778df9705f8215e08958091a55501be44628236958b5c0910a61f2845c68b not found: ID does not exist"
Sep 09 04:23:45.000581 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:45.000561    2580 scope.go:117] "RemoveContainer" containerID="61911cb036e9bb78dece2d96dabb091db0427af3cc0fb3c6f7eef8c47f41712b"
Sep 09 04:23:45.000843 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:23:45.000827    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61911cb036e9bb78dece2d96dabb091db0427af3cc0fb3c6f7eef8c47f41712b\": container with ID starting with 61911cb036e9bb78dece2d96dabb091db0427af3cc0fb3c6f7eef8c47f41712b not found: ID does not exist" containerID="61911cb036e9bb78dece2d96dabb091db0427af3cc0fb3c6f7eef8c47f41712b"
Sep 09 04:23:45.000912 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:45.000847    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61911cb036e9bb78dece2d96dabb091db0427af3cc0fb3c6f7eef8c47f41712b"} err="failed to get container status \"61911cb036e9bb78dece2d96dabb091db0427af3cc0fb3c6f7eef8c47f41712b\": rpc error: code = NotFound desc = could not find container \"61911cb036e9bb78dece2d96dabb091db0427af3cc0fb3c6f7eef8c47f41712b\": container with ID starting with 61911cb036e9bb78dece2d96dabb091db0427af3cc0fb3c6f7eef8c47f41712b not found: ID does not exist"
Sep 09 04:23:45.011325 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:45.011294    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"]
Sep 09 04:23:45.020679 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:45.020657    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-acfe5b-openshift-default-55bd4c7554-nvrdn"]
Sep 09 04:23:45.597676 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:45.597629    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="505aa41e-5775-4636-aa7d-fe082483949a" path="/var/lib/kubelet/pods/505aa41e-5775-4636-aa7d-fe082483949a/volumes"
Sep 09 04:23:50.509737 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:50.509699    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"]
Sep 09 04:23:50.510296 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:50.509999    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp" podUID="f19e7a5c-ad13-4f07-9292-317c53eefe03" containerName="maas-api" containerID="cri-o://5081c877a60a7581d62145a22bb19aaf10790b302e29c0372ba9780ad47b75cb" gracePeriod=30
Sep 09 04:23:50.771483 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:50.771415    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:23:50.923066 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:50.923033    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-maas-api-tls\") pod \"f19e7a5c-ad13-4f07-9292-317c53eefe03\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") "
Sep 09 04:23:50.923277 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:50.923153    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-metrics-certs\") pod \"f19e7a5c-ad13-4f07-9292-317c53eefe03\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") "
Sep 09 04:23:50.923606 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:50.923581    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kzdg\" (UniqueName: \"kubernetes.io/projected/f19e7a5c-ad13-4f07-9292-317c53eefe03-kube-api-access-6kzdg\") pod \"f19e7a5c-ad13-4f07-9292-317c53eefe03\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") "
Sep 09 04:23:50.924002 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:50.923975    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19e7a5c-ad13-4f07-9292-317c53eefe03-openshift-service-ca\") pod \"f19e7a5c-ad13-4f07-9292-317c53eefe03\" (UID: \"f19e7a5c-ad13-4f07-9292-317c53eefe03\") "
Sep 09 04:23:50.924705 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:50.924678    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19e7a5c-ad13-4f07-9292-317c53eefe03-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f19e7a5c-ad13-4f07-9292-317c53eefe03" (UID: "f19e7a5c-ad13-4f07-9292-317c53eefe03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:23:50.926440 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:50.926393    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f19e7a5c-ad13-4f07-9292-317c53eefe03" (UID: "f19e7a5c-ad13-4f07-9292-317c53eefe03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:23:50.926684 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:50.926658    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f19e7a5c-ad13-4f07-9292-317c53eefe03" (UID: "f19e7a5c-ad13-4f07-9292-317c53eefe03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:23:50.933487 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:50.933447    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19e7a5c-ad13-4f07-9292-317c53eefe03-kube-api-access-6kzdg" (OuterVolumeSpecName: "kube-api-access-6kzdg") pod "f19e7a5c-ad13-4f07-9292-317c53eefe03" (UID: "f19e7a5c-ad13-4f07-9292-317c53eefe03"). InnerVolumeSpecName "kube-api-access-6kzdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:23:51.018090 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.018056    2580 generic.go:356] "Generic (PLEG): container finished" podID="f19e7a5c-ad13-4f07-9292-317c53eefe03" containerID="5081c877a60a7581d62145a22bb19aaf10790b302e29c0372ba9780ad47b75cb" exitCode=0
Sep 09 04:23:51.018249 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.018122    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"
Sep 09 04:23:51.018249 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.018125    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp" event={"ID":"f19e7a5c-ad13-4f07-9292-317c53eefe03","Type":"ContainerDied","Data":"5081c877a60a7581d62145a22bb19aaf10790b302e29c0372ba9780ad47b75cb"}
Sep 09 04:23:51.018249 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.018245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp" event={"ID":"f19e7a5c-ad13-4f07-9292-317c53eefe03","Type":"ContainerDied","Data":"90121fe23e191545bac32501c4073459cf4cb1c82450924a2c037aca76514f62"}
Sep 09 04:23:51.018405 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.018266    2580 scope.go:117] "RemoveContainer" containerID="5081c877a60a7581d62145a22bb19aaf10790b302e29c0372ba9780ad47b75cb"
Sep 09 04:23:51.025174 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.025120    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19e7a5c-ad13-4f07-9292-317c53eefe03-openshift-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:51.025174 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.025143    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-maas-api-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:51.025174 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.025152    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f19e7a5c-ad13-4f07-9292-317c53eefe03-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:51.025174 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.025161    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzdg\" (UniqueName: \"kubernetes.io/projected/f19e7a5c-ad13-4f07-9292-317c53eefe03-kube-api-access-6kzdg\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:23:51.028069 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.028024    2580 scope.go:117] "RemoveContainer" containerID="5081c877a60a7581d62145a22bb19aaf10790b302e29c0372ba9780ad47b75cb"
Sep 09 04:23:51.028419 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:23:51.028370    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5081c877a60a7581d62145a22bb19aaf10790b302e29c0372ba9780ad47b75cb\": container with ID starting with 5081c877a60a7581d62145a22bb19aaf10790b302e29c0372ba9780ad47b75cb not found: ID does not exist" containerID="5081c877a60a7581d62145a22bb19aaf10790b302e29c0372ba9780ad47b75cb"
Sep 09 04:23:51.028523 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.028429    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5081c877a60a7581d62145a22bb19aaf10790b302e29c0372ba9780ad47b75cb"} err="failed to get container status \"5081c877a60a7581d62145a22bb19aaf10790b302e29c0372ba9780ad47b75cb\": rpc error: code = NotFound desc = could not find container \"5081c877a60a7581d62145a22bb19aaf10790b302e29c0372ba9780ad47b75cb\": container with ID starting with 5081c877a60a7581d62145a22bb19aaf10790b302e29c0372ba9780ad47b75cb not found: ID does not exist"
Sep 09 04:23:51.043740 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.043711    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"]
Sep 09 04:23:51.049431 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.049408    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-46ddac-c659667bf-7djqp"]
Sep 09 04:23:51.597353 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:51.597318    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f19e7a5c-ad13-4f07-9292-317c53eefe03" path="/var/lib/kubelet/pods/f19e7a5c-ad13-4f07-9292-317c53eefe03/volumes"
Sep 09 04:23:58.115050 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:58.115014    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"]
Sep 09 04:23:58.115509 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:58.115279    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts" podUID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerName="istio-proxy" containerID="cri-o://85cd24fcffc4240d5a9ad3988daf114e355f59f86ce0d43f62bc173afedb9d00" gracePeriod=30
Sep 09 04:23:59.885643 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:59.885613    2580 scope.go:117] "RemoveContainer" containerID="54da988c4d3bb016367af4caa1e6bc3d1d0b618bfb7d0b8b1f149a2e3aa0d9b8"
Sep 09 04:23:59.894429 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:59.894411    2580 scope.go:117] "RemoveContainer" containerID="c342fffcd68d0d684c2d1cb69227116e29b1885a03fd1375146269fb26aadf0c"
Sep 09 04:23:59.902954 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:23:59.902936    2580 scope.go:117] "RemoveContainer" containerID="3605db9041aa91deee497192adc16b4d1ae048a6f38b1b29116e5009f630bd61"
Sep 09 04:24:03.377421 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.377359    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:24:03.434988 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.434950    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s64qn\" (UniqueName: \"kubernetes.io/projected/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-kube-api-access-s64qn\") pod \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") "
Sep 09 04:24:03.435165 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.434997    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-podinfo\") pod \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") "
Sep 09 04:24:03.435165 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.435016    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-credential-socket\") pod \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") "
Sep 09 04:24:03.435165 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.435038    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-token\") pod \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") "
Sep 09 04:24:03.435303 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.435162    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-envoy\") pod \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") "
Sep 09 04:24:03.435303 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.435203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-data\") pod \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") "
Sep 09 04:24:03.435303 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.435231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istiod-ca-cert\") pod \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") "
Sep 09 04:24:03.435303 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.435272    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-workload-socket\") pod \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") "
Sep 09 04:24:03.435553 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.435370    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-workload-certs\") pod \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\" (UID: \"bca39590-7e3c-49d5-9b9e-14a78e7c26e8\") "
Sep 09 04:24:03.435740 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.435362    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bca39590-7e3c-49d5-9b9e-14a78e7c26e8" (UID: "bca39590-7e3c-49d5-9b9e-14a78e7c26e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:03.435920 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.435584    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bca39590-7e3c-49d5-9b9e-14a78e7c26e8" (UID: "bca39590-7e3c-49d5-9b9e-14a78e7c26e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:03.435920 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.435795    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bca39590-7e3c-49d5-9b9e-14a78e7c26e8" (UID: "bca39590-7e3c-49d5-9b9e-14a78e7c26e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:03.435920 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.435668    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bca39590-7e3c-49d5-9b9e-14a78e7c26e8" (UID: "bca39590-7e3c-49d5-9b9e-14a78e7c26e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:24:03.435920 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.435679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-data" (OuterVolumeSpecName: "istio-data") pod "bca39590-7e3c-49d5-9b9e-14a78e7c26e8" (UID: "bca39590-7e3c-49d5-9b9e-14a78e7c26e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:03.437420 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.437392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bca39590-7e3c-49d5-9b9e-14a78e7c26e8" (UID: "bca39590-7e3c-49d5-9b9e-14a78e7c26e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:24:03.437529 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.437439    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-token" (OuterVolumeSpecName: "istio-token") pod "bca39590-7e3c-49d5-9b9e-14a78e7c26e8" (UID: "bca39590-7e3c-49d5-9b9e-14a78e7c26e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:24:03.437901 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.437882    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-kube-api-access-s64qn" (OuterVolumeSpecName: "kube-api-access-s64qn") pod "bca39590-7e3c-49d5-9b9e-14a78e7c26e8" (UID: "bca39590-7e3c-49d5-9b9e-14a78e7c26e8"). InnerVolumeSpecName "kube-api-access-s64qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:24:03.437968 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.437894    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bca39590-7e3c-49d5-9b9e-14a78e7c26e8" (UID: "bca39590-7e3c-49d5-9b9e-14a78e7c26e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:03.536615 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.536584    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-workload-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:03.536615 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.536610    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s64qn\" (UniqueName: \"kubernetes.io/projected/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-kube-api-access-s64qn\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:03.536615 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.536621    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-podinfo\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:03.536887 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.536630    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-credential-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:03.536887 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.536638    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:03.536887 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.536645    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-envoy\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:03.536887 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.536653    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istio-data\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:03.536887 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.536659    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-istiod-ca-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:03.536887 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:03.536666    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca39590-7e3c-49d5-9b9e-14a78e7c26e8-workload-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:04.107497 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:04.107457    2580 generic.go:356] "Generic (PLEG): container finished" podID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerID="85cd24fcffc4240d5a9ad3988daf114e355f59f86ce0d43f62bc173afedb9d00" exitCode=0
Sep 09 04:24:04.107682 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:04.107513    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"
Sep 09 04:24:04.107682 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:04.107538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts" event={"ID":"bca39590-7e3c-49d5-9b9e-14a78e7c26e8","Type":"ContainerDied","Data":"85cd24fcffc4240d5a9ad3988daf114e355f59f86ce0d43f62bc173afedb9d00"}
Sep 09 04:24:04.107682 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:04.107624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts" event={"ID":"bca39590-7e3c-49d5-9b9e-14a78e7c26e8","Type":"ContainerDied","Data":"66f45239ae8e898cc613fd7ec7075ef2c601b53d557ab4a1c715d1394aaa40d1"}
Sep 09 04:24:04.107682 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:04.107653    2580 scope.go:117] "RemoveContainer" containerID="85cd24fcffc4240d5a9ad3988daf114e355f59f86ce0d43f62bc173afedb9d00"
Sep 09 04:24:04.117570 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:04.117551    2580 scope.go:117] "RemoveContainer" containerID="f827eec0183a7c59b4f0d90e7c273415d03df7e9ce1895fcca1aeed6f4cab64d"
Sep 09 04:24:04.127239 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:04.127220    2580 scope.go:117] "RemoveContainer" containerID="85cd24fcffc4240d5a9ad3988daf114e355f59f86ce0d43f62bc173afedb9d00"
Sep 09 04:24:04.127521 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:24:04.127501    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85cd24fcffc4240d5a9ad3988daf114e355f59f86ce0d43f62bc173afedb9d00\": container with ID starting with 85cd24fcffc4240d5a9ad3988daf114e355f59f86ce0d43f62bc173afedb9d00 not found: ID does not exist" containerID="85cd24fcffc4240d5a9ad3988daf114e355f59f86ce0d43f62bc173afedb9d00"
Sep 09 04:24:04.127609 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:04.127532    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85cd24fcffc4240d5a9ad3988daf114e355f59f86ce0d43f62bc173afedb9d00"} err="failed to get container status \"85cd24fcffc4240d5a9ad3988daf114e355f59f86ce0d43f62bc173afedb9d00\": rpc error: code = NotFound desc = could not find container \"85cd24fcffc4240d5a9ad3988daf114e355f59f86ce0d43f62bc173afedb9d00\": container with ID starting with 85cd24fcffc4240d5a9ad3988daf114e355f59f86ce0d43f62bc173afedb9d00 not found: ID does not exist"
Sep 09 04:24:04.127609 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:04.127555    2580 scope.go:117] "RemoveContainer" containerID="f827eec0183a7c59b4f0d90e7c273415d03df7e9ce1895fcca1aeed6f4cab64d"
Sep 09 04:24:04.127872 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:24:04.127852    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f827eec0183a7c59b4f0d90e7c273415d03df7e9ce1895fcca1aeed6f4cab64d\": container with ID starting with f827eec0183a7c59b4f0d90e7c273415d03df7e9ce1895fcca1aeed6f4cab64d not found: ID does not exist" containerID="f827eec0183a7c59b4f0d90e7c273415d03df7e9ce1895fcca1aeed6f4cab64d"
Sep 09 04:24:04.127930 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:04.127877    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f827eec0183a7c59b4f0d90e7c273415d03df7e9ce1895fcca1aeed6f4cab64d"} err="failed to get container status \"f827eec0183a7c59b4f0d90e7c273415d03df7e9ce1895fcca1aeed6f4cab64d\": rpc error: code = NotFound desc = could not find container \"f827eec0183a7c59b4f0d90e7c273415d03df7e9ce1895fcca1aeed6f4cab64d\": container with ID starting with f827eec0183a7c59b4f0d90e7c273415d03df7e9ce1895fcca1aeed6f4cab64d not found: ID does not exist"
Sep 09 04:24:04.130526 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:04.130507    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"]
Sep 09 04:24:04.135273 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:04.135253    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-46ddac-openshift-default-6fb7c65d6-8ddts"]
Sep 09 04:24:05.596958 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:05.596925    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" path="/var/lib/kubelet/pods/bca39590-7e3c-49d5-9b9e-14a78e7c26e8/volumes"
Sep 09 04:24:12.582576 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.582538    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7817-cc9d4d"]
Sep 09 04:24:12.582988 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.582973    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="505aa41e-5775-4636-aa7d-fe082483949a" containerName="istio-proxy"
Sep 09 04:24:12.582988 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.582989    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="505aa41e-5775-4636-aa7d-fe082483949a" containerName="istio-proxy"
Sep 09 04:24:12.583062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583000    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="505aa41e-5775-4636-aa7d-fe082483949a" containerName="istio-proxy"
Sep 09 04:24:12.583062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583005    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="505aa41e-5775-4636-aa7d-fe082483949a" containerName="istio-proxy"
Sep 09 04:24:12.583062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583013    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9" containerName="maas-api"
Sep 09 04:24:12.583062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583019    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9" containerName="maas-api"
Sep 09 04:24:12.583062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583027    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerName="istio-proxy"
Sep 09 04:24:12.583062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583033    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerName="istio-proxy"
Sep 09 04:24:12.583062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583041    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerName="istio-proxy"
Sep 09 04:24:12.583062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583047    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerName="istio-proxy"
Sep 09 04:24:12.583062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583055    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerName="istio-proxy"
Sep 09 04:24:12.583062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583060    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerName="istio-proxy"
Sep 09 04:24:12.583365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583067    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerName="istio-proxy"
Sep 09 04:24:12.583365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583073    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerName="istio-proxy"
Sep 09 04:24:12.583365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583087    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f19e7a5c-ad13-4f07-9292-317c53eefe03" containerName="maas-api"
Sep 09 04:24:12.583365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583092    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f19e7a5c-ad13-4f07-9292-317c53eefe03" containerName="maas-api"
Sep 09 04:24:12.583365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583147    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f19e7a5c-ad13-4f07-9292-317c53eefe03" containerName="maas-api"
Sep 09 04:24:12.583365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583156    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="505aa41e-5775-4636-aa7d-fe082483949a" containerName="istio-proxy"
Sep 09 04:24:12.583365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583162    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="505aa41e-5775-4636-aa7d-fe082483949a" containerName="istio-proxy"
Sep 09 04:24:12.583365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583167    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerName="istio-proxy"
Sep 09 04:24:12.583365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583174    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a7bcd9-7dc3-4bbc-8486-d203c199d256" containerName="istio-proxy"
Sep 09 04:24:12.583365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583181    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbdc20bb-a1fa-4c75-a363-e9ce0622f2b9" containerName="maas-api"
Sep 09 04:24:12.583365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583187    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerName="istio-proxy"
Sep 09 04:24:12.583365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.583192    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca39590-7e3c-49d5-9b9e-14a78e7c26e8" containerName="istio-proxy"
Sep 09 04:24:12.587834 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.587812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7817-cc9d4d"
Sep 09 04:24:12.590297 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.590274    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pcbxq\""
Sep 09 04:24:12.590927 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.590902    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7817-cc9d4d"]
Sep 09 04:24:12.619508 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.619472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscpt\" (UniqueName: \"kubernetes.io/projected/6b920959-6a84-4027-a118-04af542cecd6-kube-api-access-nscpt\") pod \"mt-curl-7817-cc9d4d\" (UID: \"6b920959-6a84-4027-a118-04af542cecd6\") " pod="opendatahub/mt-curl-7817-cc9d4d"
Sep 09 04:24:12.720835 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.720793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscpt\" (UniqueName: \"kubernetes.io/projected/6b920959-6a84-4027-a118-04af542cecd6-kube-api-access-nscpt\") pod \"mt-curl-7817-cc9d4d\" (UID: \"6b920959-6a84-4027-a118-04af542cecd6\") " pod="opendatahub/mt-curl-7817-cc9d4d"
Sep 09 04:24:12.729061 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.729033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscpt\" (UniqueName: \"kubernetes.io/projected/6b920959-6a84-4027-a118-04af542cecd6-kube-api-access-nscpt\") pod \"mt-curl-7817-cc9d4d\" (UID: \"6b920959-6a84-4027-a118-04af542cecd6\") " pod="opendatahub/mt-curl-7817-cc9d4d"
Sep 09 04:24:12.898662 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:12.898628    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7817-cc9d4d"
Sep 09 04:24:13.178635 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:13.178612    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7817-cc9d4d"]
Sep 09 04:24:13.180193 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:24:13.180167    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b920959_6a84_4027_a118_04af542cecd6.slice/crio-d542dbaa25b05cb7f5ac2026dc659ff6949ba5720964746dccb33569cb05ec43 WatchSource:0}: Error finding container d542dbaa25b05cb7f5ac2026dc659ff6949ba5720964746dccb33569cb05ec43: Status 404 returned error can't find the container with id d542dbaa25b05cb7f5ac2026dc659ff6949ba5720964746dccb33569cb05ec43
Sep 09 04:24:14.184411 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:14.184315    2580 generic.go:356] "Generic (PLEG): container finished" podID="6b920959-6a84-4027-a118-04af542cecd6" containerID="78a428e4a081b14b2d3b02ed0c7ba8041c177df1e82e1dcd370c701d8dc2d015" exitCode=0
Sep 09 04:24:14.184411 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:14.184382    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7817-cc9d4d" event={"ID":"6b920959-6a84-4027-a118-04af542cecd6","Type":"ContainerDied","Data":"78a428e4a081b14b2d3b02ed0c7ba8041c177df1e82e1dcd370c701d8dc2d015"}
Sep 09 04:24:14.184411 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:14.184409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7817-cc9d4d" event={"ID":"6b920959-6a84-4027-a118-04af542cecd6","Type":"ContainerStarted","Data":"d542dbaa25b05cb7f5ac2026dc659ff6949ba5720964746dccb33569cb05ec43"}
Sep 09 04:24:15.330985 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:15.330960    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7817-cc9d4d"
Sep 09 04:24:15.374874 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:15.374842    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7817-cc9d4d_6b920959-6a84-4027-a118-04af542cecd6/mt-curl-7817-cc9d4d/0.log"
Sep 09 04:24:15.401438 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:15.401407    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7817-cc9d4d"]
Sep 09 04:24:15.405742 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:15.405713    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7817-cc9d4d"]
Sep 09 04:24:15.447681 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:15.447606    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nscpt\" (UniqueName: \"kubernetes.io/projected/6b920959-6a84-4027-a118-04af542cecd6-kube-api-access-nscpt\") pod \"6b920959-6a84-4027-a118-04af542cecd6\" (UID: \"6b920959-6a84-4027-a118-04af542cecd6\") "
Sep 09 04:24:15.449923 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:15.449892    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b920959-6a84-4027-a118-04af542cecd6-kube-api-access-nscpt" (OuterVolumeSpecName: "kube-api-access-nscpt") pod "6b920959-6a84-4027-a118-04af542cecd6" (UID: "6b920959-6a84-4027-a118-04af542cecd6"). InnerVolumeSpecName "kube-api-access-nscpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:24:15.548650 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:15.548612    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nscpt\" (UniqueName: \"kubernetes.io/projected/6b920959-6a84-4027-a118-04af542cecd6-kube-api-access-nscpt\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:15.602400 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:15.596974    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b920959-6a84-4027-a118-04af542cecd6" path="/var/lib/kubelet/pods/6b920959-6a84-4027-a118-04af542cecd6/volumes"
Sep 09 04:24:16.132378 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:16.132343    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"]
Sep 09 04:24:16.132616 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:16.132593    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerName="istio-proxy" containerID="cri-o://6c39a43d8e9ca4a6b1ee15ead1090349e94bd9df21eafe2943f6b17892d0c381" gracePeriod=30
Sep 09 04:24:16.196840 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:16.196808    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7817-cc9d4d"
Sep 09 04:24:16.197008 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:16.196843    2580 scope.go:117] "RemoveContainer" containerID="78a428e4a081b14b2d3b02ed0c7ba8041c177df1e82e1dcd370c701d8dc2d015"
Sep 09 04:24:17.639979 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:17.639942    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 04:24:17.640351 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:17.639994    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 04:24:21.397908 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.397853    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:24:21.505323 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.505286    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-data\") pod \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") "
Sep 09 04:24:21.505498 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.505339    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-workload-certs\") pod \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") "
Sep 09 04:24:21.505498 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.505366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnk7f\" (UniqueName: \"kubernetes.io/projected/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-kube-api-access-hnk7f\") pod \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") "
Sep 09 04:24:21.505498 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.505396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istiod-ca-cert\") pod \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") "
Sep 09 04:24:21.505498 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.505450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-credential-socket\") pod \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") "
Sep 09 04:24:21.505498 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.505475    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-workload-socket\") pod \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") "
Sep 09 04:24:21.505787 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.505509    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-podinfo\") pod \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") "
Sep 09 04:24:21.505787 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.505533    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-token\") pod \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") "
Sep 09 04:24:21.505787 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.505584    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-envoy\") pod \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\" (UID: \"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e\") "
Sep 09 04:24:21.505787 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.505592    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" (UID: "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:21.505787 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.505667    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" (UID: "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:21.506048 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.505957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" (UID: "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:21.506048 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.505987    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" (UID: "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:24:21.506048 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.506028    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-data\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:21.506048 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.506047    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-workload-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:21.506298 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.506276    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" (UID: "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:21.508816 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.508789    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" (UID: "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:24:21.508816 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.508800    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" (UID: "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:24:21.508972 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.508790    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" (UID: "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:24:21.508972 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.508876    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-kube-api-access-hnk7f" (OuterVolumeSpecName: "kube-api-access-hnk7f") pod "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" (UID: "f2b35da7-d29f-4d0c-9d43-ba3c02ef203e"). InnerVolumeSpecName "kube-api-access-hnk7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:24:21.607069 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.607046    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-podinfo\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:21.607069 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.607069    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:21.607244 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.607078    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istio-envoy\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:21.607244 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.607086    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnk7f\" (UniqueName: \"kubernetes.io/projected/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-kube-api-access-hnk7f\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:21.607244 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.607094    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-istiod-ca-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:21.607244 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.607103    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-credential-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:21.607244 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:21.607110    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e-workload-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:24:22.237175 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:22.237143    2580 generic.go:356] "Generic (PLEG): container finished" podID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerID="6c39a43d8e9ca4a6b1ee15ead1090349e94bd9df21eafe2943f6b17892d0c381" exitCode=0
Sep 09 04:24:22.237349 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:22.237199    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"
Sep 09 04:24:22.237349 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:22.237213    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" event={"ID":"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e","Type":"ContainerDied","Data":"6c39a43d8e9ca4a6b1ee15ead1090349e94bd9df21eafe2943f6b17892d0c381"}
Sep 09 04:24:22.237349 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:22.237268    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w" event={"ID":"f2b35da7-d29f-4d0c-9d43-ba3c02ef203e","Type":"ContainerDied","Data":"2200d72ec268896941d558a44d05f9738d42c042e8d6d6d7d3750a2d15d0eaab"}
Sep 09 04:24:22.237349 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:22.237300    2580 scope.go:117] "RemoveContainer" containerID="6c39a43d8e9ca4a6b1ee15ead1090349e94bd9df21eafe2943f6b17892d0c381"
Sep 09 04:24:22.247171 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:22.247149    2580 scope.go:117] "RemoveContainer" containerID="e3a1358bcf93a39e60e5559ad2023988576b812d823461a790a5b7c578678d42"
Sep 09 04:24:22.257376 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:22.257200    2580 scope.go:117] "RemoveContainer" containerID="6c39a43d8e9ca4a6b1ee15ead1090349e94bd9df21eafe2943f6b17892d0c381"
Sep 09 04:24:22.257481 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:24:22.257460    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c39a43d8e9ca4a6b1ee15ead1090349e94bd9df21eafe2943f6b17892d0c381\": container with ID starting with 6c39a43d8e9ca4a6b1ee15ead1090349e94bd9df21eafe2943f6b17892d0c381 not found: ID does not exist" containerID="6c39a43d8e9ca4a6b1ee15ead1090349e94bd9df21eafe2943f6b17892d0c381"
Sep 09 04:24:22.257526 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:22.257491    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c39a43d8e9ca4a6b1ee15ead1090349e94bd9df21eafe2943f6b17892d0c381"} err="failed to get container status \"6c39a43d8e9ca4a6b1ee15ead1090349e94bd9df21eafe2943f6b17892d0c381\": rpc error: code = NotFound desc = could not find container \"6c39a43d8e9ca4a6b1ee15ead1090349e94bd9df21eafe2943f6b17892d0c381\": container with ID starting with 6c39a43d8e9ca4a6b1ee15ead1090349e94bd9df21eafe2943f6b17892d0c381 not found: ID does not exist"
Sep 09 04:24:22.257526 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:22.257507    2580 scope.go:117] "RemoveContainer" containerID="e3a1358bcf93a39e60e5559ad2023988576b812d823461a790a5b7c578678d42"
Sep 09 04:24:22.257897 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:24:22.257872    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a1358bcf93a39e60e5559ad2023988576b812d823461a790a5b7c578678d42\": container with ID starting with e3a1358bcf93a39e60e5559ad2023988576b812d823461a790a5b7c578678d42 not found: ID does not exist" containerID="e3a1358bcf93a39e60e5559ad2023988576b812d823461a790a5b7c578678d42"
Sep 09 04:24:22.258004 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:22.257907    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a1358bcf93a39e60e5559ad2023988576b812d823461a790a5b7c578678d42"} err="failed to get container status \"e3a1358bcf93a39e60e5559ad2023988576b812d823461a790a5b7c578678d42\": rpc error: code = NotFound desc = could not find container \"e3a1358bcf93a39e60e5559ad2023988576b812d823461a790a5b7c578678d42\": container with ID starting with e3a1358bcf93a39e60e5559ad2023988576b812d823461a790a5b7c578678d42 not found: ID does not exist"
Sep 09 04:24:22.258477 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:22.258454    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"]
Sep 09 04:24:22.262970 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:22.262937    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9f2be5-openshift-default-564df656c8-27v5w"]
Sep 09 04:24:23.596249 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:24:23.596221    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" path="/var/lib/kubelet/pods/f2b35da7-d29f-4d0c-9d43-ba3c02ef203e/volumes"
Sep 09 04:26:07.450258 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.450222    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7817-0aa68e"]
Sep 09 04:26:07.450699 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.450595    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerName="istio-proxy"
Sep 09 04:26:07.450699 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.450606    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerName="istio-proxy"
Sep 09 04:26:07.450699 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.450616    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b920959-6a84-4027-a118-04af542cecd6" containerName="mt-curl-7817-cc9d4d"
Sep 09 04:26:07.450699 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.450621    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b920959-6a84-4027-a118-04af542cecd6" containerName="mt-curl-7817-cc9d4d"
Sep 09 04:26:07.450699 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.450635    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerName="istio-proxy"
Sep 09 04:26:07.450699 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.450641    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerName="istio-proxy"
Sep 09 04:26:07.450921 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.450704    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerName="istio-proxy"
Sep 09 04:26:07.450921 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.450714    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b920959-6a84-4027-a118-04af542cecd6" containerName="mt-curl-7817-cc9d4d"
Sep 09 04:26:07.450921 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.450720    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b35da7-d29f-4d0c-9d43-ba3c02ef203e" containerName="istio-proxy"
Sep 09 04:26:07.453824 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.453805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7817-0aa68e"
Sep 09 04:26:07.456385 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.456359    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pcbxq\""
Sep 09 04:26:07.462083 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.462060    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7817-0aa68e"]
Sep 09 04:26:07.569671 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.569640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkhwr\" (UniqueName: \"kubernetes.io/projected/85df3e77-9f2f-4313-81fc-52c6e7ffb028-kube-api-access-bkhwr\") pod \"mt-curl-7817-0aa68e\" (UID: \"85df3e77-9f2f-4313-81fc-52c6e7ffb028\") " pod="opendatahub/mt-curl-7817-0aa68e"
Sep 09 04:26:07.670499 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.670471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkhwr\" (UniqueName: \"kubernetes.io/projected/85df3e77-9f2f-4313-81fc-52c6e7ffb028-kube-api-access-bkhwr\") pod \"mt-curl-7817-0aa68e\" (UID: \"85df3e77-9f2f-4313-81fc-52c6e7ffb028\") " pod="opendatahub/mt-curl-7817-0aa68e"
Sep 09 04:26:07.679609 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.679585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkhwr\" (UniqueName: \"kubernetes.io/projected/85df3e77-9f2f-4313-81fc-52c6e7ffb028-kube-api-access-bkhwr\") pod \"mt-curl-7817-0aa68e\" (UID: \"85df3e77-9f2f-4313-81fc-52c6e7ffb028\") " pod="opendatahub/mt-curl-7817-0aa68e"
Sep 09 04:26:07.764365 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:07.764294    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7817-0aa68e"
Sep 09 04:26:08.040928 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:08.040899    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7817-0aa68e"]
Sep 09 04:26:08.042068 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:26:08.042037    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85df3e77_9f2f_4313_81fc_52c6e7ffb028.slice/crio-308b3f8a30b741689b5d990a632d37ac9ebfbd7e93675c1217c07ea2a9d7c801 WatchSource:0}: Error finding container 308b3f8a30b741689b5d990a632d37ac9ebfbd7e93675c1217c07ea2a9d7c801: Status 404 returned error can't find the container with id 308b3f8a30b741689b5d990a632d37ac9ebfbd7e93675c1217c07ea2a9d7c801
Sep 09 04:26:08.886487 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:08.886460    2580 generic.go:356] "Generic (PLEG): container finished" podID="85df3e77-9f2f-4313-81fc-52c6e7ffb028" containerID="e48dd4985d810258ff9fce3da12bd3a6943754a0c0b1242bb22673c86282bcc4" exitCode=0
Sep 09 04:26:08.886904 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:08.886553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7817-0aa68e" event={"ID":"85df3e77-9f2f-4313-81fc-52c6e7ffb028","Type":"ContainerDied","Data":"e48dd4985d810258ff9fce3da12bd3a6943754a0c0b1242bb22673c86282bcc4"}
Sep 09 04:26:08.886904 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:08.886603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7817-0aa68e" event={"ID":"85df3e77-9f2f-4313-81fc-52c6e7ffb028","Type":"ContainerStarted","Data":"308b3f8a30b741689b5d990a632d37ac9ebfbd7e93675c1217c07ea2a9d7c801"}
Sep 09 04:26:10.029640 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.029614    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7817-0aa68e"
Sep 09 04:26:10.062402 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.062351    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7817-0aa68e_85df3e77-9f2f-4313-81fc-52c6e7ffb028/mt-curl-7817-0aa68e/0.log"
Sep 09 04:26:10.090356 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.090330    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7817-0aa68e"]
Sep 09 04:26:10.097021 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.096995    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7817-0aa68e"]
Sep 09 04:26:10.197912 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.197826    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkhwr\" (UniqueName: \"kubernetes.io/projected/85df3e77-9f2f-4313-81fc-52c6e7ffb028-kube-api-access-bkhwr\") pod \"85df3e77-9f2f-4313-81fc-52c6e7ffb028\" (UID: \"85df3e77-9f2f-4313-81fc-52c6e7ffb028\") "
Sep 09 04:26:10.200079 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.200052    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85df3e77-9f2f-4313-81fc-52c6e7ffb028-kube-api-access-bkhwr" (OuterVolumeSpecName: "kube-api-access-bkhwr") pod "85df3e77-9f2f-4313-81fc-52c6e7ffb028" (UID: "85df3e77-9f2f-4313-81fc-52c6e7ffb028"). InnerVolumeSpecName "kube-api-access-bkhwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:26:10.218292 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.218260    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7817-22f6b0"]
Sep 09 04:26:10.218660 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.218648    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85df3e77-9f2f-4313-81fc-52c6e7ffb028" containerName="mt-curl-7817-0aa68e"
Sep 09 04:26:10.218711 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.218662    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85df3e77-9f2f-4313-81fc-52c6e7ffb028" containerName="mt-curl-7817-0aa68e"
Sep 09 04:26:10.218749 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.218733    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85df3e77-9f2f-4313-81fc-52c6e7ffb028" containerName="mt-curl-7817-0aa68e"
Sep 09 04:26:10.221820 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.221805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7817-22f6b0"
Sep 09 04:26:10.228928 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.228877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7817-22f6b0"]
Sep 09 04:26:10.298700 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.298664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8hrj\" (UniqueName: \"kubernetes.io/projected/6d495033-88c6-4adb-a9ef-b0f5bfbeea02-kube-api-access-h8hrj\") pod \"mt-curl-7817-22f6b0\" (UID: \"6d495033-88c6-4adb-a9ef-b0f5bfbeea02\") " pod="opendatahub/mt-curl-7817-22f6b0"
Sep 09 04:26:10.298892 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.298736    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkhwr\" (UniqueName: \"kubernetes.io/projected/85df3e77-9f2f-4313-81fc-52c6e7ffb028-kube-api-access-bkhwr\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:26:10.399384 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.399356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8hrj\" (UniqueName: \"kubernetes.io/projected/6d495033-88c6-4adb-a9ef-b0f5bfbeea02-kube-api-access-h8hrj\") pod \"mt-curl-7817-22f6b0\" (UID: \"6d495033-88c6-4adb-a9ef-b0f5bfbeea02\") " pod="opendatahub/mt-curl-7817-22f6b0"
Sep 09 04:26:10.408841 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.408813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8hrj\" (UniqueName: \"kubernetes.io/projected/6d495033-88c6-4adb-a9ef-b0f5bfbeea02-kube-api-access-h8hrj\") pod \"mt-curl-7817-22f6b0\" (UID: \"6d495033-88c6-4adb-a9ef-b0f5bfbeea02\") " pod="opendatahub/mt-curl-7817-22f6b0"
Sep 09 04:26:10.532238 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.532163    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7817-22f6b0"
Sep 09 04:26:10.815672 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.815646    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7817-22f6b0"]
Sep 09 04:26:10.816874 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:26:10.816844    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d495033_88c6_4adb_a9ef_b0f5bfbeea02.slice/crio-443eb9c0147370a81c9d23c435e751690cff493779d11984c8a04e62a53be702 WatchSource:0}: Error finding container 443eb9c0147370a81c9d23c435e751690cff493779d11984c8a04e62a53be702: Status 404 returned error can't find the container with id 443eb9c0147370a81c9d23c435e751690cff493779d11984c8a04e62a53be702
Sep 09 04:26:10.899061 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.899036    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7817-0aa68e"
Sep 09 04:26:10.899198 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.899073    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="308b3f8a30b741689b5d990a632d37ac9ebfbd7e93675c1217c07ea2a9d7c801"
Sep 09 04:26:10.900478 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:10.900456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7817-22f6b0" event={"ID":"6d495033-88c6-4adb-a9ef-b0f5bfbeea02","Type":"ContainerStarted","Data":"443eb9c0147370a81c9d23c435e751690cff493779d11984c8a04e62a53be702"}
Sep 09 04:26:10.911483 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:26:10.911459    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7817-0aa68e\" is forbidden: User \"system:node:ip-10-0-142-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-198.ec2.internal' and this object" podUID="85df3e77-9f2f-4313-81fc-52c6e7ffb028" pod="opendatahub/mt-curl-7817-0aa68e"
Sep 09 04:26:11.595722 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:11.595694    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85df3e77-9f2f-4313-81fc-52c6e7ffb028" path="/var/lib/kubelet/pods/85df3e77-9f2f-4313-81fc-52c6e7ffb028/volumes"
Sep 09 04:26:11.908428 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:11.908331    2580 generic.go:356] "Generic (PLEG): container finished" podID="6d495033-88c6-4adb-a9ef-b0f5bfbeea02" containerID="459595ed850b833e3ed81f9f3a91c59a86cf1d7a59743d8afea1e43a65fc85b9" exitCode=0
Sep 09 04:26:11.908605 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:11.908417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7817-22f6b0" event={"ID":"6d495033-88c6-4adb-a9ef-b0f5bfbeea02","Type":"ContainerDied","Data":"459595ed850b833e3ed81f9f3a91c59a86cf1d7a59743d8afea1e43a65fc85b9"}
Sep 09 04:26:13.054217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:13.054195    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7817-22f6b0"
Sep 09 04:26:13.074811 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:13.074789    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7817-22f6b0_6d495033-88c6-4adb-a9ef-b0f5bfbeea02/mt-curl-7817-22f6b0/0.log"
Sep 09 04:26:13.100246 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:13.100217    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7817-22f6b0"]
Sep 09 04:26:13.106156 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:13.106131    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7817-22f6b0"]
Sep 09 04:26:13.229183 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:13.229104    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8hrj\" (UniqueName: \"kubernetes.io/projected/6d495033-88c6-4adb-a9ef-b0f5bfbeea02-kube-api-access-h8hrj\") pod \"6d495033-88c6-4adb-a9ef-b0f5bfbeea02\" (UID: \"6d495033-88c6-4adb-a9ef-b0f5bfbeea02\") "
Sep 09 04:26:13.231332 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:13.231305    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d495033-88c6-4adb-a9ef-b0f5bfbeea02-kube-api-access-h8hrj" (OuterVolumeSpecName: "kube-api-access-h8hrj") pod "6d495033-88c6-4adb-a9ef-b0f5bfbeea02" (UID: "6d495033-88c6-4adb-a9ef-b0f5bfbeea02"). InnerVolumeSpecName "kube-api-access-h8hrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:26:13.330614 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:13.330580    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8hrj\" (UniqueName: \"kubernetes.io/projected/6d495033-88c6-4adb-a9ef-b0f5bfbeea02-kube-api-access-h8hrj\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:26:13.595999 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:13.595971    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d495033-88c6-4adb-a9ef-b0f5bfbeea02" path="/var/lib/kubelet/pods/6d495033-88c6-4adb-a9ef-b0f5bfbeea02/volumes"
Sep 09 04:26:13.920728 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:13.920645    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7817-22f6b0"
Sep 09 04:26:13.924269 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:26:13.920676    2580 scope.go:117] "RemoveContainer" containerID="459595ed850b833e3ed81f9f3a91c59a86cf1d7a59743d8afea1e43a65fc85b9"
Sep 09 04:27:52.072485 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.072447    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"]
Sep 09 04:27:52.075094 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.072859    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d495033-88c6-4adb-a9ef-b0f5bfbeea02" containerName="mt-curl-7817-22f6b0"
Sep 09 04:27:52.075094 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.072872    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d495033-88c6-4adb-a9ef-b0f5bfbeea02" containerName="mt-curl-7817-22f6b0"
Sep 09 04:27:52.075094 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.072971    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d495033-88c6-4adb-a9ef-b0f5bfbeea02" containerName="mt-curl-7817-22f6b0"
Sep 09 04:27:52.075942 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.075922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.078488 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.078469    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a3ce90-openshift-default-dockercfg-7rv7z\""
Sep 09 04:27:52.090326 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.090301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"]
Sep 09 04:27:52.129262 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.129233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istiod-ca-cert\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.129435 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.129268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-podinfo\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.129435 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.129312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-token\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.129435 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.129391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-credential-socket\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.129607 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.129445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-data\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.129607 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.129482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc9sn\" (UniqueName: \"kubernetes.io/projected/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-kube-api-access-zc9sn\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.129607 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.129517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-envoy\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.129607 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.129578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-workload-socket\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.129796 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.129617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-workload-certs\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.230875 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.230837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-workload-socket\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.230875 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.230880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-workload-certs\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.231126 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.230917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istiod-ca-cert\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.231126 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.230947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-podinfo\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.231126 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.231004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-token\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.231279 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.231141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-credential-socket\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.231279 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.231205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-data\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.231279 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.231247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc9sn\" (UniqueName: \"kubernetes.io/projected/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-kube-api-access-zc9sn\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.231438 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.231284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-envoy\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.231438 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.231319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-workload-certs\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.231438 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.231363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-workload-socket\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.231609 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.231552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-data\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.231664 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.231631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-credential-socket\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.231732 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.231713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istiod-ca-cert\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.233615 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.233592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-envoy\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.233847 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.233816    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-podinfo\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.240701 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.240672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-token\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.240865 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.240841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc9sn\" (UniqueName: \"kubernetes.io/projected/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-kube-api-access-zc9sn\") pod \"e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") " pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.387180 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.387147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:52.673728 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.673698    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"]
Sep 09 04:27:52.674704 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:27:52.674677    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5810775_3101_4fdd_b58b_9c3ce7e5ae7e.slice/crio-10ed5fb4c6569d4a7c81a4c053663ce4073b6dd458bd68ed251fdfed9487b333 WatchSource:0}: Error finding container 10ed5fb4c6569d4a7c81a4c053663ce4073b6dd458bd68ed251fdfed9487b333: Status 404 returned error can't find the container with id 10ed5fb4c6569d4a7c81a4c053663ce4073b6dd458bd68ed251fdfed9487b333
Sep 09 04:27:52.677197 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.677164    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:27:52.677297 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.677229    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:27:52.677297 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:52.677256    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:27:53.530942 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:53.530908    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" event={"ID":"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e","Type":"ContainerStarted","Data":"f8dff45b09b8ebdfac59007e3b9892e5937cc17b1c16a13e5f9a85e4c473a4b4"}
Sep 09 04:27:53.531331 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:53.530961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" event={"ID":"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e","Type":"ContainerStarted","Data":"10ed5fb4c6569d4a7c81a4c053663ce4073b6dd458bd68ed251fdfed9487b333"}
Sep 09 04:27:53.557522 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:53.557475    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" podStartSLOduration=1.55746155 podStartE2EDuration="1.55746155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:27:53.555975349 +0000 UTC m=+1794.505666016" watchObservedRunningTime="2026-09-09 04:27:53.55746155 +0000 UTC m=+1794.507152204"
Sep 09 04:27:54.387966 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:54.387929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:54.393665 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:54.393639    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:54.537246 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:54.537214    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:54.538205 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:54.538186    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:27:59.695944 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:59.695914    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:27:59.705128 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:27:59.705101    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:28:10.538299 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:10.538251    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 04:28:10.538882 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:10.538325    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 04:28:12.655147 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:12.655115    2580 generic.go:356] "Generic (PLEG): container finished" podID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerID="f8dff45b09b8ebdfac59007e3b9892e5937cc17b1c16a13e5f9a85e4c473a4b4" exitCode=0
Sep 09 04:28:12.655562 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:12.655190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" event={"ID":"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e","Type":"ContainerDied","Data":"f8dff45b09b8ebdfac59007e3b9892e5937cc17b1c16a13e5f9a85e4c473a4b4"}
Sep 09 04:28:12.655606 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:12.655587    2580 scope.go:117] "RemoveContainer" containerID="f8dff45b09b8ebdfac59007e3b9892e5937cc17b1c16a13e5f9a85e4c473a4b4"
Sep 09 04:28:12.656721 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:12.656689    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:28:12.656878 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:12.656808    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:28:12.656878 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:12.656859    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:28:13.663730 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:13.663697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" event={"ID":"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e","Type":"ContainerStarted","Data":"61917a287e913bd645cec1b82f7bca79117df2e0b0e91665136da752686e2449"}
Sep 09 04:28:14.387851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:14.387807    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:28:14.388099 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:14.388076    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 09 04:28:14.388161 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:14.388140    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 09 04:28:15.388425 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:15.388392    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 09 04:28:15.388846 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:15.388447    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 09 04:28:16.387639 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:16.387602    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 09 04:28:16.387853 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:16.387654    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 09 04:28:17.387498 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:17.387461    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 09 04:28:17.387906 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:17.387517    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 09 04:28:18.388441 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:18.388408    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Sep 09 04:28:18.388856 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:18.388477    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 09 04:28:19.391023 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:19.390990    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:28:19.391478 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:19.391451    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:28:19.392050 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:19.392033    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:28:25.503995 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.503964    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"]
Sep 09 04:28:25.508446 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.508421    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:25.511080 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.511055    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0db87d-serving-cert\""
Sep 09 04:28:25.523017 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.522988    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"]
Sep 09 04:28:25.649779 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.649728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:25.649779 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.649781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c129ed2e-f4df-4123-8c67-86c27c2b04fa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:25.650047 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.649876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccd7w\" (UniqueName: \"kubernetes.io/projected/c129ed2e-f4df-4123-8c67-86c27c2b04fa-kube-api-access-ccd7w\") pod \"maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:25.650047 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.649925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-metrics-certs\") pod \"maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:25.750846 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.750813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:25.750846 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.750849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c129ed2e-f4df-4123-8c67-86c27c2b04fa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:25.751109 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:28:25.750964    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-serving-cert: secret "maas-api-e2e-ipp-b-0db87d-serving-cert" not found
Sep 09 04:28:25.751109 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:28:25.751017    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-maas-api-tls podName:c129ed2e-f4df-4123-8c67-86c27c2b04fa nodeName:}" failed. No retries permitted until 2026-09-09 04:28:26.251002169 +0000 UTC m=+1827.200692800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-maas-api-tls") pod "maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk" (UID: "c129ed2e-f4df-4123-8c67-86c27c2b04fa") : secret "maas-api-e2e-ipp-b-0db87d-serving-cert" not found
Sep 09 04:28:25.751109 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.751052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccd7w\" (UniqueName: \"kubernetes.io/projected/c129ed2e-f4df-4123-8c67-86c27c2b04fa-kube-api-access-ccd7w\") pod \"maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:25.751274 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.751151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-metrics-certs\") pod \"maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:25.751677 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.751659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c129ed2e-f4df-4123-8c67-86c27c2b04fa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:25.753710 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.753687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-metrics-certs\") pod \"maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:25.759312 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:25.759261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccd7w\" (UniqueName: \"kubernetes.io/projected/c129ed2e-f4df-4123-8c67-86c27c2b04fa-kube-api-access-ccd7w\") pod \"maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:26.255869 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:26.255834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:26.258539 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:26.258511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:26.430184 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:26.430151    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:26.712445 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:26.712416    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"]
Sep 09 04:28:26.714343 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:28:26.714318    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc129ed2e_f4df_4123_8c67_86c27c2b04fa.slice/crio-b7221aa8a4fffc5844f3928cb28037522e117f39b5348653c16a78dc65048067 WatchSource:0}: Error finding container b7221aa8a4fffc5844f3928cb28037522e117f39b5348653c16a78dc65048067: Status 404 returned error can't find the container with id b7221aa8a4fffc5844f3928cb28037522e117f39b5348653c16a78dc65048067
Sep 09 04:28:26.715753 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:26.715739    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 04:28:26.748032 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:26.748004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk" event={"ID":"c129ed2e-f4df-4123-8c67-86c27c2b04fa","Type":"ContainerStarted","Data":"b7221aa8a4fffc5844f3928cb28037522e117f39b5348653c16a78dc65048067"}
Sep 09 04:28:27.756234 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:27.756197    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk" event={"ID":"c129ed2e-f4df-4123-8c67-86c27c2b04fa","Type":"ContainerStarted","Data":"122788dd5a06993c14fd001d6bc2e50f4d767624a79a7c72e08a525dfc05874f"}
Sep 09 04:28:27.756623 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:27.756259    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:28:27.775523 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:27.775481    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk" podStartSLOduration=2.453035829 podStartE2EDuration="2.77546718s" totalImagesPullingTime="322.431351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:28:27.772755755 +0000 UTC m=+1828.722446422" watchObservedRunningTime="2026-09-09 04:28:27.77546718 +0000 UTC m=+1828.725157834"
Sep 09 04:28:33.768216 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:28:33.768185    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:29:00.004014 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:00.003973    2580 scope.go:117] "RemoveContainer" containerID="ee65f6b010e7f6defe3d0fe9ab4ed7046786ae38b912977c9657cb40cab926c5"
Sep 09 04:29:31.157290 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.156937    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"]
Sep 09 04:29:31.160795 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.160758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.163342 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.163313    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-e787bf-openshift-default-dockercfg-nrmvn\""
Sep 09 04:29:31.173045 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.173016    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"]
Sep 09 04:29:31.176686 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.176660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449e7ee8-cbb7-4103-867d-c22889f8c391-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.176801 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.176732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-podinfo\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.176854 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.176810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-envoy\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.176854 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.176837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvn7\" (UniqueName: \"kubernetes.io/projected/449e7ee8-cbb7-4103-867d-c22889f8c391-kube-api-access-5lvn7\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.176938 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.176872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-data\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.176938 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.176900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-workload-socket\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.177044 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.176950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-token\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.177044 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.176994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-credential-socket\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.177044 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.177026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-workload-certs\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.277957 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.277925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-credential-socket\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.278159 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.277981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-workload-certs\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.278159 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.278037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449e7ee8-cbb7-4103-867d-c22889f8c391-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.278159 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.278090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-podinfo\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.278159 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.278131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-envoy\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.278159 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.278153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvn7\" (UniqueName: \"kubernetes.io/projected/449e7ee8-cbb7-4103-867d-c22889f8c391-kube-api-access-5lvn7\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.278430 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.278185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-data\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.278430 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.278216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-workload-socket\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.278430 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.278253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-token\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.278430 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.278365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-credential-socket\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.278670 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.278642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-data\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.278823 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.278680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-workload-certs\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.278823 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.278755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-workload-socket\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.279252 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.279231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449e7ee8-cbb7-4103-867d-c22889f8c391-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.280454 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.280433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-envoy\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.280704 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.280687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-podinfo\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.287012 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.286989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-token\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.287096 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.287077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvn7\" (UniqueName: \"kubernetes.io/projected/449e7ee8-cbb7-4103-867d-c22889f8c391-kube-api-access-5lvn7\") pod \"e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") " pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.474389 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.474296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:31.756459 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.756428    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"]
Sep 09 04:29:31.757397 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:29:31.757373    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449e7ee8_cbb7_4103_867d_c22889f8c391.slice/crio-29d607da3e85db168d320c9219562564c6eddda32ce68a07f792d592fbd4c711 WatchSource:0}: Error finding container 29d607da3e85db168d320c9219562564c6eddda32ce68a07f792d592fbd4c711: Status 404 returned error can't find the container with id 29d607da3e85db168d320c9219562564c6eddda32ce68a07f792d592fbd4c711
Sep 09 04:29:31.760623 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.760578    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:29:31.760709 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.760681    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:29:31.760758 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:31.760715    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:29:32.151841 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:32.151806    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" event={"ID":"449e7ee8-cbb7-4103-867d-c22889f8c391","Type":"ContainerStarted","Data":"c43c6cdf4d3384d350db1b5435bdfb7261229f57e0bc8ac0d89a2735b2d66c18"}
Sep 09 04:29:32.152054 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:32.151862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" event={"ID":"449e7ee8-cbb7-4103-867d-c22889f8c391","Type":"ContainerStarted","Data":"29d607da3e85db168d320c9219562564c6eddda32ce68a07f792d592fbd4c711"}
Sep 09 04:29:32.173534 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:32.173488    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" podStartSLOduration=1.173474355 podStartE2EDuration="1.173474355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:29:32.17244305 +0000 UTC m=+1893.122133705" watchObservedRunningTime="2026-09-09 04:29:32.173474355 +0000 UTC m=+1893.123165010"
Sep 09 04:29:32.475573 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:32.475488    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:32.477075 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:32.477055    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 09 04:29:32.477135 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:32.477104    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 09 04:29:33.479648 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:33.479619    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:34.164196 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:34.164155    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:34.165080 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:34.165057    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:35.376698 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.376662    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"]
Sep 09 04:29:35.380926 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.380902    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:35.383362 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.383341    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-e787bf-serving-cert\""
Sep 09 04:29:35.389856 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.389832    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"]
Sep 09 04:29:35.417626 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.417595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:35.417781 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.417653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmjh\" (UniqueName: \"kubernetes.io/projected/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-kube-api-access-8pmjh\") pod \"maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:35.417781 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.417709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:35.417781 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.417740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:35.519064 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.519025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:35.519247 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.519090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmjh\" (UniqueName: \"kubernetes.io/projected/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-kube-api-access-8pmjh\") pod \"maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:35.519247 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.519127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:35.519247 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.519150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:35.519247 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:29:35.519194    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-serving-cert: secret "maas-api-e2e-ipp-cleanup-e787bf-serving-cert" not found
Sep 09 04:29:35.519404 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:29:35.519271    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-maas-api-tls podName:0cccaf17-68e3-4dc7-a97f-ce30b92a3100 nodeName:}" failed. No retries permitted until 2026-09-09 04:29:36.019249041 +0000 UTC m=+1896.968939682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt" (UID: "0cccaf17-68e3-4dc7-a97f-ce30b92a3100") : secret "maas-api-e2e-ipp-cleanup-e787bf-serving-cert" not found
Sep 09 04:29:35.519935 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.519880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:35.521784 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.521751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:35.530272 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:35.530247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmjh\" (UniqueName: \"kubernetes.io/projected/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-kube-api-access-8pmjh\") pod \"maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:36.023126 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:36.023091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:36.025754 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:36.025726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:36.296421 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:36.296331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:36.578833 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:36.578800    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"]
Sep 09 04:29:36.579342 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:29:36.579319    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cccaf17_68e3_4dc7_a97f_ce30b92a3100.slice/crio-9d3dcde8c4168eb0eb7a68bbd783f6cf2cd0d50d8b5b8a5d18fbe0840a6925fd WatchSource:0}: Error finding container 9d3dcde8c4168eb0eb7a68bbd783f6cf2cd0d50d8b5b8a5d18fbe0840a6925fd: Status 404 returned error can't find the container with id 9d3dcde8c4168eb0eb7a68bbd783f6cf2cd0d50d8b5b8a5d18fbe0840a6925fd
Sep 09 04:29:37.183390 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:37.183352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt" event={"ID":"0cccaf17-68e3-4dc7-a97f-ce30b92a3100","Type":"ContainerStarted","Data":"5727da09fe4192d96cce05347009cb5c09634f9370377b93dc35744185a0926f"}
Sep 09 04:29:37.183576 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:37.183408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt" event={"ID":"0cccaf17-68e3-4dc7-a97f-ce30b92a3100","Type":"ContainerStarted","Data":"9d3dcde8c4168eb0eb7a68bbd783f6cf2cd0d50d8b5b8a5d18fbe0840a6925fd"}
Sep 09 04:29:37.183576 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:37.183472    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:37.202969 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:37.202925    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt" podStartSLOduration=1.774842659 podStartE2EDuration="2.202910649s" totalImagesPullingTime="428.06799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:29:37.201171864 +0000 UTC m=+1898.150862518" watchObservedRunningTime="2026-09-09 04:29:37.202910649 +0000 UTC m=+1898.152601303"
Sep 09 04:29:43.195298 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:43.195265    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:29:54.292169 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:54.292129    2580 generic.go:356] "Generic (PLEG): container finished" podID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerID="c43c6cdf4d3384d350db1b5435bdfb7261229f57e0bc8ac0d89a2735b2d66c18" exitCode=0
Sep 09 04:29:54.292583 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:54.292200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" event={"ID":"449e7ee8-cbb7-4103-867d-c22889f8c391","Type":"ContainerDied","Data":"c43c6cdf4d3384d350db1b5435bdfb7261229f57e0bc8ac0d89a2735b2d66c18"}
Sep 09 04:29:54.292667 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:54.292651    2580 scope.go:117] "RemoveContainer" containerID="c43c6cdf4d3384d350db1b5435bdfb7261229f57e0bc8ac0d89a2735b2d66c18"
Sep 09 04:29:54.293950 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:54.293919    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:29:54.294052 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:54.294034    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:29:54.294099 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:54.294084    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 04:29:55.300595 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:55.300559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" event={"ID":"449e7ee8-cbb7-4103-867d-c22889f8c391","Type":"ContainerStarted","Data":"45276332ea8ba4fb05af96d050a934d1e2aa84674927398e35c15ed819c51bb1"}
Sep 09 04:29:55.475392 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:55.475355    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:29:55.475566 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:55.475531    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 09 04:29:55.475617 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:55.475591    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 09 04:29:56.475568 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:56.475534    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 09 04:29:56.476054 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:56.475594    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 09 04:29:57.474670 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:57.474631    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 09 04:29:57.474876 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:57.474693    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 09 04:29:58.475099 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:58.475056    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 09 04:29:58.475575 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:58.475133    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 09 04:29:59.474691 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:59.474656    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 09 04:29:59.474896 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:29:59.474716    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 09 04:30:00.167216 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:00.167178    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815470-dgh7w"]
Sep 09 04:30:00.171737 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:00.171712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815470-dgh7w"
Sep 09 04:30:00.175524 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:00.175468    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qlbvk\""
Sep 09 04:30:00.181545 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:00.181516    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815470-dgh7w"]
Sep 09 04:30:00.185283 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:00.185259    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815470-dgh7w"
Sep 09 04:30:00.474850 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:00.474745    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 09 04:30:00.474850 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:00.474835    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 09 04:30:00.489151 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:00.489128    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815470-dgh7w"]
Sep 09 04:30:00.491175 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:30:00.491148    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929a7f70_ddc5_40c9_a6a2_38dc554b341b.slice/crio-12858c5990b1264d2b4c87939f6ec20fa4de9b31f14aea3fd3ebe1d21a7b3274 WatchSource:0}: Error finding container 12858c5990b1264d2b4c87939f6ec20fa4de9b31f14aea3fd3ebe1d21a7b3274: Status 404 returned error can't find the container with id 12858c5990b1264d2b4c87939f6ec20fa4de9b31f14aea3fd3ebe1d21a7b3274
Sep 09 04:30:01.354139 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:01.354104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815470-dgh7w" event={"ID":"929a7f70-ddc5-40c9-a6a2-38dc554b341b","Type":"ContainerStarted","Data":"bde912087f6ae68e17638b1e7c214aa4b9912f9a1b7454d23509f395853da15e"}
Sep 09 04:30:01.354555 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:01.354172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815470-dgh7w" event={"ID":"929a7f70-ddc5-40c9-a6a2-38dc554b341b","Type":"ContainerStarted","Data":"12858c5990b1264d2b4c87939f6ec20fa4de9b31f14aea3fd3ebe1d21a7b3274"}
Sep 09 04:30:01.371519 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:01.371457    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815470-dgh7w" podStartSLOduration=1.371430922 podStartE2EDuration="1.371430922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:30:01.369141423 +0000 UTC m=+1922.318832079" watchObservedRunningTime="2026-09-09 04:30:01.371430922 +0000 UTC m=+1922.321121577"
Sep 09 04:30:01.478964 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:01.478931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:30:01.479251 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:01.479228    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:30:01.480062 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:01.480042    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:30:02.361460 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:02.361431    2580 generic.go:356] "Generic (PLEG): container finished" podID="929a7f70-ddc5-40c9-a6a2-38dc554b341b" containerID="bde912087f6ae68e17638b1e7c214aa4b9912f9a1b7454d23509f395853da15e" exitCode=0
Sep 09 04:30:02.362003 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:02.361522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815470-dgh7w" event={"ID":"929a7f70-ddc5-40c9-a6a2-38dc554b341b","Type":"ContainerDied","Data":"bde912087f6ae68e17638b1e7c214aa4b9912f9a1b7454d23509f395853da15e"}
Sep 09 04:30:03.500272 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:03.500243    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815470-dgh7w"
Sep 09 04:30:04.375340 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:04.375306    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815470-dgh7w"
Sep 09 04:30:04.375547 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:04.375343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-85732c-29815470-dgh7w" event={"ID":"929a7f70-ddc5-40c9-a6a2-38dc554b341b","Type":"ContainerDied","Data":"12858c5990b1264d2b4c87939f6ec20fa4de9b31f14aea3fd3ebe1d21a7b3274"}
Sep 09 04:30:04.375547 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:04.375387    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12858c5990b1264d2b4c87939f6ec20fa4de9b31f14aea3fd3ebe1d21a7b3274"
Sep 09 04:30:05.996075 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:05.996037    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"]
Sep 09 04:30:05.996552 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:05.996321    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt" podUID="0cccaf17-68e3-4dc7-a97f-ce30b92a3100" containerName="maas-api" containerID="cri-o://5727da09fe4192d96cce05347009cb5c09634f9370377b93dc35744185a0926f" gracePeriod=30
Sep 09 04:30:06.268464 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.268437    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:30:06.341055 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.341021    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pmjh\" (UniqueName: \"kubernetes.io/projected/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-kube-api-access-8pmjh\") pod \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") "
Sep 09 04:30:06.341246 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.341082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-openshift-service-ca\") pod \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") "
Sep 09 04:30:06.341246 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.341129    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-maas-api-tls\") pod \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") "
Sep 09 04:30:06.341246 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.341170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-metrics-certs\") pod \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\" (UID: \"0cccaf17-68e3-4dc7-a97f-ce30b92a3100\") "
Sep 09 04:30:06.341458 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.341434    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0cccaf17-68e3-4dc7-a97f-ce30b92a3100" (UID: "0cccaf17-68e3-4dc7-a97f-ce30b92a3100"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:30:06.341604 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.341580    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-openshift-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:30:06.343455 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.343417    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0cccaf17-68e3-4dc7-a97f-ce30b92a3100" (UID: "0cccaf17-68e3-4dc7-a97f-ce30b92a3100"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:30:06.343553 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.343451    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0cccaf17-68e3-4dc7-a97f-ce30b92a3100" (UID: "0cccaf17-68e3-4dc7-a97f-ce30b92a3100"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:30:06.343553 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.343519    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-kube-api-access-8pmjh" (OuterVolumeSpecName: "kube-api-access-8pmjh") pod "0cccaf17-68e3-4dc7-a97f-ce30b92a3100" (UID: "0cccaf17-68e3-4dc7-a97f-ce30b92a3100"). InnerVolumeSpecName "kube-api-access-8pmjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:30:06.395935 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.395900    2580 generic.go:356] "Generic (PLEG): container finished" podID="0cccaf17-68e3-4dc7-a97f-ce30b92a3100" containerID="5727da09fe4192d96cce05347009cb5c09634f9370377b93dc35744185a0926f" exitCode=0
Sep 09 04:30:06.396104 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.395945    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"
Sep 09 04:30:06.396104 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.395986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt" event={"ID":"0cccaf17-68e3-4dc7-a97f-ce30b92a3100","Type":"ContainerDied","Data":"5727da09fe4192d96cce05347009cb5c09634f9370377b93dc35744185a0926f"}
Sep 09 04:30:06.396104 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.396041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt" event={"ID":"0cccaf17-68e3-4dc7-a97f-ce30b92a3100","Type":"ContainerDied","Data":"9d3dcde8c4168eb0eb7a68bbd783f6cf2cd0d50d8b5b8a5d18fbe0840a6925fd"}
Sep 09 04:30:06.396104 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.396065    2580 scope.go:117] "RemoveContainer" containerID="5727da09fe4192d96cce05347009cb5c09634f9370377b93dc35744185a0926f"
Sep 09 04:30:06.422115 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.422088    2580 scope.go:117] "RemoveContainer" containerID="5727da09fe4192d96cce05347009cb5c09634f9370377b93dc35744185a0926f"
Sep 09 04:30:06.424307 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:30:06.424280    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5727da09fe4192d96cce05347009cb5c09634f9370377b93dc35744185a0926f\": container with ID starting with 5727da09fe4192d96cce05347009cb5c09634f9370377b93dc35744185a0926f not found: ID does not exist" containerID="5727da09fe4192d96cce05347009cb5c09634f9370377b93dc35744185a0926f"
Sep 09 04:30:06.424485 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.424461    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5727da09fe4192d96cce05347009cb5c09634f9370377b93dc35744185a0926f"} err="failed to get container status \"5727da09fe4192d96cce05347009cb5c09634f9370377b93dc35744185a0926f\": rpc error: code = NotFound desc = could not find container \"5727da09fe4192d96cce05347009cb5c09634f9370377b93dc35744185a0926f\": container with ID starting with 5727da09fe4192d96cce05347009cb5c09634f9370377b93dc35744185a0926f not found: ID does not exist"
Sep 09 04:30:06.429794 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.429622    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"]
Sep 09 04:30:06.434125 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.434019    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e787bf-7946bf678f-7tgrt"]
Sep 09 04:30:06.442859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.442826    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-maas-api-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:30:06.442859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.442856    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:30:06.443028 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:06.442865    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pmjh\" (UniqueName: \"kubernetes.io/projected/0cccaf17-68e3-4dc7-a97f-ce30b92a3100-kube-api-access-8pmjh\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:30:07.596526 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:07.596495    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cccaf17-68e3-4dc7-a97f-ce30b92a3100" path="/var/lib/kubelet/pods/0cccaf17-68e3-4dc7-a97f-ce30b92a3100/volumes"
Sep 09 04:30:13.196214 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:13.196178    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"]
Sep 09 04:30:13.196614 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:13.196424    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy" containerID="cri-o://45276332ea8ba4fb05af96d050a934d1e2aa84674927398e35c15ed819c51bb1" gracePeriod=30
Sep 09 04:30:16.482192 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:16.482147    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 04:30:16.482559 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:16.482223    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 04:30:18.462844 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.462788    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:30:18.475063 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.475036    2580 generic.go:356] "Generic (PLEG): container finished" podID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerID="45276332ea8ba4fb05af96d050a934d1e2aa84674927398e35c15ed819c51bb1" exitCode=0
Sep 09 04:30:18.475176 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.475083    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"
Sep 09 04:30:18.475176 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.475115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" event={"ID":"449e7ee8-cbb7-4103-867d-c22889f8c391","Type":"ContainerDied","Data":"45276332ea8ba4fb05af96d050a934d1e2aa84674927398e35c15ed819c51bb1"}
Sep 09 04:30:18.475176 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.475167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn" event={"ID":"449e7ee8-cbb7-4103-867d-c22889f8c391","Type":"ContainerDied","Data":"29d607da3e85db168d320c9219562564c6eddda32ce68a07f792d592fbd4c711"}
Sep 09 04:30:18.475282 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.475189    2580 scope.go:117] "RemoveContainer" containerID="45276332ea8ba4fb05af96d050a934d1e2aa84674927398e35c15ed819c51bb1"
Sep 09 04:30:18.486422 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.486400    2580 scope.go:117] "RemoveContainer" containerID="c43c6cdf4d3384d350db1b5435bdfb7261229f57e0bc8ac0d89a2735b2d66c18"
Sep 09 04:30:18.498271 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.498251    2580 scope.go:117] "RemoveContainer" containerID="45276332ea8ba4fb05af96d050a934d1e2aa84674927398e35c15ed819c51bb1"
Sep 09 04:30:18.498557 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:30:18.498538    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45276332ea8ba4fb05af96d050a934d1e2aa84674927398e35c15ed819c51bb1\": container with ID starting with 45276332ea8ba4fb05af96d050a934d1e2aa84674927398e35c15ed819c51bb1 not found: ID does not exist" containerID="45276332ea8ba4fb05af96d050a934d1e2aa84674927398e35c15ed819c51bb1"
Sep 09 04:30:18.498627 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.498566    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45276332ea8ba4fb05af96d050a934d1e2aa84674927398e35c15ed819c51bb1"} err="failed to get container status \"45276332ea8ba4fb05af96d050a934d1e2aa84674927398e35c15ed819c51bb1\": rpc error: code = NotFound desc = could not find container \"45276332ea8ba4fb05af96d050a934d1e2aa84674927398e35c15ed819c51bb1\": container with ID starting with 45276332ea8ba4fb05af96d050a934d1e2aa84674927398e35c15ed819c51bb1 not found: ID does not exist"
Sep 09 04:30:18.498627 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.498584    2580 scope.go:117] "RemoveContainer" containerID="c43c6cdf4d3384d350db1b5435bdfb7261229f57e0bc8ac0d89a2735b2d66c18"
Sep 09 04:30:18.498890 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:30:18.498871    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c43c6cdf4d3384d350db1b5435bdfb7261229f57e0bc8ac0d89a2735b2d66c18\": container with ID starting with c43c6cdf4d3384d350db1b5435bdfb7261229f57e0bc8ac0d89a2735b2d66c18 not found: ID does not exist" containerID="c43c6cdf4d3384d350db1b5435bdfb7261229f57e0bc8ac0d89a2735b2d66c18"
Sep 09 04:30:18.498959 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.498907    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c43c6cdf4d3384d350db1b5435bdfb7261229f57e0bc8ac0d89a2735b2d66c18"} err="failed to get container status \"c43c6cdf4d3384d350db1b5435bdfb7261229f57e0bc8ac0d89a2735b2d66c18\": rpc error: code = NotFound desc = could not find container \"c43c6cdf4d3384d350db1b5435bdfb7261229f57e0bc8ac0d89a2735b2d66c18\": container with ID starting with c43c6cdf4d3384d350db1b5435bdfb7261229f57e0bc8ac0d89a2735b2d66c18 not found: ID does not exist"
Sep 09 04:30:18.564426 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.564401    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-token\") pod \"449e7ee8-cbb7-4103-867d-c22889f8c391\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") "
Sep 09 04:30:18.564582 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.564441    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-data\") pod \"449e7ee8-cbb7-4103-867d-c22889f8c391\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") "
Sep 09 04:30:18.564582 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.564471    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-credential-socket\") pod \"449e7ee8-cbb7-4103-867d-c22889f8c391\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") "
Sep 09 04:30:18.564582 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.564510    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-podinfo\") pod \"449e7ee8-cbb7-4103-867d-c22889f8c391\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") "
Sep 09 04:30:18.564582 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.564536    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449e7ee8-cbb7-4103-867d-c22889f8c391-istiod-ca-cert\") pod \"449e7ee8-cbb7-4103-867d-c22889f8c391\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") "
Sep 09 04:30:18.564582 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.564555    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lvn7\" (UniqueName: \"kubernetes.io/projected/449e7ee8-cbb7-4103-867d-c22889f8c391-kube-api-access-5lvn7\") pod \"449e7ee8-cbb7-4103-867d-c22889f8c391\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") "
Sep 09 04:30:18.564915 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.564682    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-workload-socket\") pod \"449e7ee8-cbb7-4103-867d-c22889f8c391\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") "
Sep 09 04:30:18.564915 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.564689    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-data" (OuterVolumeSpecName: "istio-data") pod "449e7ee8-cbb7-4103-867d-c22889f8c391" (UID: "449e7ee8-cbb7-4103-867d-c22889f8c391"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:30:18.564915 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.564747    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-envoy\") pod \"449e7ee8-cbb7-4103-867d-c22889f8c391\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") "
Sep 09 04:30:18.564915 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.564806    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-workload-certs\") pod \"449e7ee8-cbb7-4103-867d-c22889f8c391\" (UID: \"449e7ee8-cbb7-4103-867d-c22889f8c391\") "
Sep 09 04:30:18.564915 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.564905    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "449e7ee8-cbb7-4103-867d-c22889f8c391" (UID: "449e7ee8-cbb7-4103-867d-c22889f8c391"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:30:18.565341 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.565040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "449e7ee8-cbb7-4103-867d-c22889f8c391" (UID: "449e7ee8-cbb7-4103-867d-c22889f8c391"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:30:18.565341 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.565088    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449e7ee8-cbb7-4103-867d-c22889f8c391-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "449e7ee8-cbb7-4103-867d-c22889f8c391" (UID: "449e7ee8-cbb7-4103-867d-c22889f8c391"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:30:18.565341 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.565109    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "449e7ee8-cbb7-4103-867d-c22889f8c391" (UID: "449e7ee8-cbb7-4103-867d-c22889f8c391"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:30:18.565341 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.565235    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449e7ee8-cbb7-4103-867d-c22889f8c391-istiod-ca-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:30:18.565341 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.565254    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-workload-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:30:18.565341 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.565267    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-workload-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:30:18.565341 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.565280    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-data\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:30:18.565341 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.565291    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-credential-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:30:18.566866 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.566846    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-token" (OuterVolumeSpecName: "istio-token") pod "449e7ee8-cbb7-4103-867d-c22889f8c391" (UID: "449e7ee8-cbb7-4103-867d-c22889f8c391"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:30:18.567125 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.567107    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449e7ee8-cbb7-4103-867d-c22889f8c391-kube-api-access-5lvn7" (OuterVolumeSpecName: "kube-api-access-5lvn7") pod "449e7ee8-cbb7-4103-867d-c22889f8c391" (UID: "449e7ee8-cbb7-4103-867d-c22889f8c391"). InnerVolumeSpecName "kube-api-access-5lvn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:30:18.567356 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.567340    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "449e7ee8-cbb7-4103-867d-c22889f8c391" (UID: "449e7ee8-cbb7-4103-867d-c22889f8c391"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:30:18.567408 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.567350    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "449e7ee8-cbb7-4103-867d-c22889f8c391" (UID: "449e7ee8-cbb7-4103-867d-c22889f8c391"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:30:18.665859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.665816    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:30:18.665859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.665853    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-podinfo\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:30:18.665859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.665863    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lvn7\" (UniqueName: \"kubernetes.io/projected/449e7ee8-cbb7-4103-867d-c22889f8c391-kube-api-access-5lvn7\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:30:18.666106 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.665872    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449e7ee8-cbb7-4103-867d-c22889f8c391-istio-envoy\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:30:18.799466 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.799438    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"]
Sep 09 04:30:18.809753 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:18.809723    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e787bf-openshift-default-745f8fcc66-4b2nn"]
Sep 09 04:30:19.596894 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:30:19.596859    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" path="/var/lib/kubelet/pods/449e7ee8-cbb7-4103-867d-c22889f8c391/volumes"
Sep 09 04:31:21.831293 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:21.831254    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"]
Sep 09 04:31:21.831858 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:21.831525    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerName="istio-proxy" containerID="cri-o://61917a287e913bd645cec1b82f7bca79117df2e0b0e91665136da752686e2449" gracePeriod=30
Sep 09 04:31:27.090608 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.090546    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:31:27.149628 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.149591    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-token\") pod \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") "
Sep 09 04:31:27.149851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.149653    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-envoy\") pod \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") "
Sep 09 04:31:27.149851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.149755    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc9sn\" (UniqueName: \"kubernetes.io/projected/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-kube-api-access-zc9sn\") pod \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") "
Sep 09 04:31:27.149851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.149810    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-data\") pod \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") "
Sep 09 04:31:27.150056 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.149881    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-podinfo\") pod \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") "
Sep 09 04:31:27.150056 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.149911    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istiod-ca-cert\") pod \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") "
Sep 09 04:31:27.150056 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.149947    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-workload-socket\") pod \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") "
Sep 09 04:31:27.150056 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.149991    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-workload-certs\") pod \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") "
Sep 09 04:31:27.150056 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.150023    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-credential-socket\") pod \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\" (UID: \"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e\") "
Sep 09 04:31:27.150308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.150128    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" (UID: "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:31:27.150369 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.150302    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" (UID: "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:31:27.150532 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.150380    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-data\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:27.150532 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.150399    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-workload-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:27.150532 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.150371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" (UID: "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:31:27.150532 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.150475    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" (UID: "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:31:27.150825 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.150798    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" (UID: "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:31:27.152198 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.152166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" (UID: "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:31:27.152304 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.152198    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" (UID: "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:31:27.152422 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.152406    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" (UID: "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:31:27.152491 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.152467    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-kube-api-access-zc9sn" (OuterVolumeSpecName: "kube-api-access-zc9sn") pod "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" (UID: "d5810775-3101-4fdd-b58b-9c3ce7e5ae7e"). InnerVolumeSpecName "kube-api-access-zc9sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:31:27.251633 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.251600    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-workload-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:27.251633 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.251630    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-credential-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:27.251633 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.251643    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:27.251914 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.251655    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-envoy\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:27.251914 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.251666    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc9sn\" (UniqueName: \"kubernetes.io/projected/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-kube-api-access-zc9sn\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:27.251914 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.251677    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istio-podinfo\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:27.251914 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.251688    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e-istiod-ca-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:27.896972 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.896891    2580 generic.go:356] "Generic (PLEG): container finished" podID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerID="61917a287e913bd645cec1b82f7bca79117df2e0b0e91665136da752686e2449" exitCode=0
Sep 09 04:31:27.896972 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.896942    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"
Sep 09 04:31:27.897150 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.896968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" event={"ID":"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e","Type":"ContainerDied","Data":"61917a287e913bd645cec1b82f7bca79117df2e0b0e91665136da752686e2449"}
Sep 09 04:31:27.897150 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.897024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r" event={"ID":"d5810775-3101-4fdd-b58b-9c3ce7e5ae7e","Type":"ContainerDied","Data":"10ed5fb4c6569d4a7c81a4c053663ce4073b6dd458bd68ed251fdfed9487b333"}
Sep 09 04:31:27.897150 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.897050    2580 scope.go:117] "RemoveContainer" containerID="61917a287e913bd645cec1b82f7bca79117df2e0b0e91665136da752686e2449"
Sep 09 04:31:27.906445 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.906426    2580 scope.go:117] "RemoveContainer" containerID="f8dff45b09b8ebdfac59007e3b9892e5937cc17b1c16a13e5f9a85e4c473a4b4"
Sep 09 04:31:27.915632 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.915606    2580 scope.go:117] "RemoveContainer" containerID="61917a287e913bd645cec1b82f7bca79117df2e0b0e91665136da752686e2449"
Sep 09 04:31:27.915986 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:31:27.915964    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61917a287e913bd645cec1b82f7bca79117df2e0b0e91665136da752686e2449\": container with ID starting with 61917a287e913bd645cec1b82f7bca79117df2e0b0e91665136da752686e2449 not found: ID does not exist" containerID="61917a287e913bd645cec1b82f7bca79117df2e0b0e91665136da752686e2449"
Sep 09 04:31:27.916099 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.915993    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61917a287e913bd645cec1b82f7bca79117df2e0b0e91665136da752686e2449"} err="failed to get container status \"61917a287e913bd645cec1b82f7bca79117df2e0b0e91665136da752686e2449\": rpc error: code = NotFound desc = could not find container \"61917a287e913bd645cec1b82f7bca79117df2e0b0e91665136da752686e2449\": container with ID starting with 61917a287e913bd645cec1b82f7bca79117df2e0b0e91665136da752686e2449 not found: ID does not exist"
Sep 09 04:31:27.916099 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.916010    2580 scope.go:117] "RemoveContainer" containerID="f8dff45b09b8ebdfac59007e3b9892e5937cc17b1c16a13e5f9a85e4c473a4b4"
Sep 09 04:31:27.916288 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:31:27.916263    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8dff45b09b8ebdfac59007e3b9892e5937cc17b1c16a13e5f9a85e4c473a4b4\": container with ID starting with f8dff45b09b8ebdfac59007e3b9892e5937cc17b1c16a13e5f9a85e4c473a4b4 not found: ID does not exist" containerID="f8dff45b09b8ebdfac59007e3b9892e5937cc17b1c16a13e5f9a85e4c473a4b4"
Sep 09 04:31:27.916348 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.916296    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8dff45b09b8ebdfac59007e3b9892e5937cc17b1c16a13e5f9a85e4c473a4b4"} err="failed to get container status \"f8dff45b09b8ebdfac59007e3b9892e5937cc17b1c16a13e5f9a85e4c473a4b4\": rpc error: code = NotFound desc = could not find container \"f8dff45b09b8ebdfac59007e3b9892e5937cc17b1c16a13e5f9a85e4c473a4b4\": container with ID starting with f8dff45b09b8ebdfac59007e3b9892e5937cc17b1c16a13e5f9a85e4c473a4b4 not found: ID does not exist"
Sep 09 04:31:27.917202 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.917181    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"]
Sep 09 04:31:27.920696 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:27.920674    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3ce90-openshift-default-79cd5ff5cb-mkp9r"]
Sep 09 04:31:29.596608 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:29.596572    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" path="/var/lib/kubelet/pods/d5810775-3101-4fdd-b58b-9c3ce7e5ae7e/volumes"
Sep 09 04:31:32.715097 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:32.715062    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"]
Sep 09 04:31:32.715588 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:32.715323    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk" podUID="c129ed2e-f4df-4123-8c67-86c27c2b04fa" containerName="maas-api" containerID="cri-o://122788dd5a06993c14fd001d6bc2e50f4d767624a79a7c72e08a525dfc05874f" gracePeriod=30
Sep 09 04:31:32.929751 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:32.929711    2580 generic.go:356] "Generic (PLEG): container finished" podID="c129ed2e-f4df-4123-8c67-86c27c2b04fa" containerID="122788dd5a06993c14fd001d6bc2e50f4d767624a79a7c72e08a525dfc05874f" exitCode=0
Sep 09 04:31:32.929939 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:32.929833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk" event={"ID":"c129ed2e-f4df-4123-8c67-86c27c2b04fa","Type":"ContainerDied","Data":"122788dd5a06993c14fd001d6bc2e50f4d767624a79a7c72e08a525dfc05874f"}
Sep 09 04:31:32.971957 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:32.971900    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:31:33.107428 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.107396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-metrics-certs\") pod \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") "
Sep 09 04:31:33.107620 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.107456    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-maas-api-tls\") pod \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") "
Sep 09 04:31:33.107620 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.107504    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c129ed2e-f4df-4123-8c67-86c27c2b04fa-openshift-service-ca\") pod \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") "
Sep 09 04:31:33.107620 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.107530    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccd7w\" (UniqueName: \"kubernetes.io/projected/c129ed2e-f4df-4123-8c67-86c27c2b04fa-kube-api-access-ccd7w\") pod \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\" (UID: \"c129ed2e-f4df-4123-8c67-86c27c2b04fa\") "
Sep 09 04:31:33.108277 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.108246    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c129ed2e-f4df-4123-8c67-86c27c2b04fa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c129ed2e-f4df-4123-8c67-86c27c2b04fa" (UID: "c129ed2e-f4df-4123-8c67-86c27c2b04fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:31:33.110543 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.110504    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c129ed2e-f4df-4123-8c67-86c27c2b04fa-kube-api-access-ccd7w" (OuterVolumeSpecName: "kube-api-access-ccd7w") pod "c129ed2e-f4df-4123-8c67-86c27c2b04fa" (UID: "c129ed2e-f4df-4123-8c67-86c27c2b04fa"). InnerVolumeSpecName "kube-api-access-ccd7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:31:33.110830 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.110802    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c129ed2e-f4df-4123-8c67-86c27c2b04fa" (UID: "c129ed2e-f4df-4123-8c67-86c27c2b04fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:31:33.110998 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.110972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c129ed2e-f4df-4123-8c67-86c27c2b04fa" (UID: "c129ed2e-f4df-4123-8c67-86c27c2b04fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:31:33.209281 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.209243    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:33.209281 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.209272    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c129ed2e-f4df-4123-8c67-86c27c2b04fa-maas-api-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:33.209281 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.209281    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c129ed2e-f4df-4123-8c67-86c27c2b04fa-openshift-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:33.209281 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.209292    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccd7w\" (UniqueName: \"kubernetes.io/projected/c129ed2e-f4df-4123-8c67-86c27c2b04fa-kube-api-access-ccd7w\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:33.937535 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.937444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk" event={"ID":"c129ed2e-f4df-4123-8c67-86c27c2b04fa","Type":"ContainerDied","Data":"b7221aa8a4fffc5844f3928cb28037522e117f39b5348653c16a78dc65048067"}
Sep 09 04:31:33.937535 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.937462    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"
Sep 09 04:31:33.937535 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.937503    2580 scope.go:117] "RemoveContainer" containerID="122788dd5a06993c14fd001d6bc2e50f4d767624a79a7c72e08a525dfc05874f"
Sep 09 04:31:33.965689 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.965663    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"]
Sep 09 04:31:33.970888 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:33.970864    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db87d-c6f55897f-7hrjk"]
Sep 09 04:31:35.595347 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:35.595317    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c129ed2e-f4df-4123-8c67-86c27c2b04fa" path="/var/lib/kubelet/pods/c129ed2e-f4df-4123-8c67-86c27c2b04fa/volumes"
Sep 09 04:31:50.828698 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:50.828662    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"]
Sep 09 04:31:50.829551 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:50.829492    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96" podUID="c5051f50-3e61-4b4f-9358-be04824a254c" containerName="maas-api" containerID="cri-o://f888cf4dddd87a52f4d4dc529ea7914f3614647dcc8af2c853e801dad184cc49" gracePeriod=30
Sep 09 04:31:51.051669 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.051631    2580 generic.go:356] "Generic (PLEG): container finished" podID="c5051f50-3e61-4b4f-9358-be04824a254c" containerID="f888cf4dddd87a52f4d4dc529ea7914f3614647dcc8af2c853e801dad184cc49" exitCode=0
Sep 09 04:31:51.051866 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.051704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96" event={"ID":"c5051f50-3e61-4b4f-9358-be04824a254c","Type":"ContainerDied","Data":"f888cf4dddd87a52f4d4dc529ea7914f3614647dcc8af2c853e801dad184cc49"}
Sep 09 04:31:51.084823 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.084731    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:31:51.280629 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.280595    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5051f50-3e61-4b4f-9358-be04824a254c-openshift-service-ca\") pod \"c5051f50-3e61-4b4f-9358-be04824a254c\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") "
Sep 09 04:31:51.280840 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.280675    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-maas-api-tls\") pod \"c5051f50-3e61-4b4f-9358-be04824a254c\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") "
Sep 09 04:31:51.280840 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.280706    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95bs4\" (UniqueName: \"kubernetes.io/projected/c5051f50-3e61-4b4f-9358-be04824a254c-kube-api-access-95bs4\") pod \"c5051f50-3e61-4b4f-9358-be04824a254c\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") "
Sep 09 04:31:51.280840 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.280744    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-metrics-certs\") pod \"c5051f50-3e61-4b4f-9358-be04824a254c\" (UID: \"c5051f50-3e61-4b4f-9358-be04824a254c\") "
Sep 09 04:31:51.281042 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.281016    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5051f50-3e61-4b4f-9358-be04824a254c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5051f50-3e61-4b4f-9358-be04824a254c" (UID: "c5051f50-3e61-4b4f-9358-be04824a254c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:31:51.283105 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.283076    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c5051f50-3e61-4b4f-9358-be04824a254c" (UID: "c5051f50-3e61-4b4f-9358-be04824a254c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:31:51.283105 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.283088    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5051f50-3e61-4b4f-9358-be04824a254c" (UID: "c5051f50-3e61-4b4f-9358-be04824a254c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:31:51.283237 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.283111    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5051f50-3e61-4b4f-9358-be04824a254c-kube-api-access-95bs4" (OuterVolumeSpecName: "kube-api-access-95bs4") pod "c5051f50-3e61-4b4f-9358-be04824a254c" (UID: "c5051f50-3e61-4b4f-9358-be04824a254c"). InnerVolumeSpecName "kube-api-access-95bs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:31:51.381548 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.381520    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-maas-api-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:51.381548 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.381547    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95bs4\" (UniqueName: \"kubernetes.io/projected/c5051f50-3e61-4b4f-9358-be04824a254c-kube-api-access-95bs4\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:51.381731 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.381556    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5051f50-3e61-4b4f-9358-be04824a254c-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:51.381731 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:51.381566    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5051f50-3e61-4b4f-9358-be04824a254c-openshift-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:31:52.059720 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:52.059680    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"
Sep 09 04:31:52.059720 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:52.059694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96" event={"ID":"c5051f50-3e61-4b4f-9358-be04824a254c","Type":"ContainerDied","Data":"c8c88f7839be94910d5ec2b1e178455ab46528727da325ac68df94ad9f1d397e"}
Sep 09 04:31:52.060290 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:52.059790    2580 scope.go:117] "RemoveContainer" containerID="f888cf4dddd87a52f4d4dc529ea7914f3614647dcc8af2c853e801dad184cc49"
Sep 09 04:31:52.080400 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:52.080373    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"]
Sep 09 04:31:52.086970 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:52.086949    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-426183-7f48f7bd58-9fj96"]
Sep 09 04:31:53.596156 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:53.596122    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5051f50-3e61-4b4f-9358-be04824a254c" path="/var/lib/kubelet/pods/c5051f50-3e61-4b4f-9358-be04824a254c/volumes"
Sep 09 04:31:58.126632 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:58.126588    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"]
Sep 09 04:31:58.127232 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:31:58.126816    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerName="istio-proxy" containerID="cri-o://90c1f5291048765deb6e222fc50c985afae55832eae6f0aed4e452ae1f7351c2" gracePeriod=30
Sep 09 04:32:00.690632 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:00.690593    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 04:32:00.691039 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:00.690669    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 04:32:03.382786 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.382738    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:32:03.494130 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-data\") pod \"0793bc60-b1ec-46bd-99a5-828a08edd26f\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") "
Sep 09 04:32:03.494130 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494079    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-token\") pod \"0793bc60-b1ec-46bd-99a5-828a08edd26f\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") "
Sep 09 04:32:03.494334 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494153    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0793bc60-b1ec-46bd-99a5-828a08edd26f-istiod-ca-cert\") pod \"0793bc60-b1ec-46bd-99a5-828a08edd26f\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") "
Sep 09 04:32:03.494334 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494187    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-workload-certs\") pod \"0793bc60-b1ec-46bd-99a5-828a08edd26f\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") "
Sep 09 04:32:03.494334 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79sv6\" (UniqueName: \"kubernetes.io/projected/0793bc60-b1ec-46bd-99a5-828a08edd26f-kube-api-access-79sv6\") pod \"0793bc60-b1ec-46bd-99a5-828a08edd26f\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") "
Sep 09 04:32:03.494464 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494359    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-credential-socket\") pod \"0793bc60-b1ec-46bd-99a5-828a08edd26f\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") "
Sep 09 04:32:03.494464 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494374    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-data" (OuterVolumeSpecName: "istio-data") pod "0793bc60-b1ec-46bd-99a5-828a08edd26f" (UID: "0793bc60-b1ec-46bd-99a5-828a08edd26f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:32:03.494464 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494398    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-podinfo\") pod \"0793bc60-b1ec-46bd-99a5-828a08edd26f\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") "
Sep 09 04:32:03.494464 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494449    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-envoy\") pod \"0793bc60-b1ec-46bd-99a5-828a08edd26f\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") "
Sep 09 04:32:03.494675 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-workload-socket\") pod \"0793bc60-b1ec-46bd-99a5-828a08edd26f\" (UID: \"0793bc60-b1ec-46bd-99a5-828a08edd26f\") "
Sep 09 04:32:03.494675 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0793bc60-b1ec-46bd-99a5-828a08edd26f" (UID: "0793bc60-b1ec-46bd-99a5-828a08edd26f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:32:03.494675 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494585    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0793bc60-b1ec-46bd-99a5-828a08edd26f" (UID: "0793bc60-b1ec-46bd-99a5-828a08edd26f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:32:03.494675 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494596    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0793bc60-b1ec-46bd-99a5-828a08edd26f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0793bc60-b1ec-46bd-99a5-828a08edd26f" (UID: "0793bc60-b1ec-46bd-99a5-828a08edd26f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:32:03.494939 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494809    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0793bc60-b1ec-46bd-99a5-828a08edd26f" (UID: "0793bc60-b1ec-46bd-99a5-828a08edd26f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:32:03.494992 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494936    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-workload-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:03.494992 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494956    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-data\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:03.494992 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494969    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0793bc60-b1ec-46bd-99a5-828a08edd26f-istiod-ca-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:03.494992 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494982    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-workload-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:03.495154 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.494995    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-credential-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:03.496655 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.496627    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0793bc60-b1ec-46bd-99a5-828a08edd26f" (UID: "0793bc60-b1ec-46bd-99a5-828a08edd26f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:32:03.496790 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.496683    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0793bc60-b1ec-46bd-99a5-828a08edd26f" (UID: "0793bc60-b1ec-46bd-99a5-828a08edd26f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:32:03.496979 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.496951    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0793bc60-b1ec-46bd-99a5-828a08edd26f-kube-api-access-79sv6" (OuterVolumeSpecName: "kube-api-access-79sv6") pod "0793bc60-b1ec-46bd-99a5-828a08edd26f" (UID: "0793bc60-b1ec-46bd-99a5-828a08edd26f"). InnerVolumeSpecName "kube-api-access-79sv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:32:03.496979 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.496968    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-token" (OuterVolumeSpecName: "istio-token") pod "0793bc60-b1ec-46bd-99a5-828a08edd26f" (UID: "0793bc60-b1ec-46bd-99a5-828a08edd26f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:32:03.596217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.596186    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-envoy\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:03.596217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.596210    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:03.596217 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.596219    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79sv6\" (UniqueName: \"kubernetes.io/projected/0793bc60-b1ec-46bd-99a5-828a08edd26f-kube-api-access-79sv6\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:03.596458 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:03.596229    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0793bc60-b1ec-46bd-99a5-828a08edd26f-istio-podinfo\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:04.131700 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:04.131667    2580 generic.go:356] "Generic (PLEG): container finished" podID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerID="90c1f5291048765deb6e222fc50c985afae55832eae6f0aed4e452ae1f7351c2" exitCode=0
Sep 09 04:32:04.131909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:04.131718    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" event={"ID":"0793bc60-b1ec-46bd-99a5-828a08edd26f","Type":"ContainerDied","Data":"90c1f5291048765deb6e222fc50c985afae55832eae6f0aed4e452ae1f7351c2"}
Sep 09 04:32:04.131909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:04.131726    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"
Sep 09 04:32:04.131909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:04.131761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4" event={"ID":"0793bc60-b1ec-46bd-99a5-828a08edd26f","Type":"ContainerDied","Data":"f939f72afbb82a0ec287880d4f66c2cb4397f1ce97ec9aa4c292ec55c1ba9827"}
Sep 09 04:32:04.131909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:04.131801    2580 scope.go:117] "RemoveContainer" containerID="90c1f5291048765deb6e222fc50c985afae55832eae6f0aed4e452ae1f7351c2"
Sep 09 04:32:04.140875 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:04.140846    2580 scope.go:117] "RemoveContainer" containerID="61ad871ea48f1a674d788c4f2870aec3c2a0a8fd04ddd76101b4e95af3e09af4"
Sep 09 04:32:04.148989 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:04.148972    2580 scope.go:117] "RemoveContainer" containerID="90c1f5291048765deb6e222fc50c985afae55832eae6f0aed4e452ae1f7351c2"
Sep 09 04:32:04.149247 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:32:04.149229    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c1f5291048765deb6e222fc50c985afae55832eae6f0aed4e452ae1f7351c2\": container with ID starting with 90c1f5291048765deb6e222fc50c985afae55832eae6f0aed4e452ae1f7351c2 not found: ID does not exist" containerID="90c1f5291048765deb6e222fc50c985afae55832eae6f0aed4e452ae1f7351c2"
Sep 09 04:32:04.149297 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:04.149254    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c1f5291048765deb6e222fc50c985afae55832eae6f0aed4e452ae1f7351c2"} err="failed to get container status \"90c1f5291048765deb6e222fc50c985afae55832eae6f0aed4e452ae1f7351c2\": rpc error: code = NotFound desc = could not find container \"90c1f5291048765deb6e222fc50c985afae55832eae6f0aed4e452ae1f7351c2\": container with ID starting with 90c1f5291048765deb6e222fc50c985afae55832eae6f0aed4e452ae1f7351c2 not found: ID does not exist"
Sep 09 04:32:04.149297 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:04.149276    2580 scope.go:117] "RemoveContainer" containerID="61ad871ea48f1a674d788c4f2870aec3c2a0a8fd04ddd76101b4e95af3e09af4"
Sep 09 04:32:04.149606 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:32:04.149566    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ad871ea48f1a674d788c4f2870aec3c2a0a8fd04ddd76101b4e95af3e09af4\": container with ID starting with 61ad871ea48f1a674d788c4f2870aec3c2a0a8fd04ddd76101b4e95af3e09af4 not found: ID does not exist" containerID="61ad871ea48f1a674d788c4f2870aec3c2a0a8fd04ddd76101b4e95af3e09af4"
Sep 09 04:32:04.149724 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:04.149607    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ad871ea48f1a674d788c4f2870aec3c2a0a8fd04ddd76101b4e95af3e09af4"} err="failed to get container status \"61ad871ea48f1a674d788c4f2870aec3c2a0a8fd04ddd76101b4e95af3e09af4\": rpc error: code = NotFound desc = could not find container \"61ad871ea48f1a674d788c4f2870aec3c2a0a8fd04ddd76101b4e95af3e09af4\": container with ID starting with 61ad871ea48f1a674d788c4f2870aec3c2a0a8fd04ddd76101b4e95af3e09af4 not found: ID does not exist"
Sep 09 04:32:04.153064 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:04.153042    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"]
Sep 09 04:32:04.156886 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:04.156864    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-426183-openshift-default-6fb4888b88-qwkk4"]
Sep 09 04:32:05.595749 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:05.595708    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" path="/var/lib/kubelet/pods/0793bc60-b1ec-46bd-99a5-828a08edd26f/volumes"
Sep 09 04:32:16.243144 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:16.243103    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"]
Sep 09 04:32:16.243564 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:16.243400    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerName="istio-proxy" containerID="cri-o://dd6d668a8841206fb5eb72fc518d3c597b8864f395b1d655d4f88a88ed5b3f94" gracePeriod=30
Sep 09 04:32:18.801292 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:18.801253    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-z82dz"]
Sep 09 04:32:18.801722 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:18.801549    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7d8bb9d654-z82dz" podUID="669da265-2e72-437a-b0c5-19aa2d470b69" containerName="manager" containerID="cri-o://c586d9f972ae7a30589082f2e92eef858826325676bb4fc0525548ffb5a52355" gracePeriod=30
Sep 09 04:32:19.053517 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.053456    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:32:19.136619 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.136585    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669da265-2e72-437a-b0c5-19aa2d470b69-metrics-certs\") pod \"669da265-2e72-437a-b0c5-19aa2d470b69\" (UID: \"669da265-2e72-437a-b0c5-19aa2d470b69\") "
Sep 09 04:32:19.136804 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.136708    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fznh\" (UniqueName: \"kubernetes.io/projected/669da265-2e72-437a-b0c5-19aa2d470b69-kube-api-access-6fznh\") pod \"669da265-2e72-437a-b0c5-19aa2d470b69\" (UID: \"669da265-2e72-437a-b0c5-19aa2d470b69\") "
Sep 09 04:32:19.136804 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.136792    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/669da265-2e72-437a-b0c5-19aa2d470b69-webhook-cert\") pod \"669da265-2e72-437a-b0c5-19aa2d470b69\" (UID: \"669da265-2e72-437a-b0c5-19aa2d470b69\") "
Sep 09 04:32:19.138871 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.138839    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/669da265-2e72-437a-b0c5-19aa2d470b69-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "669da265-2e72-437a-b0c5-19aa2d470b69" (UID: "669da265-2e72-437a-b0c5-19aa2d470b69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:32:19.138998 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.138895    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669da265-2e72-437a-b0c5-19aa2d470b69-kube-api-access-6fznh" (OuterVolumeSpecName: "kube-api-access-6fznh") pod "669da265-2e72-437a-b0c5-19aa2d470b69" (UID: "669da265-2e72-437a-b0c5-19aa2d470b69"). InnerVolumeSpecName "kube-api-access-6fznh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:32:19.138998 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.138890    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/669da265-2e72-437a-b0c5-19aa2d470b69-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "669da265-2e72-437a-b0c5-19aa2d470b69" (UID: "669da265-2e72-437a-b0c5-19aa2d470b69"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:32:19.221086 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.221057    2580 generic.go:356] "Generic (PLEG): container finished" podID="669da265-2e72-437a-b0c5-19aa2d470b69" containerID="c586d9f972ae7a30589082f2e92eef858826325676bb4fc0525548ffb5a52355" exitCode=0
Sep 09 04:32:19.221223 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.221103    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bb9d654-z82dz"
Sep 09 04:32:19.221223 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.221144    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bb9d654-z82dz" event={"ID":"669da265-2e72-437a-b0c5-19aa2d470b69","Type":"ContainerDied","Data":"c586d9f972ae7a30589082f2e92eef858826325676bb4fc0525548ffb5a52355"}
Sep 09 04:32:19.221223 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.221206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bb9d654-z82dz" event={"ID":"669da265-2e72-437a-b0c5-19aa2d470b69","Type":"ContainerDied","Data":"8786ed8ee10274d79a5230581d3ccf334ed04f12d85bbaea03b7a7a0a1d5e705"}
Sep 09 04:32:19.221348 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.221228    2580 scope.go:117] "RemoveContainer" containerID="c586d9f972ae7a30589082f2e92eef858826325676bb4fc0525548ffb5a52355"
Sep 09 04:32:19.231629 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.231612    2580 scope.go:117] "RemoveContainer" containerID="c586d9f972ae7a30589082f2e92eef858826325676bb4fc0525548ffb5a52355"
Sep 09 04:32:19.231917 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:32:19.231899    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c586d9f972ae7a30589082f2e92eef858826325676bb4fc0525548ffb5a52355\": container with ID starting with c586d9f972ae7a30589082f2e92eef858826325676bb4fc0525548ffb5a52355 not found: ID does not exist" containerID="c586d9f972ae7a30589082f2e92eef858826325676bb4fc0525548ffb5a52355"
Sep 09 04:32:19.231981 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.231926    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c586d9f972ae7a30589082f2e92eef858826325676bb4fc0525548ffb5a52355"} err="failed to get container status \"c586d9f972ae7a30589082f2e92eef858826325676bb4fc0525548ffb5a52355\": rpc error: code = NotFound desc = could not find container \"c586d9f972ae7a30589082f2e92eef858826325676bb4fc0525548ffb5a52355\": container with ID starting with c586d9f972ae7a30589082f2e92eef858826325676bb4fc0525548ffb5a52355 not found: ID does not exist"
Sep 09 04:32:19.238098 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.238079    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fznh\" (UniqueName: \"kubernetes.io/projected/669da265-2e72-437a-b0c5-19aa2d470b69-kube-api-access-6fznh\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:19.238098 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.238099    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/669da265-2e72-437a-b0c5-19aa2d470b69-webhook-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:19.238220 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.238108    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669da265-2e72-437a-b0c5-19aa2d470b69-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:19.243882 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.243857    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-z82dz"]
Sep 09 04:32:19.248314 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.248295    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-z82dz"]
Sep 09 04:32:19.595579 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:19.595548    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="669da265-2e72-437a-b0c5-19aa2d470b69" path="/var/lib/kubelet/pods/669da265-2e72-437a-b0c5-19aa2d470b69/volumes"
Sep 09 04:32:21.510257 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.510196    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:32:21.661132 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661103    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d40511-21fb-460a-8ccb-ed86caf19e3a-istiod-ca-cert\") pod \"45d40511-21fb-460a-8ccb-ed86caf19e3a\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") "
Sep 09 04:32:21.661132 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661139    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-data\") pod \"45d40511-21fb-460a-8ccb-ed86caf19e3a\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") "
Sep 09 04:32:21.661385 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661155    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-workload-socket\") pod \"45d40511-21fb-460a-8ccb-ed86caf19e3a\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") "
Sep 09 04:32:21.661385 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-podinfo\") pod \"45d40511-21fb-460a-8ccb-ed86caf19e3a\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") "
Sep 09 04:32:21.661385 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661207    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-workload-certs\") pod \"45d40511-21fb-460a-8ccb-ed86caf19e3a\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") "
Sep 09 04:32:21.661385 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661230    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-envoy\") pod \"45d40511-21fb-460a-8ccb-ed86caf19e3a\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") "
Sep 09 04:32:21.661602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661381    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-credential-socket\") pod \"45d40511-21fb-460a-8ccb-ed86caf19e3a\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") "
Sep 09 04:32:21.661602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661422    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzpjz\" (UniqueName: \"kubernetes.io/projected/45d40511-21fb-460a-8ccb-ed86caf19e3a-kube-api-access-rzpjz\") pod \"45d40511-21fb-460a-8ccb-ed86caf19e3a\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") "
Sep 09 04:32:21.661602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661496    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-token\") pod \"45d40511-21fb-460a-8ccb-ed86caf19e3a\" (UID: \"45d40511-21fb-460a-8ccb-ed86caf19e3a\") "
Sep 09 04:32:21.661602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661539    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45d40511-21fb-460a-8ccb-ed86caf19e3a" (UID: "45d40511-21fb-460a-8ccb-ed86caf19e3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:32:21.661602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661556    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45d40511-21fb-460a-8ccb-ed86caf19e3a" (UID: "45d40511-21fb-460a-8ccb-ed86caf19e3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:32:21.661602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661598    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d40511-21fb-460a-8ccb-ed86caf19e3a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45d40511-21fb-460a-8ccb-ed86caf19e3a" (UID: "45d40511-21fb-460a-8ccb-ed86caf19e3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:32:21.661957 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661707    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45d40511-21fb-460a-8ccb-ed86caf19e3a" (UID: "45d40511-21fb-460a-8ccb-ed86caf19e3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:32:21.661957 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661905    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d40511-21fb-460a-8ccb-ed86caf19e3a-istiod-ca-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:21.661957 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661928    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-workload-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:21.661957 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661942    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-workload-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:21.661957 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.661956    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-credential-socket\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:21.662180 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.662014    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-data" (OuterVolumeSpecName: "istio-data") pod "45d40511-21fb-460a-8ccb-ed86caf19e3a" (UID: "45d40511-21fb-460a-8ccb-ed86caf19e3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:32:21.663799 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.663747    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d40511-21fb-460a-8ccb-ed86caf19e3a-kube-api-access-rzpjz" (OuterVolumeSpecName: "kube-api-access-rzpjz") pod "45d40511-21fb-460a-8ccb-ed86caf19e3a" (UID: "45d40511-21fb-460a-8ccb-ed86caf19e3a"). InnerVolumeSpecName "kube-api-access-rzpjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:32:21.663911 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.663792    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-token" (OuterVolumeSpecName: "istio-token") pod "45d40511-21fb-460a-8ccb-ed86caf19e3a" (UID: "45d40511-21fb-460a-8ccb-ed86caf19e3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:32:21.663996 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.663976    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45d40511-21fb-460a-8ccb-ed86caf19e3a" (UID: "45d40511-21fb-460a-8ccb-ed86caf19e3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 04:32:21.664059 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.664029    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45d40511-21fb-460a-8ccb-ed86caf19e3a" (UID: "45d40511-21fb-460a-8ccb-ed86caf19e3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:32:21.762796 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.762689    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-token\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:21.762796 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.762717    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-data\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:21.762796 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.762726    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-podinfo\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:21.762796 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.762733    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d40511-21fb-460a-8ccb-ed86caf19e3a-istio-envoy\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:21.762796 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:21.762742    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzpjz\" (UniqueName: \"kubernetes.io/projected/45d40511-21fb-460a-8ccb-ed86caf19e3a-kube-api-access-rzpjz\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:32:22.248175 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:22.248136    2580 generic.go:356] "Generic (PLEG): container finished" podID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerID="dd6d668a8841206fb5eb72fc518d3c597b8864f395b1d655d4f88a88ed5b3f94" exitCode=0
Sep 09 04:32:22.248397 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:22.248195    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"
Sep 09 04:32:22.248397 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:22.248225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" event={"ID":"45d40511-21fb-460a-8ccb-ed86caf19e3a","Type":"ContainerDied","Data":"dd6d668a8841206fb5eb72fc518d3c597b8864f395b1d655d4f88a88ed5b3f94"}
Sep 09 04:32:22.248397 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:22.248300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv" event={"ID":"45d40511-21fb-460a-8ccb-ed86caf19e3a","Type":"ContainerDied","Data":"e177bcc123dffae60e0bf85dd6b66c709817e2235e3636566d685b9c211eeca8"}
Sep 09 04:32:22.248397 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:22.248322    2580 scope.go:117] "RemoveContainer" containerID="dd6d668a8841206fb5eb72fc518d3c597b8864f395b1d655d4f88a88ed5b3f94"
Sep 09 04:32:22.260950 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:22.260920    2580 scope.go:117] "RemoveContainer" containerID="fd99db5a2e6b2d9e3fcffedba63d06f86e224eac622b303318816757fce973cc"
Sep 09 04:32:22.270081 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:22.270066    2580 scope.go:117] "RemoveContainer" containerID="dd6d668a8841206fb5eb72fc518d3c597b8864f395b1d655d4f88a88ed5b3f94"
Sep 09 04:32:22.270337 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:32:22.270318    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6d668a8841206fb5eb72fc518d3c597b8864f395b1d655d4f88a88ed5b3f94\": container with ID starting with dd6d668a8841206fb5eb72fc518d3c597b8864f395b1d655d4f88a88ed5b3f94 not found: ID does not exist" containerID="dd6d668a8841206fb5eb72fc518d3c597b8864f395b1d655d4f88a88ed5b3f94"
Sep 09 04:32:22.270383 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:22.270344    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6d668a8841206fb5eb72fc518d3c597b8864f395b1d655d4f88a88ed5b3f94"} err="failed to get container status \"dd6d668a8841206fb5eb72fc518d3c597b8864f395b1d655d4f88a88ed5b3f94\": rpc error: code = NotFound desc = could not find container \"dd6d668a8841206fb5eb72fc518d3c597b8864f395b1d655d4f88a88ed5b3f94\": container with ID starting with dd6d668a8841206fb5eb72fc518d3c597b8864f395b1d655d4f88a88ed5b3f94 not found: ID does not exist"
Sep 09 04:32:22.270383 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:22.270359    2580 scope.go:117] "RemoveContainer" containerID="fd99db5a2e6b2d9e3fcffedba63d06f86e224eac622b303318816757fce973cc"
Sep 09 04:32:22.270697 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:32:22.270674    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd99db5a2e6b2d9e3fcffedba63d06f86e224eac622b303318816757fce973cc\": container with ID starting with fd99db5a2e6b2d9e3fcffedba63d06f86e224eac622b303318816757fce973cc not found: ID does not exist" containerID="fd99db5a2e6b2d9e3fcffedba63d06f86e224eac622b303318816757fce973cc"
Sep 09 04:32:22.270794 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:22.270704    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd99db5a2e6b2d9e3fcffedba63d06f86e224eac622b303318816757fce973cc"} err="failed to get container status \"fd99db5a2e6b2d9e3fcffedba63d06f86e224eac622b303318816757fce973cc\": rpc error: code = NotFound desc = could not find container \"fd99db5a2e6b2d9e3fcffedba63d06f86e224eac622b303318816757fce973cc\": container with ID starting with fd99db5a2e6b2d9e3fcffedba63d06f86e224eac622b303318816757fce973cc not found: ID does not exist"
Sep 09 04:32:22.273908 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:22.273886    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"]
Sep 09 04:32:22.278508 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:22.278485    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9c44b5-openshift-default-5c6d7fb59b-rcpgv"]
Sep 09 04:32:23.595394 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:23.595364    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" path="/var/lib/kubelet/pods/45d40511-21fb-460a-8ccb-ed86caf19e3a/volumes"
Sep 09 04:32:59.737213 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:59.737185    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:32:59.748467 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:32:59.748446    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:33:00.078838 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:00.078723    2580 scope.go:117] "RemoveContainer" containerID="e48dd4985d810258ff9fce3da12bd3a6943754a0c0b1242bb22673c86282bcc4"
Sep 09 04:33:20.058346 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058310    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-pvg8c"]
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058708    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerName="istio-proxy"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058720    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerName="istio-proxy"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058730    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929a7f70-ddc5-40c9-a6a2-38dc554b341b" containerName="cleanup"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058735    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="929a7f70-ddc5-40c9-a6a2-38dc554b341b" containerName="cleanup"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058742    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerName="istio-proxy"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058750    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerName="istio-proxy"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058776    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058783    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058790    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058796    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058801    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669da265-2e72-437a-b0c5-19aa2d470b69" containerName="manager"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058806    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="669da265-2e72-437a-b0c5-19aa2d470b69" containerName="manager"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058815    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5051f50-3e61-4b4f-9358-be04824a254c" containerName="maas-api"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058820    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5051f50-3e61-4b4f-9358-be04824a254c" containerName="maas-api"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058829    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerName="istio-proxy"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058837    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerName="istio-proxy"
Sep 09 04:33:20.058859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058847    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerName="istio-proxy"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058874    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerName="istio-proxy"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058883    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cccaf17-68e3-4dc7-a97f-ce30b92a3100" containerName="maas-api"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058889    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cccaf17-68e3-4dc7-a97f-ce30b92a3100" containerName="maas-api"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058897    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerName="istio-proxy"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058902    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerName="istio-proxy"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058913    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c129ed2e-f4df-4123-8c67-86c27c2b04fa" containerName="maas-api"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058921    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c129ed2e-f4df-4123-8c67-86c27c2b04fa" containerName="maas-api"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058985    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerName="istio-proxy"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.058995    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerName="istio-proxy"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.059004    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerName="istio-proxy"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.059012    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.059017    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerName="istio-proxy"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.059022    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d40511-21fb-460a-8ccb-ed86caf19e3a" containerName="istio-proxy"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.059028    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="449e7ee8-cbb7-4103-867d-c22889f8c391" containerName="istio-proxy"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.059033    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5051f50-3e61-4b4f-9358-be04824a254c" containerName="maas-api"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.059038    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="929a7f70-ddc5-40c9-a6a2-38dc554b341b" containerName="cleanup"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.059042    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="669da265-2e72-437a-b0c5-19aa2d470b69" containerName="manager"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.059047    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cccaf17-68e3-4dc7-a97f-ce30b92a3100" containerName="maas-api"
Sep 09 04:33:20.059423 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.059051    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c129ed2e-f4df-4123-8c67-86c27c2b04fa" containerName="maas-api"
Sep 09 04:33:20.062179 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.062163    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:20.066161 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.066130    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 04:33:20.066161 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.066151    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kdm6s\""
Sep 09 04:33:20.066331 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.066142    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 04:33:20.066331 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.066141    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 04:33:20.072360 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.072335    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-pvg8c"]
Sep 09 04:33:20.165780 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.165738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vm8l\" (UniqueName: \"kubernetes.io/projected/80217b39-40d9-4508-8765-be37153b2ba5-kube-api-access-5vm8l\") pod \"maas-controller-7d8bb9d654-pvg8c\" (UID: \"80217b39-40d9-4508-8765-be37153b2ba5\") " pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:20.165976 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.165863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80217b39-40d9-4508-8765-be37153b2ba5-metrics-certs\") pod \"maas-controller-7d8bb9d654-pvg8c\" (UID: \"80217b39-40d9-4508-8765-be37153b2ba5\") " pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:20.165976 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.165923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80217b39-40d9-4508-8765-be37153b2ba5-webhook-cert\") pod \"maas-controller-7d8bb9d654-pvg8c\" (UID: \"80217b39-40d9-4508-8765-be37153b2ba5\") " pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:20.267373 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.267341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80217b39-40d9-4508-8765-be37153b2ba5-webhook-cert\") pod \"maas-controller-7d8bb9d654-pvg8c\" (UID: \"80217b39-40d9-4508-8765-be37153b2ba5\") " pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:20.267555 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.267394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vm8l\" (UniqueName: \"kubernetes.io/projected/80217b39-40d9-4508-8765-be37153b2ba5-kube-api-access-5vm8l\") pod \"maas-controller-7d8bb9d654-pvg8c\" (UID: \"80217b39-40d9-4508-8765-be37153b2ba5\") " pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:20.267555 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.267431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80217b39-40d9-4508-8765-be37153b2ba5-metrics-certs\") pod \"maas-controller-7d8bb9d654-pvg8c\" (UID: \"80217b39-40d9-4508-8765-be37153b2ba5\") " pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:20.269985 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.269960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80217b39-40d9-4508-8765-be37153b2ba5-metrics-certs\") pod \"maas-controller-7d8bb9d654-pvg8c\" (UID: \"80217b39-40d9-4508-8765-be37153b2ba5\") " pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:20.270106 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.269992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80217b39-40d9-4508-8765-be37153b2ba5-webhook-cert\") pod \"maas-controller-7d8bb9d654-pvg8c\" (UID: \"80217b39-40d9-4508-8765-be37153b2ba5\") " pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:20.276190 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.276167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vm8l\" (UniqueName: \"kubernetes.io/projected/80217b39-40d9-4508-8765-be37153b2ba5-kube-api-access-5vm8l\") pod \"maas-controller-7d8bb9d654-pvg8c\" (UID: \"80217b39-40d9-4508-8765-be37153b2ba5\") " pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:20.372545 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.372522    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:20.650317 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:20.650290    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-pvg8c"]
Sep 09 04:33:20.652464 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:33:20.652436    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80217b39_40d9_4508_8765_be37153b2ba5.slice/crio-d2eaaa08cb50ed9ec4458c122e3cb6bb093b8e5cb2a5f570dbcbe66d667078f8 WatchSource:0}: Error finding container d2eaaa08cb50ed9ec4458c122e3cb6bb093b8e5cb2a5f570dbcbe66d667078f8: Status 404 returned error can't find the container with id d2eaaa08cb50ed9ec4458c122e3cb6bb093b8e5cb2a5f570dbcbe66d667078f8
Sep 09 04:33:21.575559 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:21.575459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bb9d654-pvg8c" event={"ID":"80217b39-40d9-4508-8765-be37153b2ba5","Type":"ContainerStarted","Data":"bf7cf59c7b493c7aea4a6bf503f2e47eb6976f168c4436fdae87135772112b93"}
Sep 09 04:33:21.575559 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:21.575516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bb9d654-pvg8c" event={"ID":"80217b39-40d9-4508-8765-be37153b2ba5","Type":"ContainerStarted","Data":"d2eaaa08cb50ed9ec4458c122e3cb6bb093b8e5cb2a5f570dbcbe66d667078f8"}
Sep 09 04:33:21.576101 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:21.575594    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:21.594991 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:21.594941    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d8bb9d654-pvg8c" podStartSLOduration=0.98812265 podStartE2EDuration="1.594925618s" totalImagesPullingTime="606.802968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:33:21.593271132 +0000 UTC m=+2122.542961785" watchObservedRunningTime="2026-09-09 04:33:21.594925618 +0000 UTC m=+2122.544616272"
Sep 09 04:33:32.583369 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:32.583330    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:39.875524 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:39.875477    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-pvg8c"]
Sep 09 04:33:39.876095 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:39.875719    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7d8bb9d654-pvg8c" podUID="80217b39-40d9-4508-8765-be37153b2ba5" containerName="manager" containerID="cri-o://bf7cf59c7b493c7aea4a6bf503f2e47eb6976f168c4436fdae87135772112b93" gracePeriod=30
Sep 09 04:33:40.124877 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.124850    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:40.135431 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.135371    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vm8l\" (UniqueName: \"kubernetes.io/projected/80217b39-40d9-4508-8765-be37153b2ba5-kube-api-access-5vm8l\") pod \"80217b39-40d9-4508-8765-be37153b2ba5\" (UID: \"80217b39-40d9-4508-8765-be37153b2ba5\") "
Sep 09 04:33:40.135431 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.135416    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80217b39-40d9-4508-8765-be37153b2ba5-metrics-certs\") pod \"80217b39-40d9-4508-8765-be37153b2ba5\" (UID: \"80217b39-40d9-4508-8765-be37153b2ba5\") "
Sep 09 04:33:40.135594 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.135439    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80217b39-40d9-4508-8765-be37153b2ba5-webhook-cert\") pod \"80217b39-40d9-4508-8765-be37153b2ba5\" (UID: \"80217b39-40d9-4508-8765-be37153b2ba5\") "
Sep 09 04:33:40.137695 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.137659    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80217b39-40d9-4508-8765-be37153b2ba5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "80217b39-40d9-4508-8765-be37153b2ba5" (UID: "80217b39-40d9-4508-8765-be37153b2ba5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:33:40.137833 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.137704    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80217b39-40d9-4508-8765-be37153b2ba5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "80217b39-40d9-4508-8765-be37153b2ba5" (UID: "80217b39-40d9-4508-8765-be37153b2ba5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:33:40.137833 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.137727    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80217b39-40d9-4508-8765-be37153b2ba5-kube-api-access-5vm8l" (OuterVolumeSpecName: "kube-api-access-5vm8l") pod "80217b39-40d9-4508-8765-be37153b2ba5" (UID: "80217b39-40d9-4508-8765-be37153b2ba5"). InnerVolumeSpecName "kube-api-access-5vm8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:33:40.236920 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.236888    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vm8l\" (UniqueName: \"kubernetes.io/projected/80217b39-40d9-4508-8765-be37153b2ba5-kube-api-access-5vm8l\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:33:40.236920 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.236915    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80217b39-40d9-4508-8765-be37153b2ba5-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:33:40.236920 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.236925    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80217b39-40d9-4508-8765-be37153b2ba5-webhook-cert\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:33:40.688131 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.688098    2580 generic.go:356] "Generic (PLEG): container finished" podID="80217b39-40d9-4508-8765-be37153b2ba5" containerID="bf7cf59c7b493c7aea4a6bf503f2e47eb6976f168c4436fdae87135772112b93" exitCode=0
Sep 09 04:33:40.688307 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.688140    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bb9d654-pvg8c"
Sep 09 04:33:40.688307 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.688185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bb9d654-pvg8c" event={"ID":"80217b39-40d9-4508-8765-be37153b2ba5","Type":"ContainerDied","Data":"bf7cf59c7b493c7aea4a6bf503f2e47eb6976f168c4436fdae87135772112b93"}
Sep 09 04:33:40.688307 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.688240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bb9d654-pvg8c" event={"ID":"80217b39-40d9-4508-8765-be37153b2ba5","Type":"ContainerDied","Data":"d2eaaa08cb50ed9ec4458c122e3cb6bb093b8e5cb2a5f570dbcbe66d667078f8"}
Sep 09 04:33:40.688307 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.688262    2580 scope.go:117] "RemoveContainer" containerID="bf7cf59c7b493c7aea4a6bf503f2e47eb6976f168c4436fdae87135772112b93"
Sep 09 04:33:40.697990 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.697972    2580 scope.go:117] "RemoveContainer" containerID="bf7cf59c7b493c7aea4a6bf503f2e47eb6976f168c4436fdae87135772112b93"
Sep 09 04:33:40.698264 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:33:40.698246    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7cf59c7b493c7aea4a6bf503f2e47eb6976f168c4436fdae87135772112b93\": container with ID starting with bf7cf59c7b493c7aea4a6bf503f2e47eb6976f168c4436fdae87135772112b93 not found: ID does not exist" containerID="bf7cf59c7b493c7aea4a6bf503f2e47eb6976f168c4436fdae87135772112b93"
Sep 09 04:33:40.698313 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.698273    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7cf59c7b493c7aea4a6bf503f2e47eb6976f168c4436fdae87135772112b93"} err="failed to get container status \"bf7cf59c7b493c7aea4a6bf503f2e47eb6976f168c4436fdae87135772112b93\": rpc error: code = NotFound desc = could not find container \"bf7cf59c7b493c7aea4a6bf503f2e47eb6976f168c4436fdae87135772112b93\": container with ID starting with bf7cf59c7b493c7aea4a6bf503f2e47eb6976f168c4436fdae87135772112b93 not found: ID does not exist"
Sep 09 04:33:40.711061 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.711035    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-pvg8c"]
Sep 09 04:33:40.714559 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:40.714538    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-pvg8c"]
Sep 09 04:33:41.596146 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.596118    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80217b39-40d9-4508-8765-be37153b2ba5" path="/var/lib/kubelet/pods/80217b39-40d9-4508-8765-be37153b2ba5/volumes"
Sep 09 04:33:41.904221 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.904129    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-9bxzm"]
Sep 09 04:33:41.904499 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.904485    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80217b39-40d9-4508-8765-be37153b2ba5" containerName="manager"
Sep 09 04:33:41.904499 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.904499    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="80217b39-40d9-4508-8765-be37153b2ba5" containerName="manager"
Sep 09 04:33:41.904630 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.904512    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerName="istio-proxy"
Sep 09 04:33:41.904630 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.904518    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5810775-3101-4fdd-b58b-9c3ce7e5ae7e" containerName="istio-proxy"
Sep 09 04:33:41.904630 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.904581    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0793bc60-b1ec-46bd-99a5-828a08edd26f" containerName="istio-proxy"
Sep 09 04:33:41.904630 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.904589    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="80217b39-40d9-4508-8765-be37153b2ba5" containerName="manager"
Sep 09 04:33:41.908873 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.908851    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bb9d654-9bxzm"
Sep 09 04:33:41.914411 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.913909    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 04:33:41.914411 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.914019    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kdm6s\""
Sep 09 04:33:41.914411 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.914213    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 04:33:41.914411 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.914314    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 04:33:41.918589 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.918565    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-9bxzm"]
Sep 09 04:33:41.951207 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.951176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpct2\" (UniqueName: \"kubernetes.io/projected/813df945-8d18-4d86-88cc-9a459883955c-kube-api-access-gpct2\") pod \"maas-controller-7d8bb9d654-9bxzm\" (UID: \"813df945-8d18-4d86-88cc-9a459883955c\") " pod="opendatahub/maas-controller-7d8bb9d654-9bxzm"
Sep 09 04:33:41.951207 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.951208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/813df945-8d18-4d86-88cc-9a459883955c-metrics-certs\") pod \"maas-controller-7d8bb9d654-9bxzm\" (UID: \"813df945-8d18-4d86-88cc-9a459883955c\") " pod="opendatahub/maas-controller-7d8bb9d654-9bxzm"
Sep 09 04:33:41.951404 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:41.951306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/813df945-8d18-4d86-88cc-9a459883955c-webhook-cert\") pod \"maas-controller-7d8bb9d654-9bxzm\" (UID: \"813df945-8d18-4d86-88cc-9a459883955c\") " pod="opendatahub/maas-controller-7d8bb9d654-9bxzm"
Sep 09 04:33:42.052524 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:42.052484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/813df945-8d18-4d86-88cc-9a459883955c-metrics-certs\") pod \"maas-controller-7d8bb9d654-9bxzm\" (UID: \"813df945-8d18-4d86-88cc-9a459883955c\") " pod="opendatahub/maas-controller-7d8bb9d654-9bxzm"
Sep 09 04:33:42.052704 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:42.052543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/813df945-8d18-4d86-88cc-9a459883955c-webhook-cert\") pod \"maas-controller-7d8bb9d654-9bxzm\" (UID: \"813df945-8d18-4d86-88cc-9a459883955c\") " pod="opendatahub/maas-controller-7d8bb9d654-9bxzm"
Sep 09 04:33:42.052704 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:42.052597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpct2\" (UniqueName: \"kubernetes.io/projected/813df945-8d18-4d86-88cc-9a459883955c-kube-api-access-gpct2\") pod \"maas-controller-7d8bb9d654-9bxzm\" (UID: \"813df945-8d18-4d86-88cc-9a459883955c\") " pod="opendatahub/maas-controller-7d8bb9d654-9bxzm"
Sep 09 04:33:42.055212 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:42.055186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/813df945-8d18-4d86-88cc-9a459883955c-metrics-certs\") pod \"maas-controller-7d8bb9d654-9bxzm\" (UID: \"813df945-8d18-4d86-88cc-9a459883955c\") " pod="opendatahub/maas-controller-7d8bb9d654-9bxzm"
Sep 09 04:33:42.055331 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:42.055310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/813df945-8d18-4d86-88cc-9a459883955c-webhook-cert\") pod \"maas-controller-7d8bb9d654-9bxzm\" (UID: \"813df945-8d18-4d86-88cc-9a459883955c\") " pod="opendatahub/maas-controller-7d8bb9d654-9bxzm"
Sep 09 04:33:42.061094 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:42.061063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpct2\" (UniqueName: \"kubernetes.io/projected/813df945-8d18-4d86-88cc-9a459883955c-kube-api-access-gpct2\") pod \"maas-controller-7d8bb9d654-9bxzm\" (UID: \"813df945-8d18-4d86-88cc-9a459883955c\") " pod="opendatahub/maas-controller-7d8bb9d654-9bxzm"
Sep 09 04:33:42.222187 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:42.222094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8bb9d654-9bxzm"
Sep 09 04:33:42.497941 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:42.497918    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8bb9d654-9bxzm"]
Sep 09 04:33:42.499150 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:33:42.499121    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813df945_8d18_4d86_88cc_9a459883955c.slice/crio-416102cc7fe15ff139d4f3333757132c63809009036938dbd768bca3effb72c1 WatchSource:0}: Error finding container 416102cc7fe15ff139d4f3333757132c63809009036938dbd768bca3effb72c1: Status 404 returned error can't find the container with id 416102cc7fe15ff139d4f3333757132c63809009036938dbd768bca3effb72c1
Sep 09 04:33:42.500492 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:42.500476    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 04:33:42.701425 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:42.701390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bb9d654-9bxzm" event={"ID":"813df945-8d18-4d86-88cc-9a459883955c","Type":"ContainerStarted","Data":"416102cc7fe15ff139d4f3333757132c63809009036938dbd768bca3effb72c1"}
Sep 09 04:33:44.714374 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:44.714333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8bb9d654-9bxzm" event={"ID":"813df945-8d18-4d86-88cc-9a459883955c","Type":"ContainerStarted","Data":"db6e523a1ff4a700abc0be399272d7a69b6830e0ba239282ca28d29744b9dca2"}
Sep 09 04:33:44.714814 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:44.714477    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d8bb9d654-9bxzm"
Sep 09 04:33:44.735024 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:44.734975    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d8bb9d654-9bxzm" podStartSLOduration=2.591175206 podStartE2EDuration="3.734962605s" totalImagesPullingTime="1.143787399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:33:44.732522873 +0000 UTC m=+2145.682213530" watchObservedRunningTime="2026-09-09 04:33:44.734962605 +0000 UTC m=+2145.684653258"
Sep 09 04:33:55.723420 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:33:55.723341    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d8bb9d654-9bxzm"
Sep 09 04:35:05.690819 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:05.690777    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"]
Sep 09 04:35:05.693309 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:05.690978    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s" podUID="e495ced2-49ba-427f-b0db-30a2127ce723" containerName="manager" containerID="cri-o://23fa99eaa42f18ba8608c90c2335bea3c2552438efb6565f8e00e3b8da1cbdde" gracePeriod=10
Sep 09 04:35:06.173221 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:06.173185    2580 generic.go:356] "Generic (PLEG): container finished" podID="e495ced2-49ba-427f-b0db-30a2127ce723" containerID="23fa99eaa42f18ba8608c90c2335bea3c2552438efb6565f8e00e3b8da1cbdde" exitCode=0
Sep 09 04:35:06.173401 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:06.173256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s" event={"ID":"e495ced2-49ba-427f-b0db-30a2127ce723","Type":"ContainerDied","Data":"23fa99eaa42f18ba8608c90c2335bea3c2552438efb6565f8e00e3b8da1cbdde"}
Sep 09 04:35:06.444657 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:06.444633    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"
Sep 09 04:35:06.571866 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:06.571831    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4svh\" (UniqueName: \"kubernetes.io/projected/e495ced2-49ba-427f-b0db-30a2127ce723-kube-api-access-f4svh\") pod \"e495ced2-49ba-427f-b0db-30a2127ce723\" (UID: \"e495ced2-49ba-427f-b0db-30a2127ce723\") "
Sep 09 04:35:06.572030 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:06.571878    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e495ced2-49ba-427f-b0db-30a2127ce723-extensions-socket-volume\") pod \"e495ced2-49ba-427f-b0db-30a2127ce723\" (UID: \"e495ced2-49ba-427f-b0db-30a2127ce723\") "
Sep 09 04:35:06.572246 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:06.572224    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e495ced2-49ba-427f-b0db-30a2127ce723-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e495ced2-49ba-427f-b0db-30a2127ce723" (UID: "e495ced2-49ba-427f-b0db-30a2127ce723"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 04:35:06.574074 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:06.574055    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e495ced2-49ba-427f-b0db-30a2127ce723-kube-api-access-f4svh" (OuterVolumeSpecName: "kube-api-access-f4svh") pod "e495ced2-49ba-427f-b0db-30a2127ce723" (UID: "e495ced2-49ba-427f-b0db-30a2127ce723"). InnerVolumeSpecName "kube-api-access-f4svh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:35:06.672911 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:06.672870    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4svh\" (UniqueName: \"kubernetes.io/projected/e495ced2-49ba-427f-b0db-30a2127ce723-kube-api-access-f4svh\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:35:06.672911 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:06.672902    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e495ced2-49ba-427f-b0db-30a2127ce723-extensions-socket-volume\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:35:07.180281 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:07.180245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s" event={"ID":"e495ced2-49ba-427f-b0db-30a2127ce723","Type":"ContainerDied","Data":"670f68c8c4494aea462db4e5afc9b98d797467f098246c995611b43db1269c7b"}
Sep 09 04:35:07.180281 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:07.180268    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"
Sep 09 04:35:07.180841 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:07.180312    2580 scope.go:117] "RemoveContainer" containerID="23fa99eaa42f18ba8608c90c2335bea3c2552438efb6565f8e00e3b8da1cbdde"
Sep 09 04:35:07.205200 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:07.205162    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"]
Sep 09 04:35:07.209084 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:07.209060    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rs62s"]
Sep 09 04:35:07.595151 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:35:07.595124    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e495ced2-49ba-427f-b0db-30a2127ce723" path="/var/lib/kubelet/pods/e495ced2-49ba-427f-b0db-30a2127ce723/volumes"
Sep 09 04:36:06.388457 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.388416    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km"]
Sep 09 04:36:06.388955 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.388803    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e495ced2-49ba-427f-b0db-30a2127ce723" containerName="manager"
Sep 09 04:36:06.388955 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.388815    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e495ced2-49ba-427f-b0db-30a2127ce723" containerName="manager"
Sep 09 04:36:06.388955 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.388879    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e495ced2-49ba-427f-b0db-30a2127ce723" containerName="manager"
Sep 09 04:36:06.391879 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.391863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km"
Sep 09 04:36:06.394251 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.394224    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vg2bx\""
Sep 09 04:36:06.403250 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.403227    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km"]
Sep 09 04:36:06.483330 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.483289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-628km\" (UID: \"c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km"
Sep 09 04:36:06.483495 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.483369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jkdn\" (UniqueName: \"kubernetes.io/projected/c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe-kube-api-access-2jkdn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-628km\" (UID: \"c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km"
Sep 09 04:36:06.583843 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.583807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jkdn\" (UniqueName: \"kubernetes.io/projected/c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe-kube-api-access-2jkdn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-628km\" (UID: \"c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km"
Sep 09 04:36:06.584050 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.583882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-628km\" (UID: \"c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km"
Sep 09 04:36:06.584231 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.584209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-628km\" (UID: \"c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km"
Sep 09 04:36:06.592924 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.592895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jkdn\" (UniqueName: \"kubernetes.io/projected/c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe-kube-api-access-2jkdn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-628km\" (UID: \"c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km"
Sep 09 04:36:06.702941 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.702863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km"
Sep 09 04:36:06.979723 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:06.979696    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km"]
Sep 09 04:36:06.981473 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:36:06.981446    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e7817d_2e4d_4251_a0c3_d10c7cbc99fe.slice/crio-61aecf9a5102a95ad83b4d15274e0a03f48ee878455e3967eac3badbd33c5a10 WatchSource:0}: Error finding container 61aecf9a5102a95ad83b4d15274e0a03f48ee878455e3967eac3badbd33c5a10: Status 404 returned error can't find the container with id 61aecf9a5102a95ad83b4d15274e0a03f48ee878455e3967eac3badbd33c5a10
Sep 09 04:36:07.514718 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:07.514686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km" event={"ID":"c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe","Type":"ContainerStarted","Data":"5f7a1b255111106e57d55b1df68b061baa8471f880c9d5d52a4009d973ee19ba"}
Sep 09 04:36:07.515144 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:07.514737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km" event={"ID":"c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe","Type":"ContainerStarted","Data":"61aecf9a5102a95ad83b4d15274e0a03f48ee878455e3967eac3badbd33c5a10"}
Sep 09 04:36:07.515144 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:07.514758    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km"
Sep 09 04:36:07.545284 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:07.545243    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km" podStartSLOduration=1.545230281 podStartE2EDuration="1.545230281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:36:07.543998714 +0000 UTC m=+2288.493689367" watchObservedRunningTime="2026-09-09 04:36:07.545230281 +0000 UTC m=+2288.494920934"
Sep 09 04:36:18.522305 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:36:18.522272    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-628km"
Sep 09 04:37:59.773393 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:37:59.773369    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:37:59.787263 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:37:59.787241    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6f48_2a433ebe-7762-4b8a-b605-418518973e05/ovn-acl-logging/0.log"
Sep 09 04:38:03.177859 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.177817    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"]
Sep 09 04:38:03.181407 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.181387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.192366 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.192340    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"]
Sep 09 04:38:03.323911 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.323875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5701bf83-d63d-45ff-9ce9-f85702350925-metrics-certs\") pod \"maas-api-f6c58f77f-ctxhr\" (UID: \"5701bf83-d63d-45ff-9ce9-f85702350925\") " pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.323911 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.323911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5701bf83-d63d-45ff-9ce9-f85702350925-openshift-service-ca\") pod \"maas-api-f6c58f77f-ctxhr\" (UID: \"5701bf83-d63d-45ff-9ce9-f85702350925\") " pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.324116 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.324035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5701bf83-d63d-45ff-9ce9-f85702350925-maas-api-tls\") pod \"maas-api-f6c58f77f-ctxhr\" (UID: \"5701bf83-d63d-45ff-9ce9-f85702350925\") " pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.324116 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.324078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhzg4\" (UniqueName: \"kubernetes.io/projected/5701bf83-d63d-45ff-9ce9-f85702350925-kube-api-access-rhzg4\") pod \"maas-api-f6c58f77f-ctxhr\" (UID: \"5701bf83-d63d-45ff-9ce9-f85702350925\") " pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.424639 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.424602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5701bf83-d63d-45ff-9ce9-f85702350925-maas-api-tls\") pod \"maas-api-f6c58f77f-ctxhr\" (UID: \"5701bf83-d63d-45ff-9ce9-f85702350925\") " pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.424856 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.424664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhzg4\" (UniqueName: \"kubernetes.io/projected/5701bf83-d63d-45ff-9ce9-f85702350925-kube-api-access-rhzg4\") pod \"maas-api-f6c58f77f-ctxhr\" (UID: \"5701bf83-d63d-45ff-9ce9-f85702350925\") " pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.424856 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.424688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5701bf83-d63d-45ff-9ce9-f85702350925-metrics-certs\") pod \"maas-api-f6c58f77f-ctxhr\" (UID: \"5701bf83-d63d-45ff-9ce9-f85702350925\") " pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.424856 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.424709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5701bf83-d63d-45ff-9ce9-f85702350925-openshift-service-ca\") pod \"maas-api-f6c58f77f-ctxhr\" (UID: \"5701bf83-d63d-45ff-9ce9-f85702350925\") " pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.425409 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.425386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5701bf83-d63d-45ff-9ce9-f85702350925-openshift-service-ca\") pod \"maas-api-f6c58f77f-ctxhr\" (UID: \"5701bf83-d63d-45ff-9ce9-f85702350925\") " pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.427453 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.427427    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5701bf83-d63d-45ff-9ce9-f85702350925-maas-api-tls\") pod \"maas-api-f6c58f77f-ctxhr\" (UID: \"5701bf83-d63d-45ff-9ce9-f85702350925\") " pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.427561 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.427465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5701bf83-d63d-45ff-9ce9-f85702350925-metrics-certs\") pod \"maas-api-f6c58f77f-ctxhr\" (UID: \"5701bf83-d63d-45ff-9ce9-f85702350925\") " pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.433202 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.433145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhzg4\" (UniqueName: \"kubernetes.io/projected/5701bf83-d63d-45ff-9ce9-f85702350925-kube-api-access-rhzg4\") pod \"maas-api-f6c58f77f-ctxhr\" (UID: \"5701bf83-d63d-45ff-9ce9-f85702350925\") " pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.493635 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.493608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:03.769533 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:03.769501    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"]
Sep 09 04:38:03.770949 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:38:03.770914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5701bf83_d63d_45ff_9ce9_f85702350925.slice/crio-a359f5f318135dae42a96c573594647757be39231f8c6087846320bf6c063066 WatchSource:0}: Error finding container a359f5f318135dae42a96c573594647757be39231f8c6087846320bf6c063066: Status 404 returned error can't find the container with id a359f5f318135dae42a96c573594647757be39231f8c6087846320bf6c063066
Sep 09 04:38:04.178038 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:04.178002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr" event={"ID":"5701bf83-d63d-45ff-9ce9-f85702350925","Type":"ContainerStarted","Data":"a359f5f318135dae42a96c573594647757be39231f8c6087846320bf6c063066"}
Sep 09 04:38:05.184688 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:05.184643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr" event={"ID":"5701bf83-d63d-45ff-9ce9-f85702350925","Type":"ContainerStarted","Data":"d8a861296477a06e6d0ec92ee4b3a1c7b41f91c3caecb938fd03ecc4a1f23992"}
Sep 09 04:38:05.185135 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:05.184705    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:05.204176 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:05.204124    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr" podStartSLOduration=1.754086076 podStartE2EDuration="2.204110275s" totalImagesPullingTime="450.024199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:38:05.202638724 +0000 UTC m=+2406.152329379" watchObservedRunningTime="2026-09-09 04:38:05.204110275 +0000 UTC m=+2406.153800986"
Sep 09 04:38:11.195884 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:11.195851    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f6c58f77f-ctxhr"
Sep 09 04:38:19.324872 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:19.324825    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-69989b7c55-tks28"]
Sep 09 04:38:19.325355 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:19.325106    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28" podUID="436875c8-aed4-4d1d-bcf8-57eaa9a84f88" containerName="maas-api" containerID="cri-o://9de7f349f07c39cb32989003cf66defa4d0bd5d39da9b1ce10562da94d801dc4" gracePeriod=30
Sep 09 04:38:20.408368 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.408324    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28" podUID="436875c8-aed4-4d1d-bcf8-57eaa9a84f88" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.55:8443/health\": dial tcp 10.133.0.55:8443: connect: connection refused"
Sep 09 04:38:20.575237 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.575207    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:38:20.671807 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.671689    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-maas-api-tls\") pod \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") "
Sep 09 04:38:20.671807 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.671739    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-openshift-service-ca\") pod \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") "
Sep 09 04:38:20.672039 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.671881    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m47w\" (UniqueName: \"kubernetes.io/projected/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-kube-api-access-4m47w\") pod \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") "
Sep 09 04:38:20.672039 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.671910    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-metrics-certs\") pod \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\" (UID: \"436875c8-aed4-4d1d-bcf8-57eaa9a84f88\") "
Sep 09 04:38:20.672178 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.672149    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "436875c8-aed4-4d1d-bcf8-57eaa9a84f88" (UID: "436875c8-aed4-4d1d-bcf8-57eaa9a84f88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 04:38:20.674206 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.674180    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-kube-api-access-4m47w" (OuterVolumeSpecName: "kube-api-access-4m47w") pod "436875c8-aed4-4d1d-bcf8-57eaa9a84f88" (UID: "436875c8-aed4-4d1d-bcf8-57eaa9a84f88"). InnerVolumeSpecName "kube-api-access-4m47w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 04:38:20.674320 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.674233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "436875c8-aed4-4d1d-bcf8-57eaa9a84f88" (UID: "436875c8-aed4-4d1d-bcf8-57eaa9a84f88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:38:20.674320 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.674253    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "436875c8-aed4-4d1d-bcf8-57eaa9a84f88" (UID: "436875c8-aed4-4d1d-bcf8-57eaa9a84f88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 04:38:20.772956 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.772914    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m47w\" (UniqueName: \"kubernetes.io/projected/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-kube-api-access-4m47w\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:38:20.772956 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.772948    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-metrics-certs\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:38:20.772956 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.772959    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-maas-api-tls\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:38:20.772956 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:20.772966    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/436875c8-aed4-4d1d-bcf8-57eaa9a84f88-openshift-service-ca\") on node \"ip-10-0-142-198.ec2.internal\" DevicePath \"\""
Sep 09 04:38:21.275511 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:21.275475    2580 generic.go:356] "Generic (PLEG): container finished" podID="436875c8-aed4-4d1d-bcf8-57eaa9a84f88" containerID="9de7f349f07c39cb32989003cf66defa4d0bd5d39da9b1ce10562da94d801dc4" exitCode=0
Sep 09 04:38:21.275677 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:21.275527    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28"
Sep 09 04:38:21.275677 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:21.275525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28" event={"ID":"436875c8-aed4-4d1d-bcf8-57eaa9a84f88","Type":"ContainerDied","Data":"9de7f349f07c39cb32989003cf66defa4d0bd5d39da9b1ce10562da94d801dc4"}
Sep 09 04:38:21.275677 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:21.275659    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-69989b7c55-tks28" event={"ID":"436875c8-aed4-4d1d-bcf8-57eaa9a84f88","Type":"ContainerDied","Data":"f6e9a61632c666ae27e317bb1345c79fa18999ed84cc8c361f5d7bba9708f114"}
Sep 09 04:38:21.275804 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:21.275690    2580 scope.go:117] "RemoveContainer" containerID="9de7f349f07c39cb32989003cf66defa4d0bd5d39da9b1ce10562da94d801dc4"
Sep 09 04:38:21.290333 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:21.290310    2580 scope.go:117] "RemoveContainer" containerID="9de7f349f07c39cb32989003cf66defa4d0bd5d39da9b1ce10562da94d801dc4"
Sep 09 04:38:21.290631 ip-10-0-142-198 kubenswrapper[2580]: E0909 04:38:21.290608    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de7f349f07c39cb32989003cf66defa4d0bd5d39da9b1ce10562da94d801dc4\": container with ID starting with 9de7f349f07c39cb32989003cf66defa4d0bd5d39da9b1ce10562da94d801dc4 not found: ID does not exist" containerID="9de7f349f07c39cb32989003cf66defa4d0bd5d39da9b1ce10562da94d801dc4"
Sep 09 04:38:21.290676 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:21.290650    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de7f349f07c39cb32989003cf66defa4d0bd5d39da9b1ce10562da94d801dc4"} err="failed to get container status \"9de7f349f07c39cb32989003cf66defa4d0bd5d39da9b1ce10562da94d801dc4\": rpc error: code = NotFound desc = could not find container \"9de7f349f07c39cb32989003cf66defa4d0bd5d39da9b1ce10562da94d801dc4\": container with ID starting with 9de7f349f07c39cb32989003cf66defa4d0bd5d39da9b1ce10562da94d801dc4 not found: ID does not exist"
Sep 09 04:38:21.303220 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:21.303196    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-69989b7c55-tks28"]
Sep 09 04:38:21.308102 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:21.308078    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-69989b7c55-tks28"]
Sep 09 04:38:21.596644 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:21.596611    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="436875c8-aed4-4d1d-bcf8-57eaa9a84f88" path="/var/lib/kubelet/pods/436875c8-aed4-4d1d-bcf8-57eaa9a84f88/volumes"
Sep 09 04:38:38.206357 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:38.206325    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77448c7dc7-whv4f_d07af248-a22d-4433-878d-594d325a1e6c/authorino/0.log"
Sep 09 04:38:43.670776 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:43.670727    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zxrqw_74648403-ac49-4cf3-8930-aa20f79e15b8/manager/0.log"
Sep 09 04:38:43.788630 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:43.788601    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7gnhz_6b7c20ec-f7ca-484b-a33a-e1906aae95f7/manager/0.log"
Sep 09 04:38:43.907390 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:43.907357    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d8bb9d654-9bxzm_813df945-8d18-4d86-88cc-9a459883955c/manager/0.log"
Sep 09 04:38:44.287635 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:44.287602    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5996d6cf95-89nxx_2f73fe7e-2f01-4659-8c9d-2e0411b75d66/copy-manifests/0.log"
Sep 09 04:38:44.296809 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:44.296789    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5996d6cf95-89nxx_2f73fe7e-2f01-4659-8c9d-2e0411b75d66/manager/0.log"
Sep 09 04:38:45.032308 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:45.032276    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f6c58f77f-ctxhr_5701bf83-d63d-45ff-9ce9-f85702350925/maas-api/0.log"
Sep 09 04:38:45.576043 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:45.576016    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w4-85732c-29815470-dgh7w_929a7f70-ddc5-40c9-a6a2-38dc554b341b/cleanup/0.log"
Sep 09 04:38:45.690016 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:45.689989    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-ztlvl_6395fdb6-dda5-432d-834b-a581d4ef7596/postgres/0.log"
Sep 09 04:38:46.467244 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:46.467213    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77448c7dc7-whv4f_d07af248-a22d-4433-878d-594d325a1e6c/authorino/0.log"
Sep 09 04:38:46.581373 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:46.581345    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-bxj4t_c7cea49b-35fd-455c-9e83-f133121b85c6/manager/0.log"
Sep 09 04:38:46.692391 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:46.692363    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-wlj2s_a9bb8845-4fff-4727-af27-fd9ffa666500/manager/0.log"
Sep 09 04:38:46.808265 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:46.808190    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xgxqg_ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f/kuadrant-console-plugin/0.log"
Sep 09 04:38:46.929074 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:46.929042    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-628km_c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe/manager/0.log"
Sep 09 04:38:47.175851 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:47.175813    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-czlk7_2c51c71e-8575-4a24-8ad9-985e11d84942/manager/0.log"
Sep 09 04:38:47.649354 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:47.649322    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp_f5ec7505-fc62-4ce0-9548-28b278a1c2cc/istio-proxy/1.log"
Sep 09 04:38:48.012671 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:48.012589    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7449ffbbd4-v2fcj_42362810-b011-4dd6-96d6-d82ff2685386/kube-auth-proxy/0.log"
Sep 09 04:38:48.249909 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:48.249876    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fkvbs_150e74af-5522-4537-ad9b-e4dbae244cc6/payload-pre-processing/0.log"
Sep 09 04:38:49.065438 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:49.065412    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-769pg_5ea9cce7-b914-409b-a2f4-d9a2ef99ac93/main/0.log"
Sep 09 04:38:49.403639 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:49.403610    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-pjw44_f7433f3e-fee6-4f46-be9c-519d7f3bafc1/main/0.log"
Sep 09 04:38:55.577570 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.577542    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-smpj7/must-gather-njzjw"]
Sep 09 04:38:55.577958 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.577939    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="436875c8-aed4-4d1d-bcf8-57eaa9a84f88" containerName="maas-api"
Sep 09 04:38:55.577958 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.577952    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="436875c8-aed4-4d1d-bcf8-57eaa9a84f88" containerName="maas-api"
Sep 09 04:38:55.578036 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.578023    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="436875c8-aed4-4d1d-bcf8-57eaa9a84f88" containerName="maas-api"
Sep 09 04:38:55.581561 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.581541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-smpj7/must-gather-njzjw"
Sep 09 04:38:55.584464 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.584442    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-smpj7\"/\"openshift-service-ca.crt\""
Sep 09 04:38:55.584549 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.584519    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-smpj7\"/\"kube-root-ca.crt\""
Sep 09 04:38:55.584599 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.584553    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-smpj7\"/\"default-dockercfg-s57v2\""
Sep 09 04:38:55.589011 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.588932    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-smpj7/must-gather-njzjw"]
Sep 09 04:38:55.687152 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.687118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8hns\" (UniqueName: \"kubernetes.io/projected/9fc83e9a-d5a8-4f5c-988d-454af6ef3afe-kube-api-access-q8hns\") pod \"must-gather-njzjw\" (UID: \"9fc83e9a-d5a8-4f5c-988d-454af6ef3afe\") " pod="openshift-must-gather-smpj7/must-gather-njzjw"
Sep 09 04:38:55.687329 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.687164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9fc83e9a-d5a8-4f5c-988d-454af6ef3afe-must-gather-output\") pod \"must-gather-njzjw\" (UID: \"9fc83e9a-d5a8-4f5c-988d-454af6ef3afe\") " pod="openshift-must-gather-smpj7/must-gather-njzjw"
Sep 09 04:38:55.788662 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.788634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8hns\" (UniqueName: \"kubernetes.io/projected/9fc83e9a-d5a8-4f5c-988d-454af6ef3afe-kube-api-access-q8hns\") pod \"must-gather-njzjw\" (UID: \"9fc83e9a-d5a8-4f5c-988d-454af6ef3afe\") " pod="openshift-must-gather-smpj7/must-gather-njzjw"
Sep 09 04:38:55.788835 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.788668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9fc83e9a-d5a8-4f5c-988d-454af6ef3afe-must-gather-output\") pod \"must-gather-njzjw\" (UID: \"9fc83e9a-d5a8-4f5c-988d-454af6ef3afe\") " pod="openshift-must-gather-smpj7/must-gather-njzjw"
Sep 09 04:38:55.788983 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.788967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9fc83e9a-d5a8-4f5c-988d-454af6ef3afe-must-gather-output\") pod \"must-gather-njzjw\" (UID: \"9fc83e9a-d5a8-4f5c-988d-454af6ef3afe\") " pod="openshift-must-gather-smpj7/must-gather-njzjw"
Sep 09 04:38:55.800751 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.800728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8hns\" (UniqueName: \"kubernetes.io/projected/9fc83e9a-d5a8-4f5c-988d-454af6ef3afe-kube-api-access-q8hns\") pod \"must-gather-njzjw\" (UID: \"9fc83e9a-d5a8-4f5c-988d-454af6ef3afe\") " pod="openshift-must-gather-smpj7/must-gather-njzjw"
Sep 09 04:38:55.892255 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:55.892230    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-smpj7/must-gather-njzjw"
Sep 09 04:38:56.165989 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:56.165961    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-smpj7/must-gather-njzjw"]
Sep 09 04:38:56.166948 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:38:56.166912    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fc83e9a_d5a8_4f5c_988d_454af6ef3afe.slice/crio-82634ef48ddccc26fce5e9c6eb44bfd898afc6845fa5762f5f355657bb8afadf WatchSource:0}: Error finding container 82634ef48ddccc26fce5e9c6eb44bfd898afc6845fa5762f5f355657bb8afadf: Status 404 returned error can't find the container with id 82634ef48ddccc26fce5e9c6eb44bfd898afc6845fa5762f5f355657bb8afadf
Sep 09 04:38:56.168871 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:56.168852    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 04:38:56.487065 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:56.486982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-smpj7/must-gather-njzjw" event={"ID":"9fc83e9a-d5a8-4f5c-988d-454af6ef3afe","Type":"ContainerStarted","Data":"82634ef48ddccc26fce5e9c6eb44bfd898afc6845fa5762f5f355657bb8afadf"}
Sep 09 04:38:57.498981 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:57.498574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-smpj7/must-gather-njzjw" event={"ID":"9fc83e9a-d5a8-4f5c-988d-454af6ef3afe","Type":"ContainerStarted","Data":"6942252618e00d996c95fbe3a04f5aea867265eb3b72c3490d1756b46b713d43"}
Sep 09 04:38:57.498981 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:57.498632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-smpj7/must-gather-njzjw" event={"ID":"9fc83e9a-d5a8-4f5c-988d-454af6ef3afe","Type":"ContainerStarted","Data":"26f2e3712612fbf024accf244af504b3ad0fa84dfdb06834f5815ef06d5748e1"}
Sep 09 04:38:57.517070 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:57.517001    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-smpj7/must-gather-njzjw" podStartSLOduration=1.604327085 podStartE2EDuration="2.516979598s" totalImagesPullingTime="912.652513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 04:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:38:57.515063911 +0000 UTC m=+2458.464754566" watchObservedRunningTime="2026-09-09 04:38:57.516979598 +0000 UTC m=+2458.466670253"
Sep 09 04:38:58.808095 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:58.808062    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-brsc7_93076315-afaf-43ce-a72c-104751d6d4e2/global-pull-secret-syncer/0.log"
Sep 09 04:38:58.930233 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:58.930200    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6j4fw_22cce36d-d1ae-4ae8-b3dc-30c4f8d38af3/konnectivity-agent/0.log"
Sep 09 04:38:59.031602 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:38:59.031568    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-198.ec2.internal_643c72097878f1443f4703f561b5b845/haproxy/0.log"
Sep 09 04:39:03.789418 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:03.789386    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77448c7dc7-whv4f_d07af248-a22d-4433-878d-594d325a1e6c/authorino/0.log"
Sep 09 04:39:03.822513 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:03.822403    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-bxj4t_c7cea49b-35fd-455c-9e83-f133121b85c6/manager/0.log"
Sep 09 04:39:03.855547 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:03.855515    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-wlj2s_a9bb8845-4fff-4727-af27-fd9ffa666500/manager/0.log"
Sep 09 04:39:03.886163 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:03.886135    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xgxqg_ff34cbf2-3a19-4fc7-a35f-cd4bee42be4f/kuadrant-console-plugin/0.log"
Sep 09 04:39:03.945384 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:03.945344    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-628km_c6e7817d-2e4d-4251-a0c3-d10c7cbc99fe/manager/0.log"
Sep 09 04:39:04.051661 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:04.051557    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-czlk7_2c51c71e-8575-4a24-8ad9-985e11d84942/manager/0.log"
Sep 09 04:39:06.193011 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:06.192982    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lp9vx_fb17bdc5-dc3f-4873-a01a-5c9d11e36536/node-exporter/0.log"
Sep 09 04:39:06.220726 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:06.220688    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lp9vx_fb17bdc5-dc3f-4873-a01a-5c9d11e36536/kube-rbac-proxy/0.log"
Sep 09 04:39:06.251141 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:06.251115    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lp9vx_fb17bdc5-dc3f-4873-a01a-5c9d11e36536/init-textfile/0.log"
Sep 09 04:39:06.708741 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:06.708710    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-cngkj_003d3d43-36cc-4be1-9caf-785bfd0725c0/prometheus-operator-admission-webhook/0.log"
Sep 09 04:39:08.298850 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:08.298819    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-tfn7q_1fa279b0-1e9c-454d-b0e8-24d193ca3476/networking-console-plugin/0.log"
Sep 09 04:39:09.286023 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.285980    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"]
Sep 09 04:39:09.292211 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.292182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.297952 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.297923    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"]
Sep 09 04:39:09.333349 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.332899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9skq9\" (UniqueName: \"kubernetes.io/projected/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-kube-api-access-9skq9\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.333349 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.333004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-sys\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.333349 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.333104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-podres\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.333349 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.333170    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-lib-modules\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.333349 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.333200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-proc\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.433942 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.433906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-lib-modules\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.434207 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.434186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-proc\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.434331 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.434304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-proc\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.434509 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.434354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9skq9\" (UniqueName: \"kubernetes.io/projected/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-kube-api-access-9skq9\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.434652 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.434635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-sys\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.434870 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.434850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-podres\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.435059 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.434697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-sys\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.435166 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.434248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-lib-modules\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.435268 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.435064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-podres\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.443959 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.443919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9skq9\" (UniqueName: \"kubernetes.io/projected/ff48bc82-2f4b-48d5-99c9-9d69d0b22421-kube-api-access-9skq9\") pod \"perf-node-gather-daemonset-npcf4\" (UID: \"ff48bc82-2f4b-48d5-99c9-9d69d0b22421\") " pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.542115 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.542050    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-749ddb48fc-qzxrp_1abfba12-c8de-4e3f-9041-191668c88c7b/console/0.log"
Sep 09 04:39:09.577067 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.577031    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-n4s2s_0f6327e8-5787-4a47-accc-50e18269c70c/download-server/0.log"
Sep 09 04:39:09.606990 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.606958    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:09.912596 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:09.912559    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"]
Sep 09 04:39:09.917037 ip-10-0-142-198 kubenswrapper[2580]: W0909 04:39:09.916994    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podff48bc82_2f4b_48d5_99c9_9d69d0b22421.slice/crio-dfe2aaf64914dfafc70cc7038107a4d085056a4227ede8bd725d03cab6421264 WatchSource:0}: Error finding container dfe2aaf64914dfafc70cc7038107a4d085056a4227ede8bd725d03cab6421264: Status 404 returned error can't find the container with id dfe2aaf64914dfafc70cc7038107a4d085056a4227ede8bd725d03cab6421264
Sep 09 04:39:10.622549 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:10.622496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4" event={"ID":"ff48bc82-2f4b-48d5-99c9-9d69d0b22421","Type":"ContainerStarted","Data":"b29030a0ac18829b476a48fff51d4c5e09ce500316696dbc0bfd287a94a7952c"}
Sep 09 04:39:10.623064 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:10.622594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4" event={"ID":"ff48bc82-2f4b-48d5-99c9-9d69d0b22421","Type":"ContainerStarted","Data":"dfe2aaf64914dfafc70cc7038107a4d085056a4227ede8bd725d03cab6421264"}
Sep 09 04:39:10.623498 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:10.623473    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:10.653973 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:10.653902    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4" podStartSLOduration=1.653873843 podStartE2EDuration="1.653873843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 04:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 04:39:10.642015787 +0000 UTC m=+2471.591706453" watchObservedRunningTime="2026-09-09 04:39:10.653873843 +0000 UTC m=+2471.603564498"
Sep 09 04:39:11.611027 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:11.610999    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vhd4r_f850a46e-0840-481a-8f86-0b3877277387/dns/0.log"
Sep 09 04:39:11.633620 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:11.633593    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vhd4r_f850a46e-0840-481a-8f86-0b3877277387/kube-rbac-proxy/0.log"
Sep 09 04:39:11.683235 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:11.683208    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h6bnp_1537ac59-5afb-4843-aa47-39dbbaf34a47/dns-node-resolver/0.log"
Sep 09 04:39:12.270002 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:12.269972    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55bd5d57b-h568l_77411f50-af38-4530-ac82-8d64ced36874/registry/0.log"
Sep 09 04:39:12.291879 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:12.291835    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6h7v6_f88a2c9f-c508-4bab-8a3a-eb3197e93ccb/node-ca/0.log"
Sep 09 04:39:13.418739 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:13.418709    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp_f5ec7505-fc62-4ce0-9548-28b278a1c2cc/istio-proxy/1.log"
Sep 09 04:39:13.419467 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:13.419256    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp_f5ec7505-fc62-4ce0-9548-28b278a1c2cc/istio-proxy/0.log"
Sep 09 04:39:13.736165 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:13.736057    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7449ffbbd4-v2fcj_42362810-b011-4dd6-96d6-d82ff2685386/kube-auth-proxy/0.log"
Sep 09 04:39:13.815189 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:13.815150    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fkvbs_150e74af-5522-4537-ad9b-e4dbae244cc6/payload-pre-processing/0.log"
Sep 09 04:39:14.933085 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:14.933057    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vlhhj_98eb9f21-8704-4f75-b980-19a618f95afc/serve-healthcheck-canary/0.log"
Sep 09 04:39:15.597415 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:15.597379    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ngtdv_a18afeda-adec-4705-b816-f8bdac47e629/kube-rbac-proxy/0.log"
Sep 09 04:39:15.621292 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:15.621265    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ngtdv_a18afeda-adec-4705-b816-f8bdac47e629/exporter/0.log"
Sep 09 04:39:15.663633 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:15.663606    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ngtdv_a18afeda-adec-4705-b816-f8bdac47e629/extractor/0.log"
Sep 09 04:39:17.647684 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:17.647651    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-smpj7/perf-node-gather-daemonset-npcf4"
Sep 09 04:39:17.817027 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:17.817002    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zxrqw_74648403-ac49-4cf3-8930-aa20f79e15b8/manager/0.log"
Sep 09 04:39:18.096945 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:18.096915    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7gnhz_6b7c20ec-f7ca-484b-a33a-e1906aae95f7/manager/0.log"
Sep 09 04:39:18.155407 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:18.155375    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d8bb9d654-9bxzm_813df945-8d18-4d86-88cc-9a459883955c/manager/0.log"
Sep 09 04:39:18.284384 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:18.284346    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5996d6cf95-89nxx_2f73fe7e-2f01-4659-8c9d-2e0411b75d66/manager/0.log"
Sep 09 04:39:18.305325 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:18.305296    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5996d6cf95-89nxx_2f73fe7e-2f01-4659-8c9d-2e0411b75d66/copy-manifests/0.log"
Sep 09 04:39:19.806138 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:19.806104    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp_f5ec7505-fc62-4ce0-9548-28b278a1c2cc/istio-proxy/1.log"
Sep 09 04:39:19.806690 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:19.806553    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp_f5ec7505-fc62-4ce0-9548-28b278a1c2cc/istio-proxy/0.log"
Sep 09 04:39:19.990487 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:19.990457    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-9dfsr_29e90e45-8728-4aad-9d64-9efbf18ae04f/openshift-lws-operator/0.log"
Sep 09 04:39:20.094179 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:20.094103    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7449ffbbd4-v2fcj_42362810-b011-4dd6-96d6-d82ff2685386/kube-auth-proxy/0.log"
Sep 09 04:39:20.169665 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:20.169636    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fkvbs_150e74af-5522-4537-ad9b-e4dbae244cc6/payload-pre-processing/0.log"
Sep 09 04:39:21.872025 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:21.871987    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp_f5ec7505-fc62-4ce0-9548-28b278a1c2cc/istio-proxy/0.log"
Sep 09 04:39:21.872428 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:21.872033    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-worker-w4-85732c-openshift-default-5879bccbcb-8tthp_f5ec7505-fc62-4ce0-9548-28b278a1c2cc/istio-proxy/1.log"
Sep 09 04:39:22.170391 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:22.170361    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7449ffbbd4-v2fcj_42362810-b011-4dd6-96d6-d82ff2685386/kube-auth-proxy/0.log"
Sep 09 04:39:22.235236 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:22.235194    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fkvbs_150e74af-5522-4537-ad9b-e4dbae244cc6/payload-pre-processing/0.log"
Sep 09 04:39:25.458443 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:25.458409    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-s9k24_e45776a2-d18f-4bc6-8b63-fa62745dc1a1/migrator/0.log"
Sep 09 04:39:25.480356 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:25.480327    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-s9k24_e45776a2-d18f-4bc6-8b63-fa62745dc1a1/graceful-termination/0.log"
Sep 09 04:39:27.276731 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:27.276698    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x5fkn_aa72aab2-7d1d-4c62-83e3-a01afac7ee1c/kube-multus-additional-cni-plugins/0.log"
Sep 09 04:39:27.298963 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:27.298939    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x5fkn_aa72aab2-7d1d-4c62-83e3-a01afac7ee1c/egress-router-binary-copy/0.log"
Sep 09 04:39:27.320263 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:27.320242    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x5fkn_aa72aab2-7d1d-4c62-83e3-a01afac7ee1c/cni-plugins/0.log"
Sep 09 04:39:27.343118 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:27.343090    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x5fkn_aa72aab2-7d1d-4c62-83e3-a01afac7ee1c/bond-cni-plugin/0.log"
Sep 09 04:39:27.366651 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:27.366603    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x5fkn_aa72aab2-7d1d-4c62-83e3-a01afac7ee1c/routeoverride-cni/0.log"
Sep 09 04:39:27.388309 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:27.388288    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x5fkn_aa72aab2-7d1d-4c62-83e3-a01afac7ee1c/whereabouts-cni-bincopy/0.log"
Sep 09 04:39:27.409967 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:27.409949    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x5fkn_aa72aab2-7d1d-4c62-83e3-a01afac7ee1c/whereabouts-cni/0.log"
Sep 09 04:39:27.493725 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:27.493698    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wjdpl_cbc782d4-f5b3-4fc8-8484-4f30a5f0f2ea/kube-multus/0.log"
Sep 09 04:39:27.685736 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:27.685703    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xwl9d_57ca0680-7caf-465f-bedd-44a747cc90af/network-metrics-daemon/0.log"
Sep 09 04:39:27.712402 ip-10-0-142-198 kubenswrapper[2580]: I0909 04:39:27.712374    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xwl9d_57ca0680-7caf-465f-bedd-44a747cc90af/kube-rbac-proxy/0.log"